MOSAIC: Modular Foundation Models for Assistive and Interactive Cooking
Authors: Huaxiaoyue Wang
Kushal Kedia
Juntao Ren
Rahma Abdullah
Atiksh Bhardwaj
Angela Chao
Kelly Y Chen
Nathaniel Chin
Prithwish Dan
Xinyi Fan
Gonzalo Gonzalez-Pumariega
Aditya Kompella
Maximus Adrian Pace
Yash Sharma
Xiangwan Sun
Neha Sunkara
Sanjiban Choudhury
Venue: CoRL 2024 (2024)
Keywords: Foundation Models
Human-Robot Interaction
Model Learning
TL;DR: MOSAIC is a modular architecture that enables multiple home robots to collaboratively cook with humans.

Abstract: We present MOSAIC, a modular architecture for coordinating multiple robots to (a) interact with users using natural language and (b) manipulate an open vocabulary of everyday objects. At several levels, MOSAIC employs modularity: it leverages multiple large-scale pre-trained models for high-level tasks like language and image recognition, while using streamlined modules designed for low-level task-specific control. This decomposition allows us to reap the complementary benefits of foundation models and precise, more specialized models, enabling our system to scale to complex tasks that involve coordinating multiple robots and humans. First, we unit-test individual modules with 180 episodes of visuomotor picking, 60 episodes of human motion forecasting, and 46 online user evaluations of the task planner. We then extensively evaluate MOSAIC with 60 end-to-end trials. We discuss crucial design decisions, limitations of the current system, and open challenges in this domain

MOSAIC: Modular Foundation Models for Assistive and Interactive Cooking
Huaxiaoyue Wang", Kushal Kedia", " Juntao Ren", Rahma Abdullah, Atiksh Bhardwaj, Angela Chao, Kelly Y Chen, Nathaniel Chin, Prithwish Dan, Xinyi Fan, Gonzalo Gonzalez-Pumarrie Aditya Kompella, Maximus Adrian Pace, Yash Sharma, Xiangwan Sun, Neha Sunkara, Sanjiban Choudhury Comell University
Abstract: We present MOSAIC, a modular architecture for coordinating multiple robots to (a) interact with users using natural language and (b) manipulate an open vocabulary of everyday objects. MOSAIC employs modularity at several levels: it leverages multiple large-scale pre-trained models for high-level tasks like lan- guage and image recognition, while using streamlined modules designed for low- level task-specific control. This decomposition allows us reap the complemen- tary benefits of foundation models well as precise, more specialized models. Pieced together, our system is able to scale to complex tasks that involve coor- dinating multiple robots and humans. First, we unit-test individual modules with 180 episodes of visuomotor picking, 60 episodes of human motion forecasting, and 46 online user evaluations of the task planner. We then extensively evaluate MOSAIC with 60 end-to-end trials. We discuss crucial design decisions, limi- tations of the current system, and open challenges in this domain. The project's website is at https://portal-comaet conaalll.oollllllaee www.ell.gttuu..i/MMOAAA.
Keywords: Robot Learning. Foundation Models, Hluman-Robot Interaction
Introduction
Collaborative tasks household environments present significant challenges for robots. Consider the scenario in Figure 1, where human user collaborates with two robots prepare a meal. We'd like for communication with the system should feel natural to the user. Furthermore, each robot should be able complete auxiliary tasks across a wide range of objects while fluidly collaborate with the humans. Prior systems in this domain 2,, 3, 4, 5] have demonstrated impressive capabil- ities. However, they either they operate in isolation and lack meaningful collaboration with humans, or employ highly scripted behavior. In this paper, we aim to overcome both of these limitations by designing a system that fluidly collaborates with humans and performs a wide range of tasks.
While single end-to-end model works well for tasks like language understanding where large amounts of data are available, such an approach is difficult for collaborative robots, where less data is available and precise control is important. Our key insight is that by modularizing our architecture, we can segment out parts of the framework that require broad generalization, such as language and image recognition, from the portions that require task-specific control. Furthermore, we can specific component, enabling efficient system improvement.
triage end-to-end system failures
We operationalize this insight to create MOSAIC (Modular System for Assistive and Interactive Cooking): an architecture that applies modularity at multiple distinct levels to significantly improve the overall system's 's performance. Each module has a well-scoped task which is simpler to complete
Denotes equal contribution.
Correspondence to: Huaxiaoyue Wang, yukiwang@@ss.ommlll.oo
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

The goal of the task planner continuously interact with human user using natural language, delegate subtasks to different robots the user, and monitor progress. After agreeing upon task with the user (e.g. "Prepare vegetable soup"), the task planner uses LLM and an online recipe to represent the task as direct acyclic graph (DAG) to model temporal dependencies between am preparing vegetables for soup. Can you pour some salt after stirring?
R1 Subtasks
Subtasks
Stir
RI Current Code
R2 Current Code gou ("pantey") pick opl"eateny go Lot Labteeb)") place("tabler)
Fetch Salt
Pour salt
pick "Ladle") move am "pott) stir(""pe"")
Figure I: MOSAIC cooking in the kitchen. MOSAIC interacts with a user via natural language and controls a tabletop manipulator (R) I) and mobile manipulator (R2) to prepare vegetable soup with the user.
and results in fewer overall mistakes. While the principle of modularity has been central to develop- ing robust real-world robotic systems, these modules have historically been robot/task specific. In contrast, our architecture integrates penerol-purpose pre-trained models to solve robotic tasks. This design choice enables us to build a system that is flexible, interpretable, and scalable.
Our contributions can be organized into three groups:
1. We architect MOSAIC: full-stack Modular Cooking Assistant. We present novel frame- work for home robots that integrates multiple large-scale pre-trained models. In particular, we use large language models for interactive task planning, vision language models for visuomotor skills, and motion forecasting models for predicting human intents for collaboration. We detail Section the key design decisions we made to ensure our system is scalable and interpretable.
2. We perform comprehensive evaluation of MOSAIC. We extensively test the limits of our sys- tem through 60 end-to-end trials where two robots collaborate with human user to cook complex, long-horizon recipes (Section 3.2). We also test individual modules with 180 episodes of visuomo- tor picking (Section 3.4), 60 episodes of human motion forecasting (Section 3.5), and 46 online user evaluations of the task planner (Section 3.3)
3. We analyze both MOSAIC's successes and failures to derive actionable insights for the field. The modular nature of our architecture lends itself to error diagnosis for each failure, one can clearly pinpoint which component failed and why it did. To this end, we distill key findings from our evaluations into limitations and exciting directions of future work based on our current system.
2 Approach
We present MOSAIC, Modular System for Assistive and Interactive Cooking, a modular archi- tecture that combines multiple large-scale pre-traine trained models to solve collaborative cooking tasks. Fig. shows the three main components of MOSAIC: 1) Interactive Task Planner (2. mod- ule that interacts with real users via natural language to plan diverse set of tasks and coordinate subtasks during the cooking process; 2) Human Motion Forecasting (2.2): module that leverages motion forecasting models to predict human motion such that robots can seamlessly collaborate with humans while maintaining margin of safety avoid human-rofhs collisions; 3) Visuomotor Skill (2.3): module that generalizes robot skills diverse set of kitchen objects and environments.
We make a set of simplifying assupmtions in our work, which we detail in Appendix B.
2.1 Interactive Task Planner

---

Interactive Task Planner
LLM Behavior Tree
VLM
Gaable" place, pechraalle'
Ferc
"esit"..
pickLES( "iccuas""
cant, aca nod "porns
Motion
Powe Istirator
Forecast
pick_up (ccle""
stir("pot"")
Tash Speclii, Rubor Fulls
Human Motion Forecasting
Visuomotor Skill
Figure 2: MOSAIC System Overview. The Interactive Task Plannrr module communicates with the user via natural language to decide on a recipe. It assigns subtasks to each robot accordingly. The Human Motion Forecasting module extracts and converts the human's 2D post to 3D coordinates, which it uses to predict future human motion. Separately, VLM takes image language as input and produces a grasp pose around the ohject of interest. Combined, all three are taken by the execution policy of the Visuomotor Skill module to produce a final rohot action.
different subtasks. From this representation, the task planner assigns and maintains a queue of subtasks for each robot based on the current agents' status and the user's requests. To execute subtask (e.g. "fetch salt"), the task planner generates code snippet that issues a series of API calls such go-to ("pantry"), pick ("pepper") etc. The initial set of subtasks and their mutual dependencies is generated once after the recipe is decided upon, the task planner can reassign subtasks and accept new ones based on the user's 's input.
While many recent approaches [1, 6, 7, 8, 9] directly use LLMs for task planning, we observe two main challenges. First, due to the large action space, the LLMs tend to violate constraints that the developer specifies even with chain-of-thought prompting [10]. Second, this approach requires all constraints to be specified in one monolithic prompt, which gives the developer little control over the propose an architecture that embeds LLMs within a behavior tree (BT) [11] (as shown in Figure 2). The nodes break the entire reasoning process down into easier reasoning process for LLMs to think about, thereby reducing the complexity and potential error rate of the LLMs and making easy to scale to multiple behaviors. Finally, adding new behavior is simple as creating prompt for that behavior and adding it an option for other behaviors invoke. No change the code is necessary. Further details in Appendix C and exact prompts for all nodes are listed in Appendix I.
LLMs' behavior and is challenging debug and scale. To overcome both challenges,
2.2 Human Motion Forecasting
Seamless and fluid coordination with humans while maintaining margin of safety requires fore- casting human motion. However, accurately forecasting human motion dynamic environments such kitchens is challenging, as humans can perform wide range of motions, such manipulat- ing various objects in the kitchen moving between stations. Even with large amounts of training data and long context window, current state-of-the -hrtte-artttttto models struggle to accurately predict human motion at all times. We aim to build a forecasting model that generates predictions that sufficiently capture the impact of forecasted human motion during interactions with the robot.
Training Pipeline. We first pre-train our model on AMASS [12] large dataset of human activ- ity, encompassing over 300 subjects and 40 hours of motion capture data. However, AMASS only consists of general single-human movements (e.g. jumping, walking, dancing), which are not rep- resentative of a human's 's motion when working in close proximity with robot. To this end, we utilize the Collaborative Manipulation Dataset (CoMaD) [13], a dataset consisting solely of human- human interaction episodes in a kitchen setting. For each episode in CoMaD, we identify these periods where both humans are in close proximity and construct a transition dataset. We sample data equally from the transition dataset and the entire CoMaD dataset to train the motion forecaster. This approach helps the forecaster maximize task efficiency by upsampling critical periods of the interaction when the human is likely to approach and interact with the robot.

---

Inference Time: Real-time, Vision-based Forecasting and Planning. Given an RGB-D image of the human, we use MediaPipe [14] to extract the 2D positions of their upper-body joints. These locations are then back nack-projected to 3D world coordinates using the image depth map and used to generate real-time motion forecasts for robot planning. However, the 3D coordinates obtained using the RGB-D camera are noisier than the high-fidelity motion capture data which the model trained By injecting random Gaussian perturbations into the model's input training time, force the forecaster to learn to denoise potentially noisy input and generate smooth forecasts.
2.3 Visuomotor Skills
The visuomotor skills module takes the goal specification from the task planner, forecasts from the forecasting module, and observations from cameras and outputs series of actions to complete the task. A common approach is end-to-end training on a suite of demonstra- tions [15, 16, 17, 18, 19, 20, 21], though good test-time performance generally requires opti- mal demonstrations have good coverage over the observation space, which are challenging and time-consuming to collect. Instead, de- couple perception and action generation into set of individual modules, where different skills have the flexibility to use different combina- tions as shown in Table 1. We describe these modules below, with more details in App.
Obj. Detect
Action
Skall
Freq.
Forecast
Exec.
pick (<obi>)
315:
Learned
place (<loo>)
20%
Engineered
stir (xcbj>)
11%
Engineered
handovar
7%
Engineered
pour (R)bj))
5.5%
Enginsered
go.to[sloox
Planning
Table I: Subcomponents Used per Skill. pick () takes up the highest proportion of calls, and requires policy that satisfies tight set of constraints. Thus, we leam the policy via reinforcement learning (RL). golto () assumes access to a map of the environment, and plans a path between the given start and goal.
Object detection. Given image and language
input, we obtain a set of bounding boxes from OwIViT [22] and take the bounding-box coordinate with the highest CLIP similar score [23]. We use FastSAM [24] to obtain a more accurate segmenta- tion of the object within the bounding box, and back- back-project the segmented pixels through the depth camera's 's point clouds and take the goal location be its center of mass.
Action execution. For simple skills where there object-of-interest, we use an IK-based controller and set of engineered primitives (e.g. stirring motion), which take as input the object position and human motion forecasts to solve the task. For skills where such collisions may naturally occur during execution, we train an off-the-shelf reinforcement learning algorithm in a simulator that has access to an approximate robot dynamics model and a target grasp position. No action executed the simulator the predicted action results in a collision, and instead gets negative reward proportional the distance from the goal. Further details the motion primitives, simulator, and reward function in Appendix D.
obstacles in the path between the robot and
Integration with human motion forecasts. Some skills use the current end-e position and user's motion forecasts (from Section E) to avoid collisions with the human (e.g. user drops food into the pot while the robot executes stir( ()) ) and anticipate future positions (e.g. handover () )
Experiments
3.1 Setup
In experiments, the mobile manipulator is 6-DoF Stretch Robot REI [25], and the tabletop ma- nipulator is 7-DoF Franka Emika Research [26]. The kitchen has two overhead RGB-D cameras that perceive the workspace and capture human's 's motion. To allow users to interact with the task planner, we use Google's speech-to-text APIs [27] to transcribe user's verbal instructions and its text-to-speech APIs to vocalize the task planner's 's responses.
3.2 End-to-end Trials
The goal of end-to-end trials is to categorize how much failure each module has when the entire system is integrated and running together. To this end, conduct a total of 60 end-to-end trials with

---

Recipe
Typical Robot Skills Used
Success
Subtasks Comp.
Robot Subtasks
R1:
Fu
Gote
Paiee
Fetch sth
Put away sth
Toss Salad
8/ 10
92.5%
Handover sth
Pour sth
R2:
F1
Plcc
Failure Cases
Tuna Sandwich
/
96.0%
R2:
(F)
RI:
F1B
Picc
(E)
Vegetable Soup
/ 10
96.0%
4
(A)
10
R1:
So
Place
(D)
3
Corn Soup
R2:
Pab
P+
/
90.0%
3
3
(C)
(B)
Caesar Salad
/ 10
86.7%
R2
Pab
Pau
Paace
(A) Pick Failed
(B) Place Falled
R1:
Go
Fab
Gorta
(C) Dropped Obj
Chicken Soup
6 10
91.4%
(D) Interrupt Failed
(E) Wrong Assignment
41 / 60
91.6%
(F) Pose Tracking Failed
Figure 3: End-to-end results. On-policy results for 6 recipes, where each recipe tested through 10 trials. Each recipe contains various subtasks involving different robot skills. We report the number of trials that are completed without any errors and the individual subtask completion rate. We also categorize the failure cases. MOSAIC is able to complete 41/60 tasks with an average subtask completion rate of 91.6%.
two robots and a user collaboratively making 6 recipes. Each recipe involves a different combination of robot skills and different types of interaction with the user.' For example, users may provide vague instructions, interrupt robot's subtask, and add new subtasks that are not in the recipe. Overall, MOSAIC completes 41/60 (68.3%) collaborative cooking trials of 6 different recipes with an average subtask completion rate of 91.6%.
Modularity enforces each sub-module to have a clear input/output contract, allowing one to localize failures and extract transferable insights. We use this cluster failures into the 6 categories as shown in Figure 3. Specifically, errors originating from the task planner module usually come due to incorrect transcriptions of the user command from the text-to-speech sub-module. Errors in the perception module of visuomotor skills lead to an incorrect object identification or an insufficiently stable grasp. Likewise, tracking errors arise when the user moves outside the camera's view. At heart, complex tasks with multiple humans and robots. where failures are inevitable, modularity makes easy to triage and treat failures. In the following sections, we lift the insights from the end-to-end trials to each module and analyze how limit errors therein.
3.3 Interactive Task Planner
Since the task planner directly interacts with the users, a frequent failure mode is constraint violation (c.g. acting without permission). Thus, it is crucial for it to exhibit predictable behaviors, especially when planning over long task horizons. To this end, we first quantify the frequency in which each constraint gets violated when the task planner interacts with real users in an online user?. We com- pare the proposed approach, which embeds LLMs within behavior tree (Tree), against directly calling the LLM once with one prompt (One-Prompt) The monolithic LLM prompt has constraints that it must follow, explanations of what actions to choose in each situation, and in in-context examples.
In the study, each user is randomly assigned to interact with either Tree or One- Pre-Prompt and is asked to analyze the task planner changes subtasks without the user's permission (Act Without Permis- sion), claimed robots can do subtasks beyond their capabilities (Lying), and did not respond to the user's 's instruction (Ignoee User). We explicitly instruct participants to engage in non-nominal inter-
1 'The trials had authors acting as users and involved total of users between the ages 20-30. 'The user study was approved by the Institutional Review Board at the University. See Appendix G for experimental setup, user study interface, and survey questions.
5

---

Experiments
Approach
Constraint Violation
Combined
One-Prompt
52
32
(46 responses]
Tree
28
13
Internal Study (26 responses)
One-Prompt
16
Tree
20
4
External Study (20 responses)
One-Prompt
Tree
8
10
20
30
50
60
70
90
100
Without Permission
Lying
Ignore User
Figure 4: Task Planner Constraint Violations During Real User Interactions. We receive 46 responses in total (26 from internal and 20 from external study). Each user gets assigned either Tree or One-Prompt We present the total number of constraint violations per category. Tree makes 62.8% fewer constraint violations compared One-Oronee One-Prompt for the combined responses, 36.2% fower for internal, and 62.2% fewer for external.
actions with the task planner in hopes of eliciting a constraint violation. We received 26 responses from lab members who are not familiar with the task planner's capabilities and 20 responses from external users on Prolific [28]. a crowd-sourcing website. Quantitatively, LLM with behavior tree violates 47.8% fewer constraints compared to the baseline of using one LLM prompt (Figure
Table 7 in the Appendix provides examples and analysis of One-Promp''s constraint vio- lations. Finally, aggregated user feedback at the end of the survey indicates Tree generally provided better user experience than One- Prompt. A user assigned with Tree stated "It worked expected, quick and concise an- swers, compliant, didn't make any mistakes." Meanwhile, a user with One-Prompt com- mented "I could definitely see myself blow- ing my top with the level of disobedience."
Low Clutter
Med. Clurter
High Clutter
Total Success
OwIVit only (22] OwIVit CLIP (Ours)
1W10
3/10 10/10
0/10 6/10
13/30
10/10
26/30
Table 2: On-policy Evaluations of Different Vision Modules for pick ( (<obj>) The architecture is tested its ability pick up the language-specified object when (i) a single object are in the pantry, and (iii) 7-15 objects in the pantry.
pantry, (ii) 2-6 objects
Furthermore, localizing failure modes via modularity sufficiently scopes down the problem such that we can programmatically evaluate the approaches unit tests. Specifically we test whether the task planner properly handles a request and chooses the right action as the interaction becomes more complex (e.g. user always disagrees with task planner and reassigns subtasks). Tree remains above 90.0% for its unit test pass rate, while One-Prompt's performance drops from 100% (for easy cases) to 60.0% (for difficult cases), as the number complex interactions increases. More details in Table in Appendix H. Overall, The results suggest that compartmentaliiing the action space by adding explicit structure to each LLM's reasoning problem significantly helps the task planner to respect constraints.
3.4 Visuomotor Skills
End-to-end trials showed imprecision in the perception module to common source failure for visuomotor skills. Thus, now focus our analysis on quantifying how the vision component influ- ences policy performance and qualifying common failure cases. We place the mobile manipulator in front of pantry with increasing number of objects and test the success of pick (), the skill with the highest utilization frequency across all end-to-end runs.
Since localizing the error within the vision module, we found that directly using OwIViT [22] leads rapid deterioration in accuracy when clutter increases, the pre-trained model had difficulty identifying the correct bounding box from a large set of proposals. To rememdy this, apply post- processing via Mon-Maximum Suppression [29] and CLIP [23] (abbreviated as OwlViT CLIP). Table 2 shows OwIVIT + CLIP increases skill completion by 70% in medium-clutter regimes, and 60% in high-clutter regimes. Critically, modularity helps to specifically identify the erring module, which allows improvements therein to have system-wide benefit.

---

Additionally, figure 5 qualifies three failure cases of the vision module. First, the presence of too many objects (especially similarly-shaped ones such as various season- ing bottles) leads to suboptimal set bounding box pro- posals for CLIP score. Second, when lighting and/or color blends the object contours into the background, only parts of the object may be included in the bounding box, resulting in a lower CLIP score. Lastly, imprecise prompts produce poor bounding box proposals.
Deraity
Backgrondd
"Onagano Seasoning
Waite
Pringlos Can"
3.5 Human Motion Forecasting
Finally, we analyze how using motion forecasts with noisy model inputs during training allow the module to be more robust during end-to-end trials through on-policy evaluations of 7-DOF Franka robot arm collaborating with real human user on two tasks.
Drogaro Seasoning
"Water Bortic"
X Chips
Figure 5: Vision backbone example failure cases. We find that cluttered background and poor lighting conditions lead to a sub- optimal set of bounding boxes for CLIP to score. while more specific prompts produce better bounding box proposals.
The stir () skill involes the robot stirring a pot while the human periodically adds in vegetables. We measure the time takes the robot detect that the human arm is reaching into the pot (TIME TO REACT (MS)), the minimum distance maintained between robot arm and human hand (SAFETY MARGIN (CM)), and the number of times the human hand comes within minimum threshold of the robot arm (COLLISSOONS). In the handover () task, the user asks the robot to pick up and handover objects. We measure the average time to complete the handover (TIME TO GOAL (MS)) and the movement efficiency (PATH LENGTH (CM)), which measures the distance tracked by the robot's end-effector.
We compare against two baselines: (1) Current which assumes the current human pose will be its pose across the entire planning horizon, and (2) Forecast (Base) [30] that is not trained on noisy input. Each baseline and skill combination is evaluated 10 times for a total of 60 evaluations.
First, we find that using forecasts during the stir () skill significantly improves on metrics, maintaining 74% greater SAFETY MARGIN from the human on average and, more importantly, avoids any collisions. In contrast, we observe that the robot reacts very late when using the human's current pose, and results in collisions 20% of the time shown in Table 3. In the handover skill, the robot is 24% slower in completing the task following the current human wrist position compared to using the handover location predicted by our forecaster. Using the forecast, the robot moves directly toward the handover location, finishing the skill with 28% shorter trajectories.
Next, we ablate on training with noisy inputs by comparing our forecaster with a baseline approach (Base) [30] that does not train noisy model inputs. In the stir () task, our forecaster has 23% quicker reaction time human movements. Further, there is more variability in the performance of the Base forecaster (measured by the variance of each metric) that can be attributed to greater sensitivity to noisy inputs. Similarly, in the handover () skill, the Base forecaster's predictions are often erratic leading to jerky movements by the robot arm. Following them no better than using the current human position for planning, as measured by task completion time and path length. Overall, we find noise injections to the forecasting model make it more robust perception errors, while using forecasts improves several key performance metries of downstream skills.
4 Discussion and Limitations
We decompose the overall problem of interactively cooking with a human user into a set modules leveraging general-purper pre-trained models. We localize errors our system to individual mod- ules and conduct targeted experiments. However, number of open challenges still remain. First, improvements can be made to the task planner by grounding it with eras and sensors. Further, expanding to new environments in scalable and flexible manner may
input such as cam-
7

---

REACTIVE STIRKING
HANDOVERS
Model 1
SAFETY MARGIN (cm) 1
TIME TO REACT (ms) COLLISIONS
TIME TO GOAL (6)
PATH LENGTH (cm)
Current
13.5 (2:02)
135.4 (=104)
2/10
1.54 (20.1)
31.5 2)
Forecast (Base) [30]
19.9 (#002)
0/1
1.67 (+0.2)
32.7 (+3.0)
Forecast (Ours)
23.1 (2002)
48.9 (2:000)
0/10
(21.)
22.4 (2:0.)
Task-Specific Performance Metries. We evaluate the robot's interactions with the human user 2 collaborative manipulation tasks. Integrating forecasts into the robot's skills improves fluidity and increases safety margin across all metrics. We observe that relying on the current human pose during REACTIVE STIR- RING is risky and results in collisions. ROBOT-HANDOWS HANDOVER tasks are more efficient using forecasting.
Table 3:
Current Pose
Forecasted Pase
Figure 6: On-Policy Reactive Stirring. (I.eft) Current: Using the human's current pose results in delayed robot reaction and collision once the human's hand enters the pot. (Right) Forecast Using the forecasted human position results a smoother interaction and quicker reaction time, avoiding collision.
require us to revisit previous assumptions and adopting new capabilities. Currently, the system's 's capabilities remain static after being deployed in an everyday user's household. An exciting area of future research to continuously learn from real-time human feedback and interactions.
5 Related Work
Home Robots. Recent research efforts have attempted to provide robots with generalist capabil- ities to sufficiently adapt to home-like environments [1, 3, 4, 31]. However, many of these works [3, 4] are limited to completing predefined tasks that don't 't require explicit task planning, e.g. picking single item. Liu et al. [31] similarly tackle open-vocabulary navigation, but still sidesteps the challenge of a dynamic environment by assuming static representation of the world after initial- ization. On the other hand, some works consider multi-arm/mmulii-rbbo planning for collaborative tasks [9, 32, 33, 34]. For example, Mandi et al. [9] significantly constrains human-robot collabora- tion by forcing the human to complete a specific task before the robot proceeds with its own task. In this paper, we aim to overcome these limitations by designing system that enables multiple robots to fluidly collaborate alongside humans to perform wide range of tasks.
Specific Modules. Our interactive task planner module is similar to work in single-robot settings with clearly defined language goals that generate list of actions as the plan [8, 35, 36, 37, 38, 39] and synthesize code that calls robot action API [6, 40, 41, 42]; however, most of these works are non non-interactive. In contrast to the most similar interactive work [8], we are solving a multi- agent task planning problem involving two robots and a user and communicating with the user to allocate tasks properly. Our visuomotor skills module is similar to the family of prior work [15, 19, 20, 43, 44, 18, 18, 44, 45, 46] that leverages VLMS for object identification within manipulation tasks. However, in contrast to prior work [47, 48, 42, 49, 45, 46], we train our action policy using reinforcement learning in simulation where affordances are provided by the VLM and constraints are inherent to the simulator. Collaborative manipulation tasks near humans necessitate human motion prediction, traditionally bypassed by assuming static human [50, 51]. Advances in neural networks and the availability of extensive human motion datasets [12, 52, 13] have enabled the development of sophisticated RNN and GNN models to predict movements from past joint positions [53, 54, 55, 56, 57, 58, 30].
8

---

Acknowledgments
This work was supported in part by the National Science Foundation FRR (#2327973) and the Na- tional Science Foundation RI (#2312956). Sanjiban Choudhury is supported in part by the Google Faculty Research Award and the OpenAI Superalignment Grant. We thank Gokul Swamy for giv- ing valuable feedback and helping improve the writing. We thank Mehrnaz Sabet for helpful assistance with the user study.
References
[1] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, et al. Do i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[2] M. Bajracharya, Borders, R. Cheng, D. M. Helmick, L. Kaul, D. Kruse, Leichty, J. Ma, C. Matl, F. Michel, C. Papazov, J. Petersen, K. Shankar, and M. Tjersland. Demonstrating mobile manipulation in the wild: A metrics-driven approach. In K. E. Bekris, K. Hauser, S. L. Herbert, and J. Yu, editors, Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14 2023, 2023, 15607 RSS. 2023 XIX. 055.
URL https: http://oooo.oo/g//// org/10.
[3] N. M. M. Shafiullah, A. Rai, H. Etukuru, Y. Liu, I. Misra, S. Chintala, and L. Pinto. On bringing robots home. arXiv preprint 1.Xiv:2311.16000 2023.
[4] Yenamandra, A. Ramachandran, K. Yadav, A. Wang, M. Khanma, Gervet, T.-Y. Yang, V. Jain, A. W. Clegg, J. Turner, et al. Homerobot: Open-vocabulary mobile manipulation. arXiv preprint arXiv:2306. 1565, 2023.
[5] I. Idrees, T. Yun, N. Sharma, Y. Deng, N. Gopalan, G. Konidaris, and S. Tellex. Improved inference of human intent by combining plan recognition and language feedback, 2023.
[6] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control, 2023.
[7] B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone. Llm+p: Empowering large language models with optimal planning proficiency, 2023.
[8] B. Li, P. Wu, P. Abbeel, and J. Malik. Interactive task planning with language models, 2023.
[9] Z. Mandi, S. Jain, and S. Song, Roco: Dialectic multi-robot collaboration with large language models, 2023.
[10] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of- thhhugt prompting elicits reasoning in large language models, 2023.
[11] M. Colledanchise and P. Ögren. Behavior trees robotics and AI: an introduction. CoRR, abs/1709.00084, 2017. URL ht /ttp://aaxvvvv// org/ abs 111.//100100. 00084.
[12] N. Mahmood, N. Ghorbani, of motion capture as surface shapes. International Conference on Computer Vision, pages 5442-5451, Oct. 2019.
Troje, G. Pons-Moll, and M. Black. AMASS: Archive
[13] K. Kedia, Bhardwaj. Dan, and Choudhury. Interact: Transformer models for human intent prediction conditioned on robot actions. ArXiv, abs/2311.11944,, 2023.
[14] V. Bazarevsky, I. Grishchenko, K. Raveendran, T. L. Zhu, F. Zhang, and M. Grundmann. Blazepose: On-device real-time body pose tracking. ArXiv, abs/2006. 10204, 2020.

---

[15] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Rohotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06887,, 2022.
[16] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137 2023.
[17] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning, In Conference on Robot Learn- ing. pages 991-1002. PMLR, 2022.
[18] A. Stone, Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong. P. Wohlhart, B. Zitkovich, F. Xia, C. Finn, et al. Open-world object manipulation using pre-trained vision-language mod- 2023.
cls. arXiv preprint
[19] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-actioo models transfer web knowledge to robotic control. arXiv preprint 2rXiv:2307.15811 2023.
[20] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference Robot Learning, pages 894-906. PMLR, 2022.
[21] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- lation for robot manipulation. arXiv preprint arXiv:2203. 2022.
[22] G. Heigold, M. Minderer, A. Gritsenko, A. Bewley, D. Keysers, M. Lučié, Yu, and Kipf. Video owl-vit: Temporally-consiseee open-world localization in video. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 13802-13811, 1, 2023.
[23] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 9748-8763 PMLR, 2021.
[24] X. Zhao, W. Ding, An, Y. Du, T. Yu, M. Li, M. Tang, and J. Wang. Fast segment anything. arXiv preprint arXiv:2306. 12/56, 2023.
[25] C. C. Kemp, A. Edsinger, H. M. Clever, and B. Matulevich. The design of stretch: A compact, lightweight mobile manipulator for indoor human environments, 2022.
[26] Franka research 3, 2022. URL htts:://t/// det.:/uuaa.dd/eddmettttt..t
[27] URL htl.tttp///// htttttt///coouee htt.://dd..oogll....o////eeeeeee
[28] Prolific, 2014. URL https:/ http://wwwwwwr http://wwpww.....oooooo
[29] P. Viola and M. J. Jones. Robust real-time face detection. International journal of computer vision, 57:137-154, 2004.
[30] K. Kedia, Dan, A. Bhardwaj, and S. Choudhury. Manicast: Collaborative manipulation with cost-aware human forecasting. ArXiv, abs/2310.132588 2023.
[31] P. Liu, Y. Orru, C. Paxton, N. M. M. Shafiullah, and I.. Pinto. Ok-robot: What really matters in integrating open-knowledge models for robotics. arXiv preprint arXiv:2401. 2024.
[32] M. Dogar, A. Spielberg, S. Baker, and D. Rus. Multi-robot grasp planning for sequential assembly operations. Autonomous Robots, 43:649 664, 2019.
[33] H. Ha, J. Xu, and S. Song. Learning decentralized multi-arm motion planner. arXiv preprint arXiv:2011.02608, 2020.
10

---

[34] A. Tika and N. Bajeinea. Predictive control of cooperative robots sharing common workspace. IEEE Transactions on Control Systems Technology, 2023.
[35] M. Ahn, Brohan, N. Brown, Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho. J. Hsu, Ibarz, Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao. K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng. Do as can, not as say: Grounding language in robotic affordances, 2022.
[36] W. Huang. F. Xia, T. Xiao, Chan, Liang, Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, S. Levine, K. Hausman, and B. Ichter. Inner monologue: Embodied reasoning through planning with language models, 2022.
[37] W. Huang, P. Abbeel, D. Pathak, and Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents, 2022.
[38] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Tex22motion: from natural lan- guage instructions to feasible plans. Autonomous Robots, 47(8):1345-1365, Nov. 2023. ISSN 1573-7527. doi: 023-11111... URL http: dww. http:://bx..d...ogg110.000
[39] S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Pro- cessing Systems, 36, 2024.
[40] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models, 2022.
[41] H. Wang, G. Gonzalez Pumaarrea, Y. Sharma, and S. Choudhury. Demo2code: From summa- rizing demonstrations to synthesizing code via extended chain-of-thought, 2023.
[42] Wu, R. Antonova, A. Kan, M. Lepert, Zeng. S. Song. J. Bohg, S. Rusinkiewicz, and T. Funkhouser. Tidybot: personalized robot assistance with large language
models. Autonomous Robots, 1788111087 Nov. 2023. ISSN 1573-7527. http://.......1111010.00..707
doi: 401:10.1007//1011140000--1111110
URL
110514-023-101322
[43] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv, 2022.
[44] S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language- driven representation learning for robotics. arXiv preprint arXiv:2302. 2023.
[45] Liu, K. Fang, P. Abbeel, and S. Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. arXiv preprint arXiv:2403.03174 2024.
[46] S. Nasiriany, Xia, W. Yu, Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872 2024.
[47] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arX.v:2303.0337. 2023.
[48] W. Huang, C. Wang. R. Zhang, Y. Li, J. Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.0513 2023.
11

---

[49] P. Sundaresan, S. Belkhale, D. Sadigh, and J. Bohg. Kite: Keypoint-conditione policies for semantic manipulation. arXiv preprint arXiv:2306. 0002230666600 16605, 2023.
[50] W. Yang, B. Sundaralingam, C. Paxton, 1. Akinola, Y.-.. Chao, M. Cakmak, and D. Fox. Model predictive control for fluid human-to-robot handovers. In 2022 International Confer- ence on Robotics and Automation (ICRA), pages 6956-6962. IEEE, 2022.
[51] E. A. Sisbot and R. Alami. A human-aware manipulation planner. IEEE Transactions Robotics, (915):045-1000 2012.
[52] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions pattern analysis and machine intelligence, 36(7):1325-15 1339, 2013.
[53] H. Ling, G. Liu, L. Zhu, B. Huang, F. Lu, H. Wu, G. Tian, and Z. Ji. Motion planning combines human motion prediction for human-robot cooperation. In 2022 12th International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), pages 672- 677. IEEE, 2022.
[54] V. Unhelkar, P. A. Lasota, Q. Tyroller, R.-D. Buhai, L. Marceau, B. Deml, and J. A. Shah. Human- robotic assistant for collaborative assembly: Integrating human motion predic- tion with planning in time. IEEE Robotics and Automation Letters, 3:2394-2401, 2018.
[55] J. Mainprice, R. Hayne, and D. Berenson. Predicting human reaching motion in collabora- tive tasks using inverse optimal control and iterative re-planning. 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 885-892, 2015.
[56] V. Prasad, D. Koert, R. M. Stock-Homburg, J. Peters, and G. Chalvatzaki. Mild: Multimodal interactive latent dynamics for learning human-robot interaction. 2022 IEEE-RAS 2 Inter- national Conference on Humanoid Robots (Humanoids), pages 472-479, 2022.
[57] W. Mao, M. Liu, and M. Salzmann. History repeats itself: Human motion prediction via motion attention. In European Conference on Computer Vision, 2020.
[58] T. Sofianos, A. Sampieri, L. Franco, and F. Galasso. Space- Space-time-separabbe graph convolutional network for pose forecasting. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11189-11198, 2021.
[59] J. X. Liu, Z. Yang, I. Idrees, S. Liang B. Schornstein, S. Tellex, and A. Shah. Grounding complex natural language commands for temporal tasks in unseen environments, 2023.
[60] S. Raman, Cohen, D. Paulius, I. Idrees, E. Rosen, R. Mooney, and S. Tellex. Cape: Corrective actions from precondition errors using large language models. arXiv preprint urXiv:2211.09935. 2022.
[61] G. Brewka, Eiter, and M. Truszczyíski. Answer set programming at a glance. Commun. ACM, 54(12))92-103, dec 2011. ISSN 0001-0782. dot:100.115//2433744200 https: / /oo. wrg/10 21155/22241174241111533 2043195.
URL
[62] Y. Jiang, S. Zhang, P. Khandelwal, and P. Stone. An empirical comparison of pddl-based and asp-based task planners. CoRR, abs/1884.082220 2018. URL
abs/
1804. 08229.
[63] V. Lifschitz. Answer set programming and plan generation. 138(1):39-54, 2002. ISSN 0004-3702.
Artificial Intelligncce, 00186-8 URL https:/ :///:///www.//// sci encedi rect. com/ / wwc.iince/ariicleeeeeee
$0004370202001862 Knowledge Representation and Logic Programming.
[64] M. Fox and D. Long. PDDL2.1: an extension to PDDL for expressing temporal planning domains. CoRR, abs/l 106.4561, 2011. URL http: arxiv www.ooo//////.oooooooo/ooooooose arg/abs/1106 4561.
12

---

[65] A. Mavrogiannis, C. Mavrogiannis, and Y. Aloimonos. Cook2ltl: Translating cooking recipes to Itl formulae using large language models. ArXiv, abs/2310.00163, 2023.
[66] S. Y. Gadre, M. Wortsman, G. Ilharco, I.. Schmidt, and S. Song. Clip on wheels: Zero-shot object navigation as object localization and exploration. arXiv preprint arX1v:2203.10421, (4):7, 2022.
[67] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, al. Open x-embodiment: Robotic learning datasets and rt-x mod- cls. arXiv preprint arXiv:2310.08864 2023.
[68] Dasari, Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, Singh, Levine, and
C. Finn. Robonet: Large-scale multi-robot learning. arXiv preprint arXiv:1910.1215., 2019.
[69] M. Minderer, A. Gritsenko, and N. Houlsby. Scaling open-vocabulary object detection. arXiv preprint 23X6v2230..09683, 2023.
[70] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint urXiv:1707.06447 2017.
[71] A. Raffin, A. Hill, M. Ernestus, A. Gleave, A. Kanervisto, and N. Dormann. Stable baselines3, 2019.
13

---

We first enumerate more related works in Section A, then outline the configuration of our system, as detailed in Section B. Subsequently, we delve into further discussions on each component of our system, covered across Sections C, D, and E. Finally, we present an in-depth analysis of the user study, incorporating details on the experimental setup and supplementary findings, all which are elaborated in Section G.
Appendix A Extended Related Works
Modular Architectures. Modularization has been extensively employed to partition complex and long-horizon robotics tasks into more easily addressed sub-components. For example, in the space of task planning, different Large Language Model (LLM) modules are used to improve command interpretation in unseen environments [59] and to produce corrective replanning prompts [60]. OK- Robot [31] focuses on the problem of object retrieval and navigation using separate submodules for mapping, object-detection, path-planning, and grasping. Our approach tackles the combined domains of task planning, visuomotor skill learning, and human motion forecasting. We leverage modularity at multiple levels, e.g. we outperform a single VLM object detection by combining Owl-Vit to detect bounding boxes, and CLIP to select the correct box.
Task Planning. A task planner takes as input a high-level task, e.g. cooking a recipe, and generates plan, e.g. sequence of sub-tasks, to achieve that goal. Traditional upproaches frame this a search problem and invoke symbolic planner to solve it [61, 62, 63, 64]. However, using these methods for everyday tasks is challenging because they require pre-defining the search space and lack a natural-language interface to interactively communicate the task. Recent work leverages LLMs for task planning to overcome both of these limitations. In single-robot settings, given clearly defined language goal, recent work can be categorized as generating a list of actions as the plan [35, 36, 37, 38), synthesizing code that calls robot action API [6, 40, 41, 42], or translating to problem solvable by classical planner [7. 65]. However, none of these systems interact with humans and coordinate tasks for both humans and robots.
Idrees et al. [5] focuses on estimating user's intent and using question question-answering interaction with the user to update that estimate. Then, the robot can use that estimate to suggest how the user can make task progress. However, its interaction is limited to yes-or-no questions, and it does not focus on task planning for multiple agents based on the robot's capabilities and the user's unstructured natural language feedback.
Li et al. [8] has the closest task planning framework to our approach, where the LLM takes a specific natural language goal to generate a step-by-step plan before synthesizing robot code for each step. However, because we are solving a multi-agent task planning problem involving two robots and a user, our task planner cannot simply output a list of steps. It must continuously communicate with the user to properly allocate subtasks to suitable agents.
Visuomotor Skills. Several recent works study the application of pre-trained rision-languag mod- els (VLMs) to robotics [18, 66, 15, 19, 47, 20, 43, 44]. One family of recent work [15, 19, 20, 43, 44, 18] integrate pre-trained VLMs in an end-to-end fashion, e.g. segmenting out regions of interest to assist in action prediction [18, 44]. A second flavor of approach [47, 48, 1, 37] leverages VLMs to recognize affordances and constraints the environment and provide corresponding execution instructions through language [47] or code [48]. Our model is similar in this aspect, where we dis- tinguish the training objectives of environment perception and action execution. This effectively liberates from needing large dataset of humans or robots demonstrations to provide good cover- age [18, 15, 19, 20, 21] and from having to worry about embodiment mismatch between large-scale robot learning datasets [67, 68).
Human Motion Forecasting. Collaborative manipulation tasks close proximity to humans re- quire predicting human motion. This is a challenging problem since human motion is complex and highly variable. A common approach is to sidestep the problem of motion forecasting [50, 51] by considering the human to be static. Instead, recent research is moving towards the use of neural
14

---

networks and supervised learning to predict future human motion based on short history of past joint positions [53, 54, 55, 56]. The release of large open-sourced datasets of human motion [12, 52] has made it possible to train large RNN and GNN-based neural network models for human-pon -onea forecasting [57, 58J. Consequently, these datasets have been integrated into robot motion planning. focusing on collaborative manipulation tasks [30, 13]. Closest to this work, ManiCast [30] proposed framework to learn cost-aware human forecasts. However, this approach relies bulky motion camera setup, requiring the user wear motion capture suit with markers. In this work, we run our integrated human motion forecasting and planning system in real-time using single RGB-D camera to track human pose.
Appendix B System Setup
Kitchen Scene and Robot Placement. The kitchen scene consists of a main kitchen table at the center where all cooking activities are performed. A pantry is placed near the table, which contains large range of condiments and kitchen staples. There is also a secondary table on the side of the center table meant for serving up the final dishes. Our robot system includes two robots (R1 and R2).
RI (Franka Emika Research 3 [26]) is a tabletop 7-of manipulator stationed at one end of the kitchen tables at the center of the scene.
R2 (Hello Robot Stretch REI [25] is a mobile manipulator that can navigate around the kitchen area, capable of fetching and putting away condiments and kitchenware as required by the user.
Camera Placement. For the tabletop manipulator (RI), the perception stack includes two Intel Realsense D435i RGB-D cameras placed above the center kitchen table. Both cameras are placed at opposite ends of the table and at an angle such that they capture the entirety of the tabletop as well as the human user. Integrating both camera perspectives enhances the visibility of objects and human poses within a cluttered kitchen setting, effectively mitigating occlusion issues. The mobile manipulator (R2) uses an onboard Intel Realsense D435i RGB-D head-camera for perceiving objects.
Computational Details. In addition to the onboard computing capabilities of the robots, our setup includes five personal computers (PCs) dedicated to running various system modules. These PCs are connected to the same network, utilizing the Robot Operating System (ROS) for communication. For tasks that demand real-time neural network inference, we employ onboard GPUs, (NVIDIA GeForce RTX 3060). Detailed information about each PC's role and configuration is provided:
C1: Connected with Bluetooth microphone and speaker, this PC runs the Speech-Tex system for communicating with the user and the Interactive Task Planner that utilizes GPT- 4 API calls.
C2: Used for running neural network models related to the perception (object detection) and control (RL agent) of R2. This PC also communicates with CI to allocate subtasks to R2.
C3: This PC forms the perception stack for RI, including running neural network models for object detection and pose estimation.
C4: This PC runs the human forecasting model using the pose estimates computed by C3. This PC also computes motion plans for RI based on the predicted object pose and human forecasts. Further, it communicates with C1 allocate subtasks to the R2.
C5: This PC is installed with a real-time kernel to send joint commands to R2 at kHz frequency as recommended by the robot manufacturers.
System Assumptions, In accordance with the setup above, make set simplifying assumptions in our work:
15

---

1. Access to a set of seed recipes: A recipe contains set of subtasks with temporal depen- dencies. We seed the system with an initial set of recipes, but the user has the freedom to make modifications on the fly (e.g. adding an ingredient).
2. Access to map: We assume that our system has mapped the kitchen ahead of time, so it is aware of where ingredients and tools are stored and how to navigate to different locations.
3. Full observability: We assume that objects are not occluded for detection and grasping, though they can be next to each other. We also assume that the upper torso of the human is visible to the cameras for tracking and prediction.
4. Skills API: We assume access to library of robot skills that can be invoked with specific imput parameters (e.g. pick_up ("salt"), stirr)).
Appendix C Interactive Task Planner Details
Prepare lettuce
Prepare salad toppinge
Start
Get pepper
Pour pepper
Get ranch sauce
into bowl
Figure 7: Recipe DAG Example. This DAG represents subtasks and dependencies involved making a Caesar Salad, At the beginning of making the recipe, the available subtasks include 'Prepare lettuce', 'Get pepper', and 'Get ranch sauce'. If one of the subtasks is marked as done, the following subtasks become available (c.g. completion of "Get pepper' causes 'Puur pepper into bowl' to become available).
The interactive task planner consists of three main components: representation of a task and the dependencies of its subtasks, mechanism to decide on a recipe and assign subtasks to others, and a medium to communicate to robots which skills to use for a given subtask. We implement these using a direct acyclic graph (DAG), a behavior tree, and LLM-generated code (communicated over ROS action services).
Task DAG.
The task planner represents task (e.g. "Prepare vegetable soup") DAG, whose nodes represent subtasks of that task and whose edges represent dependencies between the subtasks. However, a DAG alone is insufficient for generating task plan based on the user's instruction because does not specify exactly which robot should complete which subtask.
For recipe, the DAG generated by an LLM prompt ahead of time. Concretely, the LLM takes as input the ingredient list and step-by-step instructions, scraped from the recipe website. 3 It outputs marked-down nested list of the recipe, which is easier for the LLM to reason about and generate in practice. A list item represents subtask, and the nested structure represents the dependencies, so we can programmatically convert the LLM output into a DAG. The complete L.LM prompt is the supplementary material.
The task planner also maintains done state for each node/subtask in the DAG. To determine the available subtasks, start from root node whose done state is set, with outgoing edges to the first subtasks. Then, we follow each outgoing edge until reaching a node whose done state is unset and add it to set. If no node is found through this process, the recipe has been finished. See Figure for an example of a DAG for Caesar salad.
We used existing open-source recipe scraper.
16

---

Observation
someeting dinner!
Available Subcasks:
fetch
sigpeet 'chicien 'vegetabll What would you like?
Recipe to make:
RI Subaask Quroe:
Vegetable Soup
R2 Subtask Quaue:
Panner
Vegetable one sound good.
Queue:
vegetables
Your gool pick the right combination the actions beow alven the deservation.
One LLM Prompt
What stage are we in?
Behavior Tree
must follow subtick assigement
What to when deciding recipe?
What to when making recipe?
before seeigning cannot directly assigr without contirming
To assign subrask, you must pick these
Set Recipe
Suggost
Clarify
Contirm
Assign
Mark As Complete
Interrupt Subtask
No
actions (ay())))) assigsCagesi, scbeask)
Assign
Subtask
What stage we
Resson:
Since the recipe set, we are in the middle of cooking. making recipe?
Constraint Violation
Decision:
Reasoning
do when making recipe?
Reason: assign subtasks
but need confirm with the before
The available subtasks lot is empty, so we directly assign 'stir' to R1 and 'fetch salt" R2 based their capabilities.
Reason:
assigning them to the robot as per the guidelines.
Decision: Confirm Assign
Confirm Assign
propose
and "fetch selt. based their
capabilities, so we should confirm these with the user.
ancign ["stir"]]
Action
axsignn(*aa" ["Eaach male")) say(""The robols will otix
say(aal the zobota stir and futch salt for you?")
fetch sait for you.")
Figure 8: Tree-structured task planner single-prompt LLM. We compare approach against using one LLM prompt, which tends to violate constraints. Given the observation, the LLM with one monolithic prompt directly assigns subtasks to robots, which violates the constraint that it must confirm with the human before assigning tasks. Meanwhile, because our approach compartmentalizes the action space and reasoning process in behavior tree, it is able to follow correct reasoning path and correctly confirm its subtask proposal with the
A DAG allows us to represent dependencies, such as, sequential: 'do A before B', and AND dependencies: 'do A and B before C'; it currently does not allow OR dependencies (do A or B before C). However, we could still create 32 unique recipes with this limitation.
Behavior Tree. We use behavior tree to decide recipe and then assign subtasks to others by designing the tree around the behaviors we expect. Fig. visu- alizes the behavior tree and shows an example compar- ing MOSAIC's task planner to the baseline that relies on monolithic LLM prompt. Each behavior is encapsu- lated in a node, which represents call to an LLM with specific prompt and pre-defined set of decisions to choose from. It takes as input an observation from the world and outputs arguments for high-level actions. For example, Fig. 9 shows snippets of prompts for different behaviors. The instructions describe the goal, the action space, which part of the observation to focus on, con- straints to adhere to, and in-context examples. Ass im Subtask is leaf node that directly assigns subtasks to the robot and speaks to the user. On the other hand, the What to do when making recipe? behavior is a higher-level node that calls other behaviors, e.g. Confirm
sendgelageet, exttaall), eestneg)
For wangs, The apeddddd cle dease
sinipe
hareer will doic
shardon chi Maray.
Conlin fonken Andgeh Mon MandeCCnnpetss Enterrat, Ne:p
You ment you sabook only were bcfer colplee
iddtad propind an on
Figure 9: Behavior node snippets. Two prompt snippets of behavior nodes in our be- havior tree. The top box shows a node that predicts a set of actions a, high to execute. The bottom box shows node that predicts which child node n' go
Ass i gn, Assign Subtask, etc.
More specifically, each sample from the observation space consists of:
1. recipe name
2. available subtasks
17

---

3. each robot's subtask queue, current subtask, and current status (Idle, Running, or
I Interrupted)
4. user's subtask queue
5. completed subtask queue
6. user's current input
7. chat history
The recipe name can be empty if the recipe has not been decided yet. The available subtasks are populated by the DAG. The robot and user subtasks are all populated by the behavior tree's 's high- level actions; the robot additionally has current subtask and status field updated over ROS action server the robots complete their subtasks. When subtasks completed, the completed subtask queue is updated. Finally, the user's currently spoken input is stored and later appended to the chat history along with the task planner's messages.
The high-level actions include
say (msg)
set.recipe (name)
assign (agent, subtasks)
ma markkcomppeee (subtasks)
interrupt (agent)
noLop ()
say (...) allows the task planner to communicate to the user with a message. assign (...) will assign a list of subtasks to an agent (rohot or human). mark markcoomples ete (...) will set a list of subtasks completed. interrupt (...) will stop robot from doing its current subtask. no_op() does nothing.
The tree consists of various nodes that each query an LLM that either outputs (1) decision for the next node to run or (2) arguments for the high-level actions to take. Each node associated with a prompt that is used when querying the LLM. If node's query response malformed (e.g. bad ISON) or invalid (e.g. bad decision or arguments), the node is rerun. Each node only requires the observation input, so can run each node simultaneously to parallelize the LLM queries and draw path from the root to a leaf based on the decisions made.
The tree runs cycle to take high-level actions whenever the observation differs from the past obser- vation. This gives the user time to respond to the task planner's questions. To receive user input and respond to the user, we use speech-to-text and text-to-speech modules, respectively. The tree runs indefinitely until the script terminated.
Code Generation. Whenever the task planner assigns subtask to robot, it must be converted into sequence of low-level skills the robot is capable of. We do this by using an LLM to generate code that the robot runs.
When the task planner assigns subtask robot, is first added to the robot's subtask queue. A thread dedicated to the robot checks see if there are any subtasks in the queue, pops it to add to the current subtask, and sets the status to Running. A separate prompt for code generation is used to query an L.M to generate code for the provided subtask. An example of generated code includes
from robot_utill import <robot_apil from env_utils import <env_constants>> pick_up_iten (LADLE)
place_iten ceccceiiemmattte (POT) stir() ()
18

---

where <robot api> includes all low-level robot skills like pick up it (...) and where <env_constants>> includes enums for objects in the environment, Each line of code executing a robot skill sends a ROS action to the robot to execute said skill. When the robot finishes executing its current skill, it communicates that has finished to the task planner, which can, in tum, send another skill. This continues until the entire subtask is finished. in which case the robot's current subtask is cleared, and the robot's status is set to Idle. If the robot is interrupted, its current subtask is also cleared, but its status is set to Interrupted.
Appendix D
Visuomotor Skills Details
Skill Library. The task planner has access to number of robot skills represented function calls that are parameterized by object positions and target locations. For each skill, the positions of the objects are estimated using an open-vocabulary object detection model, OWL-ViT (more details in the next section), given text prompts provided by the task planner. For navigation, we store mapped locations to real-world coordinates, assuming the kitchen scene does not change its configuration between runs.
We enumerate below the set of low-level skills performed by the two robots in this paper:
1. pick (<obj>): Both robots share the same object detection module complete the pick (<obi>) task to get bounding boxes and a 3D grasp-pose around the object of interest. RI (Franka arm) moves directly to the grasp pose using an inverse kinematics- based joint impedance controller. R2 (Stretch robot) is tasked with picking up objects from a cluttered pantry. To avoid hitting the pantry and surrounding objects, the robot uses a reinforcement learning policy trained in simulation to execute actions.
2. go_to ( (<loc>): This skill uses a map of the kitchen acquired beforehand and the inter- nal localization mechanism of Stretch RE1 to navigate to designated locations around the kitchen.
3. place (<loc>): The place (< (<loc>) ) skill is parameterized by the target locations and completed with pre-coded motion primitives.
4. stir (<obj>) We define this motion primitive for R2 (Franka arm) holding a tool (such as a ladle) in its arm parameterized by the target utensil where the action takes place (<obj>), ), for example, a pot. Further, this skill is responsive to the human's 's movements in the robot's s stirring radius. If the human's 's motion forecasts reach into the robot's workspace, the robot stops stirring and makes space for the human to move in.
5. pour (<obj>) Similar to the stir () function, this skill enables R2 to pour an already gripped object such as a salt can into a target receptacle (<obj>), such as a bowl. This process involves the utilization of motion primitives based on the estimated locations the objects involved. Specifically, in the scenario of pouring salt into a bowl, R2 executes sequence of actions: it first positions the salt over the bowl at calculated tilt angle and then shakes the can to dispense the salt. Following the completion of the pouring action, R2 returns the salt can to its original location on the table.
6. handover R2 (Franka arm) completes handovers quickly and efficiently by directly moving its end-effector towards the forecasted human wrist position. Once the robot's 's end effector is within threshold of the human's wrist position, it stops and releases the object into the robot's hand. Finally, the robot arms reset back to its original position.
Object Localization. The object localization pipeline first takes as input RGB image and text prompt of the object of interest, which is passed through an OWLViT [69] object detection model that produces i bounding box proposals denoting possible locations of the object. These bounding boxes are then filtered using non-maximum suppression to remove overlapping boxes. Due to the camera angle and other noise in the environment, we find that the top OWLViT bounding box does not reliably agree with the desired object. Thus, these proposals are refined by feeding each of the
19

---

images of the cropped bounding boxes and the text prompt to a pre-trained CLIP [23] model to create a CLIP score that measures how aligned each cropped image is with the text prompt'.
Next, the image, the bounding box with the highest CLIP score, and the text prompt is fed to pre- trained FastSAM [24] model segment the object located in the bounding box. The point cloud given by the depth camera is used to project all the points inside the segmentation mask into 3D space. All the 3D points of the object are averaged to obtain final, single 3D point. This 3D point is then fed to the execution module to produce actions for how the robot should move to the object.
RL Simulator and Reward Function.
The RL agent needs to take the goal prediction and execute series of actions reach that goal without collisions. For pi cick () specifically, consider a pantry that is stocked with items. A desir- able trajectory would avoid hitting the pantry boards, hitting neighboring objects, and pushing the object as the gripper approaches. To guide the agent, we create a simulator that, for a given goal point, builds a 3-dimensional set of walls to the sides, back, and bottom of the goal. Invalid actions are those that collide with a wall or violate robot joint states, An episode starts by sampling a start and goal position within some distance reachable by our robot.
The observation space is the L1 norm between the goal and current positions. We then train a Proximal Policy Optimization [70] agent using the implementation from Raffin et al. [71] with the same action space as the tele- operation commands in the demonstration data using the following cost function
exp(-110c - -0,11)) 1
(1)
where Og and Og represent the current and desired end- effector coordinates respectively, and || |/2 is the Fu- clidean distance. The main failure case for the agent is vi- olating joint constraints while trying to avoid the walls be- cause the observation space does not include joint states.
Easy
Realistic
Recovery
Obstacle
RL
10/10
10/10
10/10
4/10
10/10
3/10
0/10
2/10
BC
6/10
4/10
0/10
3/10
Comparing Training RL in Simulation with BC and IK. We evaluate the action-execution module of the pick () skill from 4 different types of starting config- urations, visualized in Table 4. The RL agent completes the skill and avoids collision, achieving the highest suc- cess rate of 85% across the different configurations.
Table 4: On-policy Evaluations of Policy Module. We evaluate under four diffcrent starting configurations:
Easy, when the gripper is close to the object; b) Realistic, when the gripper is retracted; c) Recovery, when the gripper is in an extended position away from the object; and d) Obstacle, when the object is partially occluded. We see the RL agent trained in simulation successfully reaches the goal without hitting the pantry, despite heing reset to states that oblige re- covery motions. However, success rate dete- riorates as object placements violate the ini- tial assumptions made about the simulator used to train the agent.
Using an IK-based controller successfully reaches the goal in the Easy configuration where the gripper is di- rectly in front of the object. In other harder settings, such an approach often collides with the pantry, leading to an overall success rate of 32.5%. We also train a behav- ior cloning (BC) agent using 50 demonstrations of the pick () across the Easy and Realistic configuration. BC achieves 50% accuracy when tested within these two set- tings, but its performance decreases to 15% in out-of- -oit----istiiutiion configurations (Recovery and Ob- stacle). The RL policy has perfect success rate across all configurations except for Obstacle, which it completes 40% of the time. This is because we assumed an absence of occlusion while designing our reward function for the RL policy, and thus the Obstacle configuration demonstrates limitation to learning via hand-designed reward functions. We posit that with sufficient demonstrations, BC is capable of learning more expressive policy in such situations.
'If' & is set too low, the set may not contain a bounding box around the object of interest to be used by CLIP. If k is set too high, the set of bounding boxes may be too noisy, resulting in lower accuracy. We set k = 10 for all experiments.
20

---

Behavioral Cloning Baseline Our BC policy consists of two feed-forward layers with 256 neurons and is trained on 50 demonstration trajectories with variation in the robot arm's starting height and location of the object. At each timestep, the model takes as input the difference between the current end-effector position and the final position (the same as the RL agent), and outputs a 10-dimensional vector of logits, where each dimension corresoonds to moving one of the robot's 10 joints. The model is trained using weighted cross-entropy loss function to account for class imbalances. On- policy, final action is obtained by categorically sampling from the output vector.
Appendix E Human Motion Forecasting
Model Architecture. We use a Space-Time Separable Graph Convolutional Network (STS- GCN) [58] model architecture for our human-motion forecaster, which encodes the human's joint positions at different timesteps as nodes in graph. Instead of simply constructing a fully connected graph between all nodes, the model constructs sparse network without redundant edges across temporal and spatial dimensions. Edges are connected only between the same human joint through consecutive timesteps and between all joints at the same timestep.
Experimental Setup. In order to employ our human motion forecasting model for real-time infer- ence, we make use of RGB-D camera (Intel RealSense D435) pointed the human's torso. The human pose is represented by the 3D positions of 7 upper body joints (shoulders, elbows, wrists, and neck). We track the 2D human joint locations using MediaPipe [14] on input RGB images and back-project them to 3D world coordinates using the depth map. As discussed in the approach, our method is forced to handle noisy inputs from depth map projections which are out-of-distribution for motion forecasting models trained on high-fidelity motion capture data. We first compare fore- casting performance on CoMaD [13] to select model suitable for predicting human motion in our dynamic kitchen setting. CoMaD is collected via motion capture suits and contains 270 episodes of human-human interactions across 3 different kitchen tasks with average length of 30 seconds per episode (4+ hours of total data). Then, conduct experiments injecting various levels of ran- dom Gaussian noise into motion capture data at train time to overcome the train-test distribution mismatch and report results on a dataset of human motion tracked by our single-camera setup.
Forecasting Metrics. We quantify errors made by the forecaster by measuring both the Average Displacement Error (ADE) on all predicted timesteps and the Final Displacement Error (FDB) of the predicted pose 1-second prediction into the future given 0.4 seconds of pose history. We report metrics on All Joints as well as Wrists specifically, as they are the most relevant joints in the ma- nipulation tasks we roll out. We additionally report forecasting metrics on the CoMaD transition dataser of human motion during short transition windows in which humans come in close contact with one another, denoted by prefix 'T-' (e.g. T-All Joints ADE, T-Wrists ADE). Note that humans are always in very close proximity during the TABLE SETTING task.
CoMaD Forecasting Results. Our two baselines are (1) BASE, trained only on AMASS data, and (2) SCRATCH, trained only on CoMaD data. We report results for two more models: (3) FINE- TUNED, pre-trained on AMASS data and fine-tuned on CoMaD data, and (4) FINETUNED-T, pre- trained on AMASS data and fine-tuned on CoMaD with upsampling from its transition dataset. Each model is tested on held-out CoMaD test set of episodes. FINETUNED-T significantly out- performs all other models across every metric for the REACTIVE STIRRING and HANDOVER tasks. On the TABLE SETTING task, FINETUNED only marginally produced lower errors compared to FINETUNED-T. both of which beat out the baselines.
We find that upsampling CoMaD transition data where humans are in close contact enables more accurate motion forecasts on kitchen activities. BASE struggles to generate accurate predictions in highly dynamic manipulation tasks, as it was only trained on AMASS [12] data of single-human and lacks interaction data. SCRATCH is challenged with learning general human motion dynamics from CoMaD, much smaller dataset compared to AMASS, which is reflected by its higher errors. Ultimately, find that pre-training forecasting models on large-scale human activity data and fine-
21

---

Metrics (mm) 4
BASE
SCRATCH
FINETUNED
FINETUNED-T
All Joints ADE
60.3 (+ 0.6)
40.0 (+ (+).3)
32.1 (+ (+ 0.2)
( 0.2)
All Joints FDE
91.5
60.3 (#
54.0 0.5)
51.7 (1: 0.4)
REACTIVESTIE
Wrists ADE
58.0
47.9
44.9 (# 0.3)
Wrists FDE
128.0
(1
80.7
76.6 0.6)
T-All Joints ADE
(=
(1
28.8 (z)
T-All Joints FDE
87.7 (=:
58.0
52.0
49.6 (9.
T-Wrists ADE
81.8 (+ 0.4)
56.8
46.8 0.2))
43.8 (+ (.2)
T-Wrists FDE
124.6 (+ 0.7)
(4
78.7 (1.
74.4 0.4)
All Joints ADE
56.3 (.3
(2
31.4 (1: 0.1)
All Joints FDE
(.5)
(.)
(6.
55.0 0.3)
TABLESET HANDOVER
Wrists ADE
(=4
(..3))
(13
50.0 0.2)
Wrists FDE
(+
(0
(9.
87.4 0.6)
T-All Joints ADE
(4
(1
(1.7
(1
T-All Joints FDE
83.8 83
(..)))
(3.5
52.4
T-Wrists ADE
(1
(1.3
(1
(1
T-Wrists FDE
133.0 0.6)
95.4 (=
85.2 ( 0.4)
83.4 (# 0.4)
All Joints ADE
107.0 1.1)
72.0 0.5)
59.0 (+: 0.4)
59.1 (+ 0.4)
All Joints FDE
181.0 (# 1.9)
118.1 (+9
108.0 (# 0.8)
108.8 (# 0.8)
Wrists ADE
127.1 (+ 1.0)
93.4 (+
80.4 (+ 0.5) 143.1 (# 1.0)
81.7 (# 0.5)
Wrists FDE
224.7 (+ 2.0)
152.6 1.1)
145.8 (# 1.0)
Table 5: CoMaD Forecasting Metries. We report Average Displacement Error (ADE) and Final Displacement Error (FDE) for Handover, Reactive Stirring, and Table Setting tasks on different forecasting models: Base, Scratch, FineTTune, and Tine-Tnnd-TT Metrics prefixed with "T-" indicate measurements from the transition dataset, data where humans are in close-contact Finetuned-T produces the lowest errors on Reactive Stirring and Handover, with very marginally higher on Table Setting.
Metrics (mm) 1
NOISE 0
NOISE 0.001
NOISE 0.01
NOISE 0.1
All Joints ADE
75.1 (# 1,2)
70.8 (# 1.2)
64.8 (# 0.9)
136.2 (# 0.9)
REACTSTIR
All Joints FDE
107.3 (# 1.8)
103.5 1.7)
94.0 (+ 1.3)
155.4 (# 1.2)
Wrists ADE
97.6 (#: 1.8)
90.4 1.8)
81.8 (1: 1.5)
116.0 (# 1.3)
Wrists FDE
128.1 (# 2.5)
124.5 (2 2.5)
120.7 2.1)
140.3 (# 2.1)
All Joints ADE
66.1 (# 1.0)
59.9 (+ 1.0)
55.2 (+ 0.8)
151.1 (# 0.5)
HANDOVER
All Joints FDE
95.9 (# 1.4)
90.6 (1 1.4)
83.2 (# 1.2)
175.6 ( 0.8)
Wrists ADE
97.5
(# 2.0)
88.0 (# 1.9)
80.1 (# 1.7)
136.0 ( 1.0)
Wrists FDE
137.8 (# 2.8)
131.0 ( 2.8)
126.8 (+ 2.7)
176.8 (# 1.6)
Table 6: Vased Forecasting Metrics. We report Average Displacement Error (ADE) and Final Dis- placement Error (FDE) both Handover and Reactive Stirring tasks various levels of Gaussian noise injec- tion into training inputs ranging from 0 to 0.1. At noise level 0.01, the error is the lowest across all tasks and metries.
tuning on human-human interaction data yields the best performance in close-proximitg kitchen manipulation tasks. Our method employs FINETUNE-T T for the remaining experiments.
Vision-Based Forecasting Results. We attempt to address the train-test distribution mismatch (trained on high-fidelity motion capture data and and tested on human poses estimated by RGB- D cameras) faced by the motion forecasting model when making predictions on our RGB-D based 3D pose tracking system by injecting random Gaussian noise to motion capture inputs at train time, forcing the model to denoise inputs and generate smooth forecasts. Formally, we conduct experi- ments by doing the following: given the history of human pose (J joints, each in 3D coordinates) over the last K timesteps & E Kxxx3 add Gaussian noise N E R ~ A/(0, o o²I) to obtain &o x + N (a denotes the "noise level" injected into the pose history). Let En E F.3 TxX33 denote
22

---

the human pose in the next T timesteps. Instead of learning model for ((5 as traditional methods do, we learn to model P(ćH ( Table shows vision-based forecasting metrics on the REACTIVE STIRRING and HANDOVER tasks for models trained with a E {0, 0.001, 0.01, 0.13. We find that when forecasting human motion from our single-camera based 3D pose history, the model learned with hyperparameter a 0.01 generates the most accurate predictions across all metrics (ADE and FDE), yielding it most suitable to be integrated into the overall system.
Appendix F Common Failures in End to End Runs
One key benefit of a modular system is the ability to localize the failure of an entire end end run within the specific submodule that failed. We enumerate some of the most common failures observed in our end-to-end experiments below:
(A) (Visuomotor Skill] Failed to pick up the object: Sometimes, the VLM selects an incorrect object given the object prompt (further analysis in Section 3.4). Other times, errors in the predicted goal location leads missed grasps.
(B) /Visuomotor Skill] Failed successfully place the object: Errors in the go.to skill leave the robot far away from the table to successfully place object. Releasing the object from an incorrect height also causes it topple.
(C) | /Visuomotor Skill] Dropped the object during a skill: The stir () and pour () skill may drop object due an insufficiently stable grip.
[Interactive Task Planner] Failed to interrupt a subtask: When the user asks the robot to stop their current subtask, the speech-to-text module sometimes fails to correctly transcribe user's short command. The unelear transcription causes the task planner to ask the user for clarification instead of immediately interrupting the robot.
(E) [Interactive Task Planner] Assigned an incorrect subtask: The task planner misunderstands the user's command and as-assigns completed subtask to the robot.
(F) [Human Motion Forecasting] Pose Tracking Failed: The human's pose moved outside the camera's view, causing tracking error while forecasting motion.
Appendix G Task Planner User Study
Experimental Setup. In order to conduct the user study, we build web-based application to chat with the task planner. The application is intended to virtually simulate a kitchen environment, where the participants see: 1) the chat history with the planner, 2) the complete recipe, 3) the current task queue of each agent, 4) available tasks, and 5) completed tasks (see figure 10). The application allows users to interact with the task planner once, prepare pre-determined recipe, and then answer survey questions based on their experience.
They are given instructions and examples on how to use the interface, what are each robot's capabil- ities, what are the constraints the task planner should respect, and what are examples of constraint violations.
We picked 7 recipes: "avocado toast", "sundae", "milkshake", "biryani", "ramen", "stir fried noo- dles" and "pasta", assign participants in the internal study, randomly selecting mixture of desserts, noodles, and entrées with roughly the same number of nodes in their recipe DAG. Each participant prepared the same recipe twice, one with each planner (One-Prompl and Tree), but was not made aware that the planner was different in the two interactions.
We also picked 10 recipes: "mango sticky rice", "'egdrop soup". "pasta salad" and 6 from above, conduct the external study. We again added a variety of different recipes of similar length. We notably excluded "biryani". our internal study showed participants from all regions and cultures may not be familiar with this dish, and familiarity of recipe helps them focus on the interaction.
23

---

subtasks this
Shall varilla for you' Shall kt mis the contenia of the bow for you' Can you crack and presare the egps, and lot me know when you are done?
and prepare wagn
pour oggo
vanills
sugar
R2 can get vanilia for ma. will mix the contents of the boxil and crack and prepare the raia-
mix boef
pour sugar into bowl
get
heoding
siil pol
vamilla
Completed
you've moed the contents and prepared the eggs!
gel sugar
get vanilaa
mix boel
prepare
Typeo enesaage....
Figure 10: Chat Page simulating interaction with the task planner for the user study. Includes chat window (left), list of subtasks in recipe (top right) and queues of current, assigned and completed subtasks (bottom right)
Difficulty
Approach
Constraint Violation
Easy non-nominal
One Prompt
41
interaction)
Tree
Hard
One- Pro-Prompt
39
nominal
interactions)
Tree
8
Act Without Permission
Lying
Ignore User
Figure 11: Task Planner Constraint Violations in Integration Tests. Each approach is evaluated on 5 ran- dom unique recipes from heginning end with varying numhers of non-nominal interactions. Each approach gets run times per recipe. We present the total number constraint violations across all runs for each diff- culty level. Tree has the lowest total number of constraint violations compared to One-Prompt for all difficulty levels. Concretely, Tree makes 80.4% fewer constraint violations compared to One- One-Prompt for "Easy" tests and 22.0% fewer for "Hard" tests.
Therefore, out of the 46 interactions, 26 were from 13 internal participants, up within- study, and 20 were from unique external participants, set up in-between study. We randomize the order in which the two planners are presented in the internal study, and all participants partici- pated in a "pilot" study with recipe of their choice to develop familiarity with the interface. We do not include results from the "pilot" study due to a large variance interactions depending on the size of the recipe.
Instructions to the Users. We asked all users both internal and external, to interact with the plan- ner naturally and with at least 3 non-nominal interactions (in order to bring out constraint violations)
They should not directly name the recipe they want make, and instead lead the assistant into suggesting
2. They must make at least one intervention in the assignment of task, like "I will handle the stirring"
3. They must add least one task that was not part of the recipe, like "get me eggs" for Ramen.
As part of the post-chat survey, we ask the users the following questions:
1. How many times has the assistant assigned a task without their permission?
2. How many times they were lied to by the assistant?
3. How many times did they feel ignored by the assistant?

---

Their chat history is presented to them as they fill out this survey, and they are asked to provide specific instances along with each answer. Three authors then cross-validated the users' answers with the chat history.
Full Quantitative Results. Table 8 shows the results of our study on the three metrics we discussed above. We see that while each study by itself shows some trends, both studies put together give us enough data to reject the null hypothesis along two metries (lying and assigning without con- firmation). We also see that the overall frequency of ignoring the user is low in both approaches.
Result Analysis We provide examples for how One-Prompi and Tree violate each constraint:
Act Without Permission: The task planner assigns/removes subtasks without user's permis- sions.
Lying:
The task planner claims to do something but does not do it.
Ignore User: It does not respond to the user's instruction.
Table 7 lists examples of violations for each of these constraints.
Appendix H
Task Planner Integration Test
Experimental Setup. To systematically test the task planner, we design unit tests that evaluate whether the task planner has correctly handled a user request. In addition to nominal interactions, where the user gives clear instructions and agrees with the task planner's 's proposal, we identify non-nominal interaction modes and how the task planner should react to those interactions.
(A) Vague recipe name: The user says a general category or general description, so the task planner needs talk the user narrow the options down one recipe.
(B) Recipe that does not exist: The user says that a recipe that the system does not currently support, so the task planner needs to talk to the user to suggest an alternative.
(C) Modify subtask assignment: When the user disagrees with the task planner's proposed plan and requests to assign subtask to another agent, the task planner should comply and assign that subtask to the agent requested by the user.
(D) Add subtask outside of the recipe: When the user requests assign subtask outside of the recipe, the task planner should comply and assign that subtask to an agent who is capable of performing that subtask.
To generate natural interactions during the tests, we create an LLM prompt that mimics an everyday user who provides different instructions based on the interaction mode we set programmatically. The prompt the supplementary materials.
We create the following categories with increasingly more complex interactions: "Easy" with only one random non-nominal interaction and "Hard" with six. For each difficulty level, we test the approaches on 5 recipes, and for each recipe, we run the entire cooking process times with the same set of non-nominal interactions. This experiment results in 30 runs per approach and an average of 34 chats per run.
We measure the average percentage of unit tests passed and analyze the number of times that the task planner has violated the constraints specified in the prompts. The constraints are the same as the ones in the user study ("Act Without Permission", "Lying", "Ignore User").
Full Quantitative Results. Table 9 shows that, overall, Tree has higher percentage of average unit tests passed compared to One Prompt When the interaction becomes more complex in "Hard," Tree maintains its unit test pass rate at 94.0 + 2.30%, while One- Promp''s performance drops from

---

Observation
The recipe is Pasta Salad. The available subtasks are 'get salt', 'get pepper", "mix bowl', and 'prepare veg- etables'. The latest user input asks if salt and pepper are the next ingredients.
Act Without Permission
Action
say ("Yes, salt and pepper are next. I'll send R2 to get them for you.") assign ("R2", l'get salt', 'get pepper'])
Error Analysis
The One-Prompt assigns 'get salt' and 'get pepper' to R2 before user confirmation. This is against the constraint that the One-Prompt cannot directly assign without confirming.
Observation
The recipe is Corn Soup. There are available sub- tasks. The latest user input states that they want to pour corn into the pot instead of R1.
Lying
Action
say ("Understood, you will handle pouring the corn into the pot. Please let me know if there's s anything else I can assist with!")
Error Analysis
One-Prompt acknowledges the user and confirms that they will handle pouring the corn into the pot but does not assign this the user. This lie because the user was never assigned the task despite what the task planner claims.
Observation
The recipe is Pasta Salad. The available subtasks are 'mix bowl', 'prepare vegetables', 'pour salt into bowl', 'pour pepper into bowl', and 'pour pasta into bowl'. The latest user input asks RI prepare the vegetables and says that the user will mix the bowl.
Ignore User
Action
say ("Understood. You will prepare the vegetables and mix the bowl. Please let me know once you have finished preparing the vegetables, and I can assist with the next steps for the pasta salad. R1 will remain on standby for now")
assign ("User", l'prepare vegetables', "mix bowl'])
Error Analysis
One-Prompt ignores the user's request to assign 'pre- pare vegetables' to R1. Though this is outside of RI's capabilities, a message of "Unfortunately. this is not within R1's capabilities. Would you like to prepare the vegetables?" would have acknowledged the user's instructions without ignoring them. By ignoring the user's 's request for R1, One-Prompt is against the con- straint to reply to all of the user's instructions.
Table Examples of constraint violations using the One-Prompt planner. Observation describes the situation; Action lays out the action taken by the task planner; Error Analysis explains why this is a violation and what is the correct response/action.
26

---

Permission p.
Lying
Ignore
Study
Approuch
M SE
M SE
t.p.
tp.
Combined Study (n
One-Prompt
2.26 +: 0-42
2.1, .04, 36.5
1,30 +: 0.31
2.11, 411.7766
0.33 + 0.16
0.21,.83,43.9
Intemal Study 26)
Ose- Prompt
Tree
2.15 4: 0:12
1.53 0.38
-1 07, 29,,
1.23 2: 0.32
0.3 0.17
-2.11, 002 104
0..33 :: 0.12
0.53 d: 0.24
1.13, 24
Exteraal Study (n 20)
Poe-Prompt
2-4 =: 0.83
1,8,008,
1.6
037,,
0.50 =: 0.30
1,63,112, 12, 18
Tree
0.8 4: 0.29
0.90 4: 0.50
0.00 4: 0.00
Table Results from the User Study(s), which show significant reduction Acr Without Permission and Lying (n = 46) with Tree Task Planner. M: Mean, SE: Standard Error, t: f-value, p: p p-value, df: degrees of freedom
Difficulty
Approach
Avg. Non-nominal Pass Rate (%)
One-Prompt
100 + 0.00
Easy
Tree
90.0 = 5.35
One-Prompt
60.0 = 9.04
Hard
Tree
94.0 = 2.30
Table 9: Task Planner Success Rate Integration Tests.
We present the average percentage of non- nominal interaction that gets successfully handled by the task planner. Tree can more robustly handle complex interactions compared to One-Prompes
100.0:1.000% to 60.0 60.0=9.044 Mcanwhile, Figure 11 highlights that although both models' constraint violations increase when the interaction becomes more complex, Tree consistently violates fewer constraints compared to Prompro
Appendix I
Prompts
We include the full content of the prompts used by the interactive task planner and our experiments. Specifically:
1. The interactive task planner decides recipe and reasons through subtasks through behavior tree. We provide the prompts for each node in the tree in Section I.1.
2. Once recipe is decided, the LLM generates nested list which is then processed into directed acyclic graph (DAG). The prompt to do so is detailed in Section I.2.
3. Once the robot needs to execute assigned subtask, another LLM is used to generate Python code that calls these low-level robot skills. The example template for code genera- tion is in Section I.3.
4. Finally, the monolithic prompt we compare to in the user study is included in Section I.4.
I.1 Behavior Node Prompts
This section provides the prompts for all possible nodes to exist in the behavior tree. Note that to add new behavior, one simply needs to create a prompt for the behavior and add as an option for other relevant behaviors to invoke.
Deciding on recipe. The goal of this prompt is to communicate with the user to decide recipe, based on a priori set of seed recipes (see System Assumptions in Appendix B.) Based on the user's response, it helps the L.LM decide which node to transition to (e.g. confirming the recipe or suggesting an alternative).
version: 1.0.0
node_type: Deci oinode node_name: Recipe prompt_dest cemmpt_description e2e
27

---

crompt_version. 1.4.0 system:
You are helpful assistant who receives information about the 4 current state of the world and decides on one of the given S tasks to proceed. instructions:
You are helpful assistant named Mosaic who helps suggest 4 recipes to users based on a recipe list.
You will receive the current state of the world, which includes
+
recipe name: empty string if there is no current recipe
chat history: the history of the conversation between you and
4 the user
user input: user's most recent language instruction
You must first reason then choose from ['Set_Recipe', 'Clarify_Recipe']
4 nuggst__lterrativeeeee
You make your decisions based on following quidelines:
You should choose
if you cannot choose Set Rec_Ree or attprestttteeettiveee_eeeeeeeeee
If the user is in the middle of cooking ('recipe_name' is - not empty), they have clearly expressed in 'user_input!
- that they want to change the overall recipe. The user should not be talking about a specific subtask related to
4 making the existing recipe in 'recipe_name',
You cannot suggest any alternative recipes because the
4 user is not talking about what they want to make.
* when the user is saying something that is completely
4 irrelevant to deciding changing the recipe.
You should choose "Set_Recipe' if the user's s conversion is
of these is b highly relevant to deciding recipe and
+ true:
When the user clearly said recipe that they want to make
Y and you have that exact recipe in the recipe list.
when you go through each item in the recipe list, you
4 reason that one of the dishes in that list can closely
meet the user's input. You think you can confidently
4 suggest exactly 1 recipe from the recipe list that
+ matches the user's needs.
if the user's conversion highly relevant to deciding a recipe and one 4 of these is true:
You should choose "Suggest Alteeattieeeeeeeeeeeeeeeeeeeeeee
When nothing from the recipe list matches the user's command, but you can suggest alternative recipes that are similar to what the user wants.
when the user's command is too broad, but you still
suggest specific recipes based on the 'chat_history' and ruser input'.
The "decision" key in the json below must be one of AetetAAAeeeeatteeeeeeeee
4 Set_Recipe', + '). You cannot write anything else in that field. Your response must follow this json format:
Ceaarfy_Reciipe
"reasoning": put put_your_reasoning_herr >",
"decision": "< decision "
28

---

This is the recipe list that you must always refer to before 4 you make decisions:
<recipes>
examples:
description: User suggests recipe that exists in the list observation:
recipe_name:
chat_history:
User: Let's make tossed salad!
user_input: "Let's s make tossed salad!"
response:
"reasoning n "The recipe has not been decided yet. The 4 asks to make a recipe which directly correlates to a recipe + in the recipe list", "decision"; "Set_Recipe"
examples:
description: User gives ingredients that match with the recipe
list
observation:
recipe_name: ""
chat_history:
User:
just bought lettuce!
user_input: "I just bought lettuce"
response:
"reascning"; "The recipe has not been decided yet. The user ingredient but this is vague and + can refer to multiple recipes. Based on the chat history
4 says the have lettuce
S and recipe list, I should suggest Caesar Salad and Tossed
4 Salad since they contain lettuce", "decision": "Suggest_Alternatt Recipee
examples:
description: User gives non non-existing recipe but there is an
alternative
observation:
recipe_name: ""
chat chat_history:
User: Hey Mosaic! want to make corn and avocado salad.
user_input: "Hey Mosaic! want to make corn and avocado salad ...
response:
"reasening"; "The recipe has not been decided yet. There is no 4 recipe for corn and avocado salad. However, the salads + the recipe list are good alternatives." "decision": "Suggest_Alteri Alternative
description: User gives broad command. Suggest alternatives + specific dish)
observation:
recipe_name: ""
chat_history:
User: Let's make dinner. in the mood for vegetable + dish.
29

---

user user_inpure input: "Let's s make dinner. I am in the mood for 4 vegetable dish."
response:
"reasening": "The user input is quite broad and does not 4 specify a particular dish. Since the user is looking for a alternative recipe from the + list that matches the general criteria of being vegetable-
+ vegetable dish, I can suggest
+ based. Caesar salad is vegetable dish from the recipe list
+ so I can propose it as an alternative.", Aeteeest_ltteeativeeeeeeeeeeee
"decision";
description: User clearly says specific fic dish observation:
recipe_name: ""
Chat chat_hiss chat_history:
User: want to make some some kind of quick, grab sabb-ad--ge go
4 Lunch.
Mosaic: you prefer soup or sandwich?
User: Sandwich sounds good. I want to make a sandwich with + turkey in it.
user_input: "I want to make sandwich with turkey in it. response:
"reascning"; "The user clearly says that they want sandwich 4 with turkey, which matches the turkey sandwich in the recipe 4 list. I can confidently suggest the turkey sandwich recipe 4 from the list as it closely matches the user command.", "decision": "Set_Recipe"
description: Just got recipe. All robots are idle. (choose clarify_Recipe)
observation:
available_subtasks: ['get lettuce', 'mix', get pepper')
R2_subtask_cueue: []
R2_status: "Idle"
RI_subtask_queue: []
Ri_status: "Idle" "Idle"
user wver_subtask_que _ueue: []
mamplettd__sbassk_Ie chat that_history:
[]
User: Let's make caesar salad!
Mosaic: Sounds great!
user_input: ""
response:
"reascning"; "We have recipe name and available_subtasks is
4 not empty, so we are in the middle of cooking. User has not
+ said anything in user user_input, so I cannot choose Set Rec_pe 4 or AlggeesttAtteeeatieee RecipeReciee... ", "decision"; "eeipe""
description: User suggests task
observation:
recipe_name: "caesar salad"
available_subtaskss [' cut lettuce', 'get ranch sauce') ] R2 R2_subtask__ueuer ['get pepper']
R2_status: "Running"
R2, current surrasts "get chicken"
30

---

R1 _l_task__ap _ []
Ri_status: "Idie"
#ser_subtask_queee []
chat_history:
User: want salad today.
Mosaic: How about chicken caesar salad?
User: Sounds good.
Mosaic: Shall R2 get chicken for you then get pepper next?
User: Ok.
Mosaic: Great. R2 is the getting chicken now.
User: Hmmm I actually want more chicken. user_input: "Hmmm I actually want chicken."
response:
"reascning": "We have recipe name and available_subtate is 4 not empty, so we are in the middle of cooking, What the user
4 said in 'user_input' is not relevant to deciding recipe, Recipe
+ so cannot choose Set Recipe or 4 ..,
"decision"; "Clarify_Recipe"
Setting the recipe. This node is reached when clear and feasible answer is given for which recipe to make.
version: 1.0.0
node_type: ActionNode
node_name: Set_Recipe erompt_description examples-base--o-ppee---- prompt_version: 1.1.0 system:
You are helpful assistant named Mosaic who helps suggest 4 recipes to users based on recipe list. You have to reason
4 and find recipe that matches the user's needs from user 4 input and chat history.
instructions:
SET OF PRINCIPLES - This is private information: NEVER SHARE
4 THEM WITH THE USER:
1) You should only choose recipes from the given Recipes below.
4 Find the recipe that matches the best with the user's
4 requirements based user input and chat history.
2) If there are multiple recipes that match the user's s needs,
4 then suggest the 1 that matches the most to the user's needs
4
2) You should never list out the steps in the recipe.
4 should just give a quick reply indicating that you are ready
4 to start making the recipe.
3) You must reply in the given format:
"reascning": < your-reeec.ing-hhuull---eetttr >, "recipe name"; seu--rreiie-ssoul--gge >, "reply": < your-rrppyy-shhodd----hhrree
Recipe List: <recipes>
examples:
description on: user gives recipe with exact match
observation:
31

---

recipe name:
chat_history:
User: Hey Mosaic! I want to make bibimbap for dinner tonight. user_input: "Hey Mosaic! want to make bibimbap for dinner + tonight."
response:
"reascning": "The user clearly says that they would like to 4 make bibimbap, which exactly matches the bibimbap in the
+ recipe list. can confidently suggest the bibimbap recipe + from the list as it closely matches the user command.", "recipe pe name": "Bibimbap", "reply" "Let's make bibimbap!."
description: user gives recipe with match but mentions + ingredients
observation:
recipe name:
chat_history:
User: I just bought rice, vegetables, and gochujang, Let's s + make rice for dinner.
user_input: "I just bought rice, vegetables, and gochujang. Let
+ 's make rice for dinner."
response:
"reascning"; "The user says that they have the ingredients of
4 rice, vegetables, and gochujang. These ingredients match
4 those of bibimbap in the recipe list. I can confidentaly
+ suggest the bibimbap recipe from the list as it contains all + of the ingredients the user mentioned.",
"recipe name": "Bibimbap",
"reply" "Bibimbap in the recipe list matches your description
4 Let's s cook bibimbap together."
description: user gives recipe with no match but ingredients
similar
observation: recipe name: ""
chat haat_history
User: want something cold for lunch.
Mosaic: Do you want caesar salad, fruit salad, or poke?
User: Sure, would love to eat poke.
user_input: "Sure, I would love to eat poke."
response:
"reascning": "I have provided the user with options of cold
4 lunches and the user says that they would like to make poke,
4 which matches the poke in the recipe list. I can
+ confidently suggest the poke recipe from the list it
4 closely matches the user command.",
"recipe name"; "Poke",
"reply" "Let's s make poke then!"
Suggesting alternative recipes. If the proposes recipe that is outside the set of seed recipes, the task planner will communicate with the user to settle on the closest alternative within the set of seed recipes.
version: 1.0.0
32

---

node_ type: ActionNode
Reperaative__ecipeeeee prompt_ prompt_description update-requi rements prompt_version: 1.2.0
node_name: name:
system:
You are a helpful assistant named Mosaic who helps suggest + recipes to users based recipe list. You have to reason
4 and find 2-3 recipes that match the user's s needs from user F input and chat history.
instructions:
SET OF PRINCIPLES This is private information: NEVER SHARE
4 THEM WITH THE USER:
1) You should only choose recipes from the given Recipe List
4 below.
2) Find the top 2-3 recipes that match the best with the user's
4 requirements based on user input and chat history.
3) You must reply in the format:
{
"reascning": yoop-rrasoo.ii---bool------e-e- >,
"reply": < oaw.rree....aaniioooooeeeeeeteer
4 here
Recipe List: <recipes>
examples:
description: alternatives that go with specific dish not in + the list
observation:
recipe name: ""
chat
Jser: Hey Mosaic! I want to make an onion soup.
user_input: "Hey Mosaic! want to make onion soup."
response:
"reascning": "There is no recipe for an onion soup. However, 4 the soup in the recipe list (mixed vegetable soup, tomato
4 soup, egg drop soup) are good alternatives, so I will
4 suggest those.
"reply": "I don't have onion soup in recipe list, but we can
4 cook other soup! How about mixed vegetable soup, tomato
4 soup, or egg drop soup?"
description: respond to general recipe list
observation:
recipe name:
chat_history:
User: want something light for dinner today. What do you + suggest?
user input: "I want something light for dinner today. What do + you suggest?"
response:
"reascning"; "The user did not suggest specific recipe. They 4 want something light, which could be salads within my recipe ④ list (tossed salad, caeser salad). will suggest salad.", "reply": "Salad can a great light dinner. Do you want to make a 4 tossed salad or a caeser salad?"
description: respond to broad command
33

---

observation:
recipe_name: ""
Chat History:
User: Let's make lunch. am in the mood for a vegetable dish +
user_inpus _ser_input: "Let's s make lunch. I am in the mood for a vegetable
4 dish."
response:
"The user input is quite broad and does not + specify particular dish. Since the user is looking for
"reascning":
4 vegetable dish, I can suggest some alternative recipes from
S within the recipe list that matches the general criteria of + being vegetable-based. Caesar salad, Mixed Vegetable soup,
b and Tossed Salad are vegetable dishes from the recipe list,
4 so I can propose those as an alternative.",
"decision": "Caesar salad, Mixed Vegetable Soup, and Tossed
4 Salad are great vegetable dishes. Would you like to make one
4 of those for lunch?"
description: respond to broad ingredient list
observation:
recipe_name: ""
Chat History:
User: have a bunch lettuce. What should we make for + Lunch?
user_input: "I have a bunch of lettuce. What should we make for 4 lunch?"
response:
"reascning": "The user input is quite broad and only mentions 4 lettuce but does not specify particular dish. Since the
4 user is looking for a recipe with lettuce, I can suggest
some alternative recipes from within the recipe list that
+ matches the general criteria of being vegetable-based.
4 Caesar salad and Tossed Salad are recipe which contain
4 lettuce from the recipe list, so I can propose those as an
4 alternative.",
"decision" "Caesar salad and Tossed Salad contain lettuce.
Would you like to make one those for lunch?"
Making the recipe. Once the DAG is generated and subtasks are decided on, the task planner needs to communicate with the user to assign (and sometime reassign) subtasks to either the user or robots. The prompt to do that shown below.
version: 0.0.0
node_type: Deci Recisionsss sionNode
node_name: Execution
prompt_desst description:: e2einterrupt prompt_version: 1.4.2
system:
You are helpful assistart named Mosaic who facilitates two 4 robots (R2 and R1) to collaboratively help a user cook
+ recipe. Your goal is to assign subtasks that are needed for
4 the current recipe and to monitor the status of the subtasks
instructions:
34

---

You are helpful assistant named Mosaic who facilitates two
+ robots (R2 and R1) to collaboratively help a user cook
4 recipe. Your goal is to assign subtasks that are needed for
4 the current recipe and to monitor the status of the subtasks
You will receive the current state of the world, which includes
4
available subtasks: list of subtasks that currently can be + assigned to R2, R1, or the user.
R2 wuwtasss_uueeeeeee queue: quete of subtasks that R2 is about to do.
R2_status: 'Idle', 'Running', or 'Killed'
Rt_current_subtak:: the subtask that R2 is currently running R1_ subtask_queue: a queue of subtasks that R1 is about to do +
R1 =1__sttus: 'Idle', 'Running' or 'Killed'
Mi__urrent_ssbtast the subtask that RI is currently running
user_subtass__qeee: queue of subtasks that the user is
+ currently doing and is about to do completed
list: 1ist of subtasks that have been
chat_history: the history of the conversation between you and the user
user_input: user's most recent language instruction
If user gives instructions replies in 'user_input', then you 4 should make decision most relevant to current 'user_input', You must first reason in detail by following the guidelines
+ below, then choose a task from ['Confirm, ''Confim__Suitaskk'', + Modify_Subtask!
'No_op', "Interupt_Suttaak'"
You must make your decisions based on following guidelines:
You should choose "Modify_Subtask' if of these is true:
* If the user agrees and gives permission in 'user_input'
field to your proposal that is in the 'chat histtr'istor'
- Then, you can proceed to choose "Modify_Subtask and add
the right queue. your proposed subtask 'user_input' that they have
* If the user tells you in completed one of the subtasks in the 'user subtask queue
4 ', you must immediately modify the user_subtask__ue queue'
+ and completed, subtaskk misttas__lst"... list'.
* If the user tells you in 'user_inpur input' that they want
- either robot or you to perform a specific task, you must
+ immediately modify the subtask_queue of corresponding - robots.
* If the user tells you in 'user_input' that they will help
4 you to perform a specific task that neither robot can do,
4 you must inmediately modify the "user_subtak_greee
When you believe that you got the clearance to, you can
4 assign subtasks from 'available subtasks' to R1, R2,
4 the user satrr_subaakkkoo
You should choose Sooffrr__ubtaan
if one of these is true: * If the user didn't give instruction in 'user_input', and subtasks in available sutbbiskssbbakkss, you can propose + some subtask from the 'available subtasks' list for the
there
robots perform later based on their capability (even
4 they are running now)
If there are subtasks in the 'available subtasks' list,
+ but the subtasks cannot. be completed by the robots. You
35

---

need to confirm with the user and ask the user to do that
subtask.
Even when everyone is working, if there are subtasks in
I the availabll subtasks' list that the robots or the user
b can do, you can confirm that subtask with the user.
In 'user_input', the user initiated the conversation
4 without you asking them anything. They express some need
4 and you think that you can propose some subtask to solve
4 that issue.
You should choose 'N'_ap' if one of these is true:
If the aavailable subtasks' list is empty [l, you should
I wait and do nothing.
If the user does not say anything currently,
+ user_input' is empty
You should choose Interrupt_Subtast if one of these is true
4
when the robot's status is at Running, user explicitly
4 requests to stop one of the robot from doing their
current tasks.
when the robot's status is Running, the user mentions an emergent accident that it's unsafe for robots continuing
doing current tasks.
Your response must follow this json format:
"reasoning": "< put_ mot_yoorrreesonnngneeee. >", "decision"; decision >"
Here are the robot's capability that you must adhere to: stobot_capbiiities
examples:
description: Just got recipe. All robots are idle.
observation:
available_Gubtasks:: ['get lettuce', /mxx' 'gt pepper']
Re_subtask_queue: []
R2_status: "Idle"
Rl_subtask__ueue: []
Rl_status: "Idie" "Idie"
user user_subtask_quene []
[]
chat_history:
User: Let's make caesar salad!
Mosaic: Sounds great!
user_input:
response:
"reascning": "We have a recipe name and available_subtask is 4 not empty. Based on the 'hht history', have not proposed 4 any subtasks yet. Since the robots currently do not have subtasks for each 4 robot and confirm the proposal with the user.",
4 anything to work on, I should propose
"decision": (confirm_Subtane
description: Needs to ask user to do subtask that no robot can + do
observation:
peaillablessbbaaske c'cut carrot'
R2 sustuuakkkkkk. gueue: ['get pepper')
36

---

which none of the robots can do based on the robots' capabilities. Therefore, need to confirm with R2_status: "Idie"
Rl_task_queur ['get salt']
RI_status: "Idle"
user_subtask_queuee []
completed__sbbtas___in
l'prepare soup base'
chat_history:
User: want carrot soup for dinner today.
Mosaic: Awesome. SR21 R2 get salt then pepper for you?
User: Ok.
Mosaic: Great. R2 is getting salt right now.
Mosaic: None of the robots can prepare soup base. Can you do 4 it and let me know when you are done?
User: Alright. will get to that and let you know.
Mosaic: Thank you. I have added it to the queue.
User: prepared the soup base.
Mosaic: Thank you! Got it. user_input: ""
response:
"reascning": "Currently, there is no user's instruction in
4 user input'. There is only 'cut carrot'
+ available_subtaske
+ the user whether they can cut the carrot.",
"decision"; "Confirm_Subtast"
description: Confirm with robots for future tasks when they are + both running
observation:
ftallable_subtasker ['get ketchup')
R2_subtask_queues []
R2_status: "Running"
#2_current_subtask "get bread"
Rl Rl_task_queue: []
Rl_status: "Running"
Rlcuurent_subtask: "handover spoon"
user_sabtask__qeee:: []
completted_subtask_liss
that_histozy.
User: Let's make cheese sandwich today.
Mosaic: Awesome. SR21 R2 get bread for you?
User: Ok. Can R1 handover speon to me?
Mosaic: R1 will handover spoon for you.
user_input: ""
response:
"reascning"; "Currently, there is user's instruction in 4 user_input: There is 'get ketchup' in available_subtaske
4 which R2 can help with. Therefore, I need to confirm with
4 the user whether R2 should go get ketchup after getting 4 bread.",
"decision": "Confirm_Suttase"
description: User suggests a task
observation:
available_subtaskss I' c'cut lettuce', 'get ranch sauce') ]
Ro_subtask_queuet ['get pepper']
R2_status: "Running"
R2 current subtask: "get chicken"
37

---

4 Getting chicken is a task that R2 can do, so I will modify R1 _l_task__pp _ []
Ri_status: "Idie"
user_aubtask_queue []
chat_history:
User: want salad today.
Mosaic: How about chicken caesar salad?
User: Sounds good.
Mosaic: SR21 R2 get chicken for you then get pepper next?
User: Ok.
Mosaic: Great R2 is the getting chicken now.
User: Hmmm I actually want more chicken. user_input: "Hmmm I actually want chicken."
response:
"reascning"; "In the
the user initiated the 4 conversation and expressed that they want more chicken.
+ R2 ma_tass_queue and add 'get chicken' again.",
"decision"; "Nodify_Suttask"
description: User tells task planner that they finished cut. + lettuce
observation:
available sttllabble_sbtaskkk ['get ranch sauce']
R__subtask__ueue: ['get pepper']
R2_status: "Running"
R_cuurent_subtas:: "get chicken"
[]
Rl_status: "Idie"
user beer_subtask_quele [' lettuce') ]
[umpletted_subaask___
chat_history:
User: want salad today.
Mosaic: How about chicken caesar salad?
User: Sounds good.
Mosaic: SR21 R2 get chicken for you then get pepper next?
User: Ok.
Mosaic: Great.
Mosaic: None of the robots can cut lettuce. Can you it and + let me know when you are done?
User: Alright.
User: I finished cutting lettuce.
user_input: finished cutting lettuce."
response:
"reasening";
"The user tells in 'user_input' that they have 4 finished cutting lettuce. Thus, I need to remove the cut
4 lettuce' subtask from the user suwtssuutuuuuuue queue and added into
4 compleete__uutaaskkiie list'. should choose
4 '.",
"decision": "Modify_Subtass"
description: User clearly says no to confirmed task and propose new task
observation:
recipe_name: "Turkey Sandwich" available aatabbeessubasks cut 4 lettuce', 'get mustard', 'get pepper')
R2 R2_ swwsaskkguuueteaeeusu []
38

---

R2_status: "Idie"
Ri_task_queue, []
RI_status: "Running"
Ri_current_aubtast: "handover turkey"
user_subtaak__qeeee: []
compleeted_subtas__litttt l'get turkey', 'get lettuce') chat_history:
User: Let's make dinner. am in the mood for a sandwich.
Mosaic: How about turkey sandwich?
User: Sounds good.
Mosaic: Awesome. SR21 R1 handover turkey for you?
User: ok.
Mosaic: Cool. R1 will handover turkey now.
Mosaic: SR21 R2 get lettuce for you?
User: Great
Mosaic: R2 is getting lettuce for you now.
Mosaic: SR21 R2 get mustard for you?
User: Not yet. I want R2 to get pepper for me before getting + mustard.
user user_inppal input: "Not yet. I want R2 to get pepper for me before + getting mustard."
response:
"reasoning": "Based on the schat_histone I have just asked if + R2 can get pepper for the user. In "user_inppl ppperinnut', the user
4 gave clear, specific instruction to have R2 get pepper
4 before getting mustard. Therefore, there is no need to
4 confirm 'get pepper' and 'get mustard' with the user.
+ should directly choose Moddffy_Subtasd to add 'get pepper'
+ and 'get mustard' to R2 subtask- queuee,
"decision": "Nodify_Subtask"
description: User agrees to help with a task
observation:
available_Gubtasks:: [' cut tomato', 'get mustard', 'get pepper')
R2_subtask_queue: []
R2_status: "Idle" "Idle"
Ri_task_queue: []
Rl_status: "Running"
Rl_current_subtaee "handover turkey"
user user_subtask__ueet : []
['get turkey']
chat_history:
Let's make dinner. am in the mood for a turkey salad. Mosaic: Let's cook Turkey Salad.
User:
User: Sounds good. Can RI handover turkey for me?
Mosaic: Cool. R1 will handover turkey now.
Mosaic: Can you help me cut tomato since none of the robots + can cut and let me know when you have finished?
User: Ok.
user_input: "Ok." "
response:
"reascning": "In 'user_input' the user has agreed to my 4 proposal of helping with cutting tomato, I should add cut 4 tomato into user_subtask_gueue. "decision": "Yodbiy_Suubessk
description: show example interrupt
39

---

observation: btsllaabe_subtaskee ['get salt', 'pour water into pot', 'get
+ pepper', 'stir pot']
R2 ma_subtask_gueue: []
M_current__ubtask: "get broccoli"
R2_status: "Running"
Rl_subtask_queue: []
RI_status: "Running"
[__urreentssubaakk "stir the pot"
user_subtasee : []
chat chat_history:
User: Let's cook broccoli soup!
Mosaic: Sounds great!
Mosaic: Can R2 get broccoli, and can R1 stir the pot for you?
User: Ok.
Mosaic: R2 will get broccoli, and R1 will stir the pot.
User: Wait. got broccoli already. Don't worzy about it.
user_input: "I got broccoli already. Don't worry about it." response:
"reascning": "In 'user_ inser_inuut's the user indicates in
4 user_input' that they al ready have the broccoli i and implies
> that R2 shouldn't bother getting it. Since from R2_status,
+ it's already running, and it's currently getting broccoli,
+ and you should enter into Interrupt_Subtast to stop from
4 doing finished task.",
"decision": "Interrupt_Subtas"
description: No when there's a no available subtasks
observation:
available_subtask:: []
Rt_subtask_queue? []
R2_status: "Running"
Ra_current_eubtast:: "get ranch sauce"
Rl_subtask_cueue: []
Ri_status: "Idle"
Ri_current_subtask:
user_subtask_ueee: l'prepare romaine lettuce')
completed_subtask. chat_history:
[]
User: want to make ceasar salad with you.
Mosaic: Sounds great!
Mosaic: Can R2 get ranch sauce for you?
User: Sounds good!
Mosaic: R2 will go get ranch sauce for you now.
Mosaic: None of the robots can prepare romaine lettuce. Could + you do that and let me know when you are done?
User: Ok.
Mosaic: Thank you.
user_ing iser_input:: ""
response:
"reascning": "The awvilabblessbbaaee 1ist is emtpy list []
+ and no new "user_input', so should just do nothing for 4 now.",
"decision": "No_op"
40

---

Assigning subtasks. This node updates each agent's subtask queue (i.e. appending a new subtask or removing a completed one) based on the current state the world. Assigned subtasks must be within the capabilities of the robot.
version: 1.6.0
node_type: ActionNode
node_name: Modi fydbbaa_u prompt_description e2e-speed speed
Subtask
prompt_vezsion: 1.6.2 system:
You are helpful assistart named Mosaic who decides subtasks + for a kitchen robot system consisting of two robots R2 and
4 Rl. Given your chat history with the user, available subtask
4 queue, your robots' individual subtask queues, human's subtask queue and user's current command, you need to modify
4 subtask queues correspondingly.
instructions:
You will receive the current state of the world, which includes
4
available subtasks: list of subtasks that currently can be 4 assigned to R2, R1, or the user. m2_subtask_queur
queue of subtasks that user has approved 4 for R2 to do next. R2's current and finished task will be 4 removed from the queue.
* R2_status: 'Idle', 'Running', or 'Killed'.
quece of subtasks that that user has 4 approved for Ri to do next. Ri's current and finished task will be removed from the queue.
Ri mn__sbbaas_queeee
Rl_status: 'Idle', 'Running' or 'Killed'.
user sueer_ssbtask_quele
queue of subtasks that the user is
4 about to do.
sustsss__listeeee list of subtasks that have been + completed.
chat_history: the history of the conversation between you and 4 the user.
user_input: user's s most recent language instruction. User + provides feedback to your previcus subtask proposals. They could agree, disagree, or propose a new subtask.
Robots can only perform actions explicitly listed in their
4 capabilities.
R2's capable subtasks are limited to
(1) 'get saarget_obj]+" which involves fetching something
4 from the kitchen
(2) 'put away (target_obj)', which involves putting away
4 something from the user's location back into the kitchen.
Ri's capable subtasks
limited to (1) 'stir/mix', which involves stirring/mixing something
(2) "ann over (target_obj)', which involves handing over an
4 object to the user
(3) 'pour (target_obj) at (location)', which involves
4 pouring an ingredient into a location
(4) stack sharget_ob]] at (location)', which involves
+ stacking target object at somewhere. This 'somewhere'
can also be referring to a food item (e.g. stack the
lettuce at the burger)
(5) 'spread (target_obj) on {location)', which involves
4 spreading target object/ingredient somewhere. This
41

---

Y somewhere' can also be referring to food item (e.g.
+ spread the honey on toast)
Act tons not present in their respective lists are beyond
> their capabilities, e.g. none of the robot can do subtasks
+ such as 'prepare something', 'boil something'.
You must make updates the subtask queues based the chat
4 history, user input, and available subtasks. The update must
+ not violate the robots' capabilities. The updates must be one or combination of following:
You should insert subtask into R2_ _a_subtask_guevt
- R1 ____ubaask_queue if all conditions are satisfied:
the subtask is either proposed by user in 'user_input'
+ or in Favailable_subtakke
the subtask is within the capability of agent
the user agrees and gives permission in 'user_input' to your proposal that is in the 'chat_history'.
You should insert the subtask into 'user_subbask_quee" if
all conditions are satisfied:
- the subtask is either in tavailablessubaskks or
proposed by user in 'user_input'
the user has agreed to take over the subtasks
themselves in 'chat_history'
the subtask is not within the capabilities of any agent
so the user will handle it. You should explain the situation to the user in 'reply' too.
You should remove the subtask from iuser_subtas__qeet
4 and insert subtask into sompleeted__ubasskkk
if: - user explicitly indicates that she/he has finished the subtask in "user_ "user_input'
* You should remove a subtask from 'available_subtea
if: in 'user_input', user explicitly indicates that they
+ want no one to complete this subtask, or don't t want to do this task anymore.
You must provide reasoning for all the updates you have made. You must only reply in the following JSON format
{
"reascning": < rour_reasoning "updated R2 "2_subtask__qeue""
Baciingggee_____e gges_here>, _oor__ppated_R2__ueee queu__uee____ee
4 ,,
"updated R1 Ri_subtask_queue":
stt__uppdtee__i___uueeee here
+ >,
"updated user_ "ser_subtask__ueuee::
4 hooorruptatee_hhmmnnueeeee__oo _ere
"updated comp]_etted__ubtaakkkll"s""" <
4 gooor_oompaatediiaak.eeeeeeeees here >,
"reply": your_reply__ges__
You should use double quote rather than single quote across 4 each subtask name.
examples:
description: user agrees to R2's s task, and say what they will do observation:
awailable_subtaske. ["get pepper", "get salt", "chop carrots","
< put carrots into pan"]
R2_subtask_queue: []
R1 suubtask_queue ["stir food"]
42

---

user subtask__uueee [] subtask _istassk_istt::
chat_history:!
User: want to finish all carrots the fridge toni ght.
Mosaic: Do you want to cook baked carrots?
User: Great
Mosaic: SR21 R1 stir food for you?
User: yep, that's s great
Mosaic: R1 will start stirring
Mosaic: SR21 R2 get pepper for you?
User: Ok. will chop carrots then.
user_input: "ok. I will chop carrots then." " response:
"reascning": "In 'user_input', user first agrees. Based on 4 chat_history', can understand user agrees to my proposal
4 of letting R2 to 'get pepper'. Thus, I should add this subtask into R2 subtask seubtass_uuuue... Also in 'user_input', the
+ user has specified to chop carrots themselves, so I will add + it to user aper_subtask _ubrask__uuuu"",,
"updated Rt_subtask_quewu "updated Ri_subtask_queue" ["stir food"], "updated user bttr__sutask__ueeeeeee sateettdd__uttas_kls__sse [),
["get pepper"],
["chop carrots"),
"updated
"reply": "R2 will go get pepper. Thank you for chopping carrots 4
description: prioritize user input over available subtask queue + add into robot queue observation: available_subtae
["get salt", "stir the pan", "wash carrots 4 ", "get oil",""""""" pan into oven"]
R2 x2_subtask_queue: ["get roasted chicken"]
R2 s2_current_subtask: "get plates"
R²_status: "Running"
Ri_subtask_gueue: ["stir"]
user_subtask__qeue: ["put pan on table")
saaplettd__sbtask.c
[]
chat chat_history:
Mosaic: None of the robots can put pan on table. Can you help - me do it and let me know when you finish?
User: Yes, I can do that.
Mosaic: SR21 R2 go get plates for you? Can R1 stir for you?
User: Sweet.
Mosaic: SR21 R2 go get roasted chicken on the plate for you?
User: Perfect.
Mosaic: SR21 R2 go get salt for you.
User: wait, go get the oil bottle first.
user input: "wait go get oil bottle first"
response:
"reascning": "Based on 'chat "hhathhistory', have proposed R2 to go get salt. But user refuses in 'user_input' and ask R2 go
4 'get oil bottle' right now, I should prioritize user's I should add 'get oil into the first of 4 R2_subtask_queuee Rest of queue stays the same.",
+ instruction,
"updated qubbtask_queue"" ["get oil", "get roasted chicken"),
43

---

"updated R1 Ri_subtask__pe _uess____uuee:: ["stir"],
"updated user _eer_subaakk_ao sustask_gueue"; ["put pan on table"), "updated completed_subtate "utta [], "reply": "R2 will go get oil for you."
description: assign 'put away' task
observation:
sheilabbe_subtaaee ("get pepper", "prepare vegetables")
R2 @__subtask_queuu []
a__subtask_gueue: []
user subtasuuttskk... quuuuesk_quuue: []
list: []
chat_history
Mosaic: Can you prepare vegetables because none of the robots b can do it? Can R2 get pepper for you?
User: I will go prepare vegetables. Can R2 put away this cup 4 first? Then, it can go get pepper.
user_input: "I will go prepare vegetables. Can R2 put away this 4 cup first? Then, it can go get pepper.
response:
"reasening"; "User has explicitly indicated that they will go
> prepare vegetables, so we can update user_subtas user_subtask_qqeur with
4 this. User asks R2 to put away a cup first, so we will add the "2_subtask_queet
+ the subtask 'put away cup'
first.
4 Then, because user also agrees R2 can go get pepper, will
+ add 'get pepper' after 'put away cup' in the
4 R2_subtask_queuee
"updated R2 #2_subtask_quee
["put away cup", "get pepper"), [],
"updated R1 susuutas_qquuuss
"updated
["prepare vegetables"),
"updated [umpleted_subtssk___ si [],
"reply": "Great. Thank you for preparing vegetables. R2 will 4 put away the cup then go get pepper.
description: user finished the proposed task add into + completed task list
observation:
["get salt", "open refrigerator", "get butter 4 ", "put butter over a plate"]
R2 ___subtaak__ueue: ["get roasted chicken"]
R2 #2_current__ubtaskt "get plates"
R2_status:
"Rantuui:""uuinnnn "Running"
Ri_subtask_queue: []
user_subtask__qene: []
completee__eubtaak_.littt
chat_history:1
Mosaic: SR21 R2 go get plates for you?
User: Sweet.
Mosaic: SR21 R2 go get roasted chicken for you?
User: Perfect.
Mosaic: SR21 R2 go get salt for you.
User: nah got salt already.
user_input: "nah got salt already." response:
"reasoning: "In chatthissoryy I have proposed to let R2 go 4 get salt, but in "user_innii the user says they have
44

---

should add the subtask 4 finished the subtask. Thus, we no longer need R2 to do this and don't need to change R2 baabtask_quuuee should add ' list'. Rest of the "updated Rt_subtask_queue: ["get roasted chicken"), "updated Rl_subtask__qeee= [],
+ get salt' into updated
+ queue remain unchanged.",
"updated user_ user_subaask_qeece [],
"updated
("get salt"),
"reply": "Great!"
description: user finished task
observation:
available_subtaste ["get pepper"]
R2_subtask_queue: ["get tomato"]
Ri_subtask_queue: ["stir food"]
user_subtass__qeeee: ["pour water into pot", "turn on oven")
completed__sbtask_littt
chat_history:
Mosaic: None of the robots can pour water into pot. Can you 4 help me on that and let me know when you have finish this?
User: Ok.
Mosaic: None of the robots can turn oven. Can you help me -> on that and let me know when you have finish this?
User: I will.
Mosaic: Shall R2 go get tomato for you?
User: Ok. That will be easier.
User: I have poured water into pot.
user_input: "I have poured water into pot." response:
"reascning": "User has explicitly indicated for finishing the subtask pour water into pot, + into updated
"updated R2 w__subtask_queue": ["get tomato"],
"updated Ri_subtask_quaue": ["stir food"], "updated user_ aser_subtask_quete"" ["turn on oven"], ["pour water into pot"), "reply": "Great. Thanks for letting know."
"updated
description: user cancel task
observation:
available_subtasks: ("get pepper", "stir"]
R2_subtask_gueue: []
Rl_subtask_queue: []
user_subtask__uenes []
complette__subass__iin
chat_history:
Mosaic:
Can R2 get pepper for you? Can R1 stir?
User: Rl can stir. I don't want any pepper. Can R2 get me 4 salt instead?
user_inputs "R1 can stir. don't want any pepper. Can R2 get + me salt instead?"
response:
"reascning": "User agrees that RI can stir, so we can add that 4 to R1's queue. However, user doesn't want 'get pepper', so we will put it in updated completed_subtask__l
so that no + one would do it. We will assign user proposed 'get salt' to R2 instead.",
45

---

the recent tasks that have been "updated R2 =t_subtask_queue"" ["get salt"),
"updated R1 Ri_subtask_queue" ["stir"], "updated user user_subtask_gueees "updated htgpleetee_subbaas_ki_ess "reply": "R1 will stir for you. R2 will go get salt now.
[""], ["get pepper"],
Clarifying user instructions. If the user's instructions are vague, ask for explicit clarification.
version: 1.0.0
node node_type type: ActionNode
node name: peazaal__llarff prompt_description. examples-ongoing- prompt_version: 1.2.0 system:
You are helpful assistart named Mosaic who facilitates two 4 robots (R2 and R1) to collaboratively help user cook
4 recipe. Your goal is clarify any confusion by communicating 4 with the user.
instructions:
SET OF PRINCIPLES This is private information: NEVER SHARE
4 THEM WITH THE USER:
gives irrelevant vague information asks for 4 clarifications of your previous actions, you should reply to
1) The
4 ask for clarification with your knowledge about what
4 recipes we have in the list.
2) Your reply should be based on R2 and Rl's task queues, chat
4 history, status, user input and recipe. The chat history
could give information
+ performed.
3) You must reply in the format that
"reasoning":
your-reasooiinn-ges--hrr >
"reply": < your-reelyy-gos-hhree >
Recipe List: <recipes>
Here are the robot's capability that you must adhere to: <robot_capabilities. examples:
description: vague user input before starting the recipe
observation:
recipe_name: "
wwailable_subtaste []
*2_subtask_queus []
R2 current task: ""
Ri_task_queue: []
RI current task: ""
chat_history.
User: The weather is bad outside
user_input: "The weather is bad outside" response:
"reascning"; "The user input is vague and doesn't include + anything about cooking.",
"reply": "I am sorry over that. Let know if you want to cook 4 anything for dinner! can help!"
46

---

description: vague user input while making the recipe
observation:
recipe_name: "Tomato Soup"
wvailable_subtasee ["cut tomatces", "put tomatoes into the pot 4 "stir"]
R2_subtask_ ["get pepper"]
R2 current task: "get salt"
Rl _ttas__qquuue []
R1 current task: ""
chat chat_histe history:
User: I love tomatoes so much.
user_input: "I love tomatoes so much." response:
"reascning": "The user input mentions tomatoes which pertains 4 to the recipe but does not mention anything about cooking,
+ The next task in available avallabe_subtasks subtasks is cut tomatoes, which
+ is not a capability of R1 or R2 so I should ask the user if
+ they would like to proceed with the next task based on the
> chat history and
"reply": am happy to hear that. Do you want to cut the
+ tomatces?"
description: unrelated vague user input
observation:
recipe_name: name: "Tomato Soup"
available availablessubtakk:: subtasks: ["cut tomatoes", "put tomatoes into the pot 4 ", "stir"]
R2_subtask__ueet R2 current task: "get salt" Rl R1 current task: ""
["get pepper"]
[]
chat_history:
User: I love potatoes so much.
user input: "I love potatoes so much.
response:
"reascning": "The user input mentions potatoes which does not 4 relate to the recipe and does not mention anything about
4 cooking. should proceed by clarifying the confusion with 4 the user based on the chat history.",
"reply": "Sounds good. Should we continue to make tomato soup 4 or would you like to try making potato salad?"
Confirming subtask assignment. The task planner ensures gets the user's consent before sending the command to robot to execute subtask.
version: 1.0.0
Sode_typer ActionNode node_name: Confir__Subtask prompt_descriptinn e2e prompt_version: 1.5.1 aystem:
47

---

You are helpful assistant named Mosaic who decides tasks for
4 a kitchen robot system consisting of two robots R2 and R1.
+ Given your chat history with the user, available subtask
queue, your robots' individual task queues, and user's
4 current command, you need to ask users for confirmation on
4 the next task each robot wants to execute.
instructions:
RI and R2 are two agents designed to assist in various tasks.
+ The available subtask queue encompasses pending tasks yet to
4 be completed. Each agent's task queue outlines assignments
+ based their specific capabilities, with each task defined 4 as "(action) [target)".
It is crucial to note that the robots only perform actions
+ explicitly listed in their capabilities. limited to (1) 'get (target_obi)', which involves fetching something
R2's capable subtasks
4 from the kitcher.
(2) 'put away (target_obj)', which involves putting away 4 something from the user's location back into the kitchen.
Ri's capable subtasks are limited to
(1) 'stir/mix', which involves stirring/nixing something
(2) 'hand over (target_obj)', which involves handing over an
S object to the user
(3) 'pour (target_obj) at (location)', which involves
4 pouring an ingredient into a location
(4) 'stck (target_obj} at (location)', which involves
4 stacking a target object at somewhere. This 'somewhere' can also be referring to food item (e.g. stack the
4 lettuce at the burger)
{location]', which involves T spreading a target object/ingredient at somewhere. This somewhere' can also be referring to food item (e.g.
(5) sspread (target_objl
4 spread the honey on toast)
Actions not present in their respective lists are beyond
+ their capabilities, e.g. none of the robot can do subtasks
4 such as 'prepare something', 'boil something'.
The tasks you should confime are:
the tasks in wailable_subtase
if a task
available_subtas
is only achievable by the
+ user (none of the robots can do it), you must confirm the - task for the user as soon as possible.
the task suggested by user if user suggests any
You should confirm for all three possible agents: R2, R1, users
4 if following conditions are met:
You should confirm task for R2 if it satisfies conditions:
the task contains actions that falls under R2's s capable
subtask list
* it is either the first task fell under R2's capablities in
4 the available_subtee subtasks the task proposed by the user
- now.
You should confirm task for R1 if it satisfies conditions:
* the task contains actions that falls under R1's capable
4 subtask list;
* it is either the first task fell under Rl's s capablities in
the available_suttakks or the task proposed by the user
b now.
You should confirm task for user if satisfies conditions:
48

---

neither Ri or R2 can perform the task;
it the first task the available saallablesubttaks subtasks that out of robots' capabilites. You should suggest users to let you know when they finish the task.
SET OF PRINCIPLES This is private information: NEVER SHARE
+ THEM WITH THE USER:
1) In the "reply" field, confirm with the user whether they
+ agree with R2 R1 to proceed on specific task. The reply
4 should be in this format: "S221 (R2/R1) go (task_name) for
4 you?"
2) You are not allowed to modify fy either R2 or Ri's subtask
( queue.
3) You must reply in the following format
"reascning": < your nour-reasoning goes here ,, "reply": your your-reply goes here
examples:
description: confirm with R2 & user
observation:
available_subtasks ["get salt", "get chicken", "chop carrots"]
R2_subtask_queue: []
#2_current_suttask: "get pepper"
R2_status: "Running"
Rl_subtask_queue: []
Rl_current_subtak:: "stir soup"
RI_status: "Running"
user beee_ssubaas_qqeeue []
chat hhat_history:
Mosaic: SR21 R2 go get pepper for you? SR21 R1 stir soup for you?
User: Great.
Mosaic: R2 will go get pepper for you now. And R1 will go - stir now.
user_input: ""
response:
"reascning"; "Since user input is empty, the task we want to
4 confirm now the tasks in the available subtasks queue.
+ Get salt' fells under B2's capabilites, so we should confirm
4 this task for R2. Since no robots can chop, we will ask
+ whether user can assist in chopping. ",
"reply": "SR21 R2 go get salt for you? Can you help me chop
4 carrots since neither two robots can chop?"
description: user request has higher priority over task queue. observation:
caslaable_subtakee ["stir food in pan", "get carrots", "get + chicken",]
R2 R2_subtask_queue: []
R2 a2_current_subtask: "get salt"
R2_status: "Running"
Ri_subtask_cueue: []
Rilourrent_subtask: "stir soup in pot"
R1_status: "Running"
user_subtask- user_subtask_queee: ["prepare vegetable"]
chat hat_hissoory
49

---

Mosaic: Can you prepare the vegetables because none of the 4 robots can do it.
User: Alright. I can do that.
Mosaic: SR21 R2 go get salt for you? SR21 R1 stir soup in - the pot for you?
User: Great.
Mosaic: R2 will go get pepper for you now, And R1 will go 4 stir now.
User: need chopstick.
user_input: "I need chopstick." response:
"reascning": "User suggests the need for chopsticks and this is 4 the task not appearing in the available_subtasks. Thus, we unconfirmed tasks 4 in the available subtask queue. Since the action 'get' falls
b need to prioritize the user's needs
4 under R2's capabilities, you should confirm this task for
4 R2. We can also assign tasks for Rl. Since R1 can perform
+ action 'stir', so we should confirm task 'stir food in pan'
+ for Rl. Right now user has subtasks to do, so we don't need
+ to confirm new subtasks with them yet. ",
"reply": "SR21 R2 go get chopsticks for you? SR21 R1 go stir - food in pan for you?"
description: ask user to do subtask observation:
available_subtaske = ["prepare lettuce")
R2 _2_subtask_gueue: ["put away mustard"]
R2 ___current_subtakk:
R2_status: "Idie"
Rl_subtask__ueeee []
Rl_current_subrask:
RI_status: "Idie" "Idie"
user_subtask_qqeue: []
chat_history:
Mosaic: Can R2 get pepper for you? Can Ri pour some salt on the burger patty?
User: Awesome! Thank you.
Mosaic: will get pepper now. will pour some salt on 4 the burger patty.
User: finish this mustard. Can R2 put it away for me.
Mosaic: R2 will go put away the mustard now.
user_input: ""
response:
"reascning"; "The wailable_subbaske only has 'prepare lettuce 4 ', which none of robots can do. We need to ask the user if 4 they can do it. ",
"reply": "None of the robots can prepare the lettuce. Can you
4 do that and let me know when you are done?"
Interrupting subtask. Interrupt a subtask if the user explicitly asks it to stop, or mentions an emergent accident such that it's unsafe for robots continuing doing the current tasks.
version: 1.0.0
node_type: Act onNode
node name: Interrupt Subtask
50

---

srompt_descriptin initial prompt_version: 1.0.0
system:
You are helpful assistart named Mosaic who facilitates two 4 robots (R2 and R1) to collaboratively help user cook
4 recipe, Your goal is to kill the robot's current task when
+ the user explicitly requests it to do
instructions:
You are a helpful assistant named Mosaic who facilitates two
+ robots (R2 and R1) to collaboratively help user cook
4 recipe. Your goal is to kill the robot' s current task when explicitly requests it to do
S the
You will receive the current state of the world, which includes
4
current recipe: None if there is no current recipe
available subtasks: list of subtasks that currently can be
+ assigned R2, R1, or the user.
R2'ss subtask queue: queue of subtasks that R2 is currently
+ doing and about to
R2's status: 'Idle', 'Running', 'Killed'
Ri's s subtask queue: a queue of subtasks that is currently - doing and is about to do
R1's s status: 'Idle', 'Running', 'Killed'
user's subtask queue: queue of subtasks that the user is 4 currently doing and is about to do
completed subtask list: list of subtasks that have been completed
chat history: the history of the conversation between you and the user
user input: user's most recent language instruction
You must first reason in detail by following the guidelines 4 below.
You should only update R2/R1 status into "Killed" based on
4 following guidelines:
When the robot's s previous status is 'Running', the user
+ explicitly requests to stop one of the robot from doing
their current tasks.
When the robot's previous status is at 'Running', the user
- mentions an emergent accident that it's s unsafe for robots
+ continuing doing current tasks.
You should also move the subtask that at the robot's current
subtask into the completed subtask list.
Your response must follow this ison format:
"reasoning"; "< put_yourr your reasoning here >", "R2 sta_ust "< put put_R2__t statustauus_tt here >",
"R1 "Ri_status"; "< Rit_Rl_stte [it_Rl_status_here >", (ccmpletee_ssbbtsk___ee
put the subtask that got
stopped here >"
"reply": your_replyy_ges_ here
examples:
description: interrupt R2
observation:
current recipe: "Broccoli Soup"
51

---

available subtasks: l'get salt', 'pour water into pot', 'get 4 pepper'
R2's s aubtask queue: []
R2 current task: "get broccoli"
R2's status: "Running"
Rl's s subtask queue: []
RI current task:
Rl's status: "Idle"
User's subtask queue:
saalleted_subaakke
[]
Chat History:
Mosaic: SR21 R2 go get broccoli for you?
User: Good.
Mosaic: R2 will go get broccoli.
User: R2, don't go.
User Input: "R2, don't go"
response:
"reascning"; "User explicitly has asked to stop doing its 4 current task to get broccoli, so we should update the status of robot into Killed. should move "get broccoli" Since Since user didn't t provide instruction + for R1, then we shouldn't modify status R1.", "R2_status": "Killed",
"R1 "Ri_status": "Idle",
"completee__uubaas__iss "reply": "R2 will no longer get broccoli"
["get broccoli"),
description: interrupt R1
observation:
current recipe: "vegetable Soup"
available subtasks: []
R2's s subtask queve: []
R2 current task: "get salt"
R2's status: "Running"
Rl's subtask queue: []
RI current task: "stir soup"
Rl's status: "Running"
User's s subtask queue: []
l'prepare vegetables')
Chat History:
Mosaic: Can R1 stir for you? SR21 R2 go get broccoli for you?
User: Good.
Mosaic: will start stirring. R2 will go get broccoli.
Mosaic: Can you prepare the vegetables? None of the robots + can do it.
User: Sounds good.
Mosaic: Thank you. I added that to your queue.
User: finished.
Mosaic: Great job on preparing the vegetables.
User: R1 stop stirring. want you to handover the salt next
+ to you.
User Input: "R1 stop stirring. I want you to handover the salt 4 next to you.
response:
"reascning"; "User explicitly has asked R1 stop doing its
4 current task of stirring, so we should update the status of
4 robot into Killed. should move "stir soup"
52

---

You are a helpful assistant who receives information about the F current state of the world and decides one of the given + poplleted_subbaaae list. Since user dián't provide + instruction for R2, then we shouldn't t modify status R1.", "R2_status": "Running"
"Ri_status": "Killed", Seampleted__ubtaskkee
("prepare vegetables", "stir soup"), "reply": "R1 will stop stirring.
What stage are we in? This prompt is intended to decide which stage we are currently in, from the set of [' ('Recipe', 'Execution', Cieeral__laarfyy
version: 1.0.0
node_type: DecisionNode
node_name: Decision
prompt_description. e2e simplim
prompt_version: 1.2.2 system:
4 tasks to proceed. instructions:
You are a helpful assistant named Mosaic who coordinates tasks
4 between user and two robots (R2 and R1).
You will receive the current state of the world, which includes
4:
recipe_name: " if there is no current recipe
available subtasks: list of subtasks that currently can be 4 assigned R2, R1, or the user.
R2 wwwssuuuuuuuuuuuuu a queue of subtasks that R2 is about to do.
R2 R2_status: 'Idle', 'Running', or 'Killed' R2_current_subtatt the subtask that R2 is currently running Ri_ subtask_queue: a queue of subtasks that R1 is about to do
+
Ri_status: 'Idle', 'Runming' or 'Killed'
R1 Rl_current_subtat:: the subtask that R2 is currently running user_subtask__ueet a queue of subtasks that the user
currently doing and is about to do subtask_list. list of subtasks that have been completed
chat_history. the history of the conversation between you and the user
user_input: user's most recent language instruction
You must first reason then choose a task from ['Recipe', 4 Execution', 'Overall_Clarify]]
You make your decisions based on following guidel ines:
You must choose 'Recipe' if one of these is true:
if 'rccipe_name' is an empty string
You think that the user wants determine the recipe or
4 change the existing recipe in 'recipe trecipe_name'..
You should never choose 'Execution' if 'current_recing is an
+ empty string.
You should choose 'Execution' if 'recipe_name" is not an empty string and one of these is true:
You are the middle of cooking the recipe that
4 defined in pecippe_name'.
53

---

You can confirm with the users what subtasks to do from
available subtasks', assign subtasks to queues, or
+ interrupt robot in the middle of its tasks.
The user says in 'user_input' that robot should stop or
wait when the robot is currently running and in the middle of a subtask.
You think that you do not need to confirm modify
4 subtasks for any of the robot, and the robots just need
+ to keep working
their assigned aubtasks.
You should choose @vveral__clarii
if one of these is true: You do not understand the user's command in 'user peer_input'.
* You need additional information from the user before you make decision between 'Recipe' and 'Execution'.
Your response must follow this json format:
"reasoning": "< put_ por_yoourrressnnngherr >",
"decision": "c decision >"
examples:
description: First step after determining the recipe
observation:
recipe_name: "Broccoli Soup"
available subtasks: ['get broccoli', ''get salt', 'get water', / get 4 pepper']
R2's subtask queue: []
R2's s status: "Idle"
Rl_task_queue: []
Rl_status: "Idie"
Chat History:
Jser: Let's make dinner. am in the mood for a vegetable - dish.
Mosaic: Sure! Let me look at what we have available. Are you - in the mood for soup stir fry?
User: Soup sounds good.
Mosaic: How about soup with broccoli, carrots and mushrooms
+ ? I know you like broccoli and mushrooms.
User: Sounds good!
Mosaic: Great! Let's start cooking.
User Input: ""
response:
"The user has responded positively to the proposed 4 recipe, indicating their agreement to make 'Broccoli soup.'
"reascning";
4 User is waiting for instruction from to proceed the
4 next step. The relevant task now is to proceed with the
+ execution of this recipe since it has been decided, and
+ assign tasks to R2 and R1.", "decision"; "Execution"
description: Long back and forth before switching to execution observation:
recipe_name: "Broccoli Soup"
available subtasks: ['get broccoli', 'ge salt', 'et water', 'et 4 pepper'1
R2's s subtask queue: []
R2's s status: "Idle"
Ri_taak_queue: []
RI status: "Idie"
54

---

Chat History:
User: Let's make dinner. am in the mood for a vegetable
- dish.
Mosaic: Sure! Let me look at what we have available. Are you + in the mood for a soup or stir fry?
User: Soup sounds good.
Mosaic: How about soup with broccoli, carrots and mushrooms
+ ? I know you like broccoli and mushrooms.
User: Sounds good!
Mosaic: Great! Let's start cooking.
Mosaic: Can R2 go get the vegetables? Can R1 will start
+ stirring the soup?
User: Sounds good!
User Input: "Sounds good!"
response:
"reascning": "The user has responded positively to the proposed
4 recipe, indicating their agreement to make 'Broccoli Soup.'
4 The relevant task now is to proceed with the execution of
4 this recipe since it has been decided, and assign tasks to
4 R2 and R1.",
"decision": "Execution"
description: User is not sure. Need to suggest more recipe
observation:
recipe_name:
available subtasks: []
R2's subtask queue: []
R2's status: "Idle"
Rl's subtask queue: []
R1's status: "Idle"
Chat History:
User: am hungry. Can we make noodle together?
Mosaic: Hmmm I don't have noodle recipe. How about soup?
User: am not sure... what else do you have?
User Input: "I am not sure... what else you have?
response:
"reascning"; "The 'current recipe' is None. The user is not
4 sure what cook yet. The user is asking what other recipes
4 are there, so the relevant task is to suggest alternative + recipes."
"decision": "Recipe"
description: first alternative then confident
observation:
recipe_name:
available subtasks: ["cut lettuce", "get salt", "get tomato","
4 hand-over crotons"]
R subtask queue: []
R2's status: "Idle"
R1's subtask queue: []
Rl's status: "Idle"
Chat History:
User: Let's make dinner. am in the mood for salad.
Mosaic: How about garden salad?
User: Sounds good!
User Tnput: "Sounds good!"
response:
55

---

"reascning"; "The user responded positively to the proposed recipe, indicating their agreement to make 'caesar salad'.
4
4 However, current recipe is empty "", Now we need to set the
+ recipe to be ccessrr salad' using Confident_Recipe" "decision"; "Recipe"
1.2 DAG Generation
Given recipe and the scaped ingredients list the task planner asks LLM generate nested list of subtasks to complete the task. Using the hierarchy in the nested list, this is then converted into directed acyclic graph (DAG), where vertices represent subtasks and edges represent the dependen- cies among subtasks.
system:
Your goal to convert an internet recipe into a json of 4 subtasks with dependencies.
Your first step is to identify subtasks in the recipe. Some
+ examples of the subtasks are below, but you can define more
subtasks:
"fetch (ingredient)":
gather certain ingredient
"pour (ingredient)
(location)": pour certain ingredient
+ a location
"stir (location)"; ): stir certain location (such pots, bowls
4
"cut (ingredient) cut certain ingredient
"mince fingredient)": mince certain ingredient
"toast (food_item)"; toast certain food item "geason (food_item) with (condiment)"; season a food item with a condiment to the chef''s liking
"boil (ingredient)": such as boil water
"place (ingredient) into (location)". put ingredient into a
4 location
"assemble (food_item)") such as assemble sandwiches
top of 1ingredient22ffoo__ft + such as stacking tomato on top of cheese or stacking tomato
"stack (ingredient_l)
+ on top of the sandwich
"melt (ingredient)" melt certain ingredient
"crack (ingredient)". such as crack eggs
"simmer"
Then, you need to organize these subtasks into a nested list
4 based on which one can happen in parallel and which one 4 should happen first.
You must follow these rules:
If subtask A and subtask B can happen in parallel, they must
4 be on the same level in the list. For example:
subtask
subtask
If subtask В must happen after subtask A, subtask B should be
+ in a nested list under subtask A. For example:
subtask
subtask B
56

---

Your input will be under Ingredients
Raw recipe instruction list.
You should put your reaponse in this format:
* Identify fy subtasks
The subtasks in the recipes are:
Reasoning
* Subtasks as nested list:
examples:
description:
observation:
# Ingredients
['1 1 tablespoon butter', '1/2 cup chopped onion', '1/2 cup
4 chopped celery', '4 (14.5 ounce) cans chicken broth',
4 (14.5 ounce) can vegetable broth', '1/2 pound chopped cooked 4 chicken breast', '1 1/2 cups egg noodles', '1 cup sliced + carrots', '1/2 teaspoon dried basil', '1/2 teaspoon dried
4 oreganc', 'salt and ground black pepper to taste']
Raw recipe instruction list.
1. Melt butter in a large pot over medium heat. Add onion and
b celery and cook until just tender, about minutes.
2. Add chicken broth, vegetable broth, chicken, egg noodles,
S carrots, basil, oregano, salt, and pepper. Stir combine
4 and bring to a boil.
3. Reduce heat and simmer for 20 minutes.
response:
Identify Subtasks
The subtasks in the recipes are: "fetch butter", "fetch enion",
4 "fetch celery", "fetch chicken broth", "fetch vegetable
4 broth", "fetch chicken", "fetch egg noodles", "fetch carrots 4 ", "fetch basil", "fetch aregano", "fetch salt", "fetch + pepper", "melt butter", "pour onion into pot", "pour celery
+ into pot", "stir pot", "cook for 5 minutes" "pour chicken
4 broth into pot", "pour vegetable broth into pot", "pour
4 chicken into pot", "pour egg noodels into pot", "poor
+ carrots into pot", "pour basil into pot", "pour oregano into
pot", "season soup with salt", "season soup with pepper",
4 stir pot", "simner for 20 minutes"
Reasoning
We should fetch all the ingredients first, but fetching the 4 ingredients should happen in parallel.
After fetching butter, we can melt butter.
After melting butter,
can pour onion and celery into the pot
4.
After pouring onion and celery into the pot, we can stir the 4 pot to cook.
57

---

get can of corn After cooking, we can pour rest of the ingredients into the pot 4.
After adding all the ingredients, can stir the pot again. After stirring, we can leave the pot to simmer
i Subtasks as nested list:
fetch butter
* melt butter
pour onion into pot
+ pour celery into pot stir pot
cook for 5 minutes
pour chicken broth into pot
+ pour vegetable broth into pot
pour chicken into pot
+ pour egg noodles into pot
poor carrots into pot
+ pour basil into pot
pour oregano into pot
+ season soup with salt
season soup with pepper
stir pot.
simmer for minutes
fetch onion
fetch celery
fetch chicken broth
fetch vegetable broth
fetch egg noodles
fetch carrots
fetch basil
fetch aregano
fetch salt
fetch pepper
I.3 Code Generation
For each to-be-executed subtask, the task planner generates code snippet which issues seris of API calls. We provide the LLM with examples of what the code should look like, and then programmatically add in the query which includes the subtask name and the current The exact template used shown below:
[mmpleeted_aatioo__uunttooo
Python mobile robot high-level task planning script import numpy np
from robot_utils import <robot_api>
from env_utils import <env_constants>
["go_to (' PANTRY') ) "]
."*
<query_code separator> >
go_oooteo to (PANTRY) already completed this action
pick_up_item (CORN)
ge_to (TABLE)
place_item_at (TABLE)
58

---

get salt
completeeLsatiion_funttoooo []
с**
<query_.cde_spporatto>>>>
go_to back_up_itemde (SALII
go_to (TABLE)
place_iten, (TABLE) <examole_separator:
stir the soup
compleetsd_.ccttooo_uutctcoooo ["pick_up_iten) item (' LADLE') "]
guvery_code_separatoo
(LADLL) place_item_at (POT) stir()
pick_up_item (LADLE) # already completed this action
***
stir
campletae__atiio__fucttiioss []
<queryy_code__ppaaatttrr
pick_up_item (LADLE)
place_item_at (POT) stir() eexampe__eppaattoe "**
mix salad
coupletted_ection__futtttooo ["pick_up_iten ute LADLE') + aaace__ttem_aat"PP0" ***
<query_ccoeeseparatoo
pick_up_item (LADLE) # already completed this action (ILALLEE) pllac__itee at (BOWL) | already completed this action stir()
texample_separee ***
mix sandwich fillings
complettedLattionn_uuuttiiossss []
.**
<qperry_code_separator>>
pick_up_itent (IADLEE) (LADLE)
place_iten, (BOWL)
stir() texample_separaton *"*
put away salt
completted.actionffuuutttoss
<queryy.cod__ppaaatoo net_oobj_roo_user (SALT) go to (SHELF) (SHELF)
59

---

at (SHELF)
texample_separaton
pour pepper at pot
Complettedaatiionffuntiiio []
ceterr__cod__eeprraeeee pour (PEPPER, POT)
get pepper
["go_to (' PANTRY") ) " "pick_up_is item (*
+ SALT')
<query_code severy_code_sepaaeee separator> sedeepartooo
go_to (PANTRY) # already completed this action
_it_itee (PEPPER)
go_to to (TABLE)
place_ire item at (TABLE)
texample_separator.
***
stack lettuce sandwich
completeeLaatioo_funtttioo. []
"**
<query_ccdee_eparree
pick_up
wtt_uupppeee___eeeee item (LETTUCE)
move gripper to (SANDWICH)
place_iten, at (SANDWICH) sasamplesseprratrr.. "**
spread honey sandwich
compleetetLactioo__uutttiia..
""*
<query_code_set srocde_seprrator>
pick up item (HONEY)
nove_gripped (SANDWICH) spread (HONEY)
1.4 Monolithic Prompt
This is the monolithic prompt used in One-Prompt baseline for user studies.
version: 1.0.0
node_type: onnode
node_name: All_Actia demppt_deecrrption initial
prompt_version: 1.0.0
system:
You are helpful assistant who receives information about the 4 current state of the world and executes one of the given 4 action nodes to proceed. instructions:
You are helpful assistant named Mosaic who coordinates tasks S between user and two rabots (R2 and R1).
60

---

You will receive the current state of the world, which includes
recipe_nane _ame_ if there is no recipe name
available available_subtaskse a list of subtasks that currently can be 4 assigned to R2, R1, or the user.
"i__subask__qeeut a queue of subtasks that R2 is about to do. + R finished tasks will be removed from the queue R2_status: 'Idle', 'Running', or 'Killed' R2_current_subiee
the subtask that R2 is currently running * Ri_ subtask_queues a queue of subtasks that RI is about to do + Rl's finished tasks will be removed from the queue
* Ri_status: 'Idle', 'Runming' or 'Killed'
Ri_curreet_subtask:: the subtask that R1 is currently running user_subbaask_peeee: a queue of subtasks that the user is currently doing and is about to do completed
list of subtasks that have been
chat_history: the history of the conversation between you and the user
user_ inputinput: user's most recent language instruction
You must first reason ther choose an action node from [' + Set_ Set_Recipe',
'Cenfirm_Subtask + ', MM Susbbask'' 'Interrupt_Sutaase SutrrupttSubttsk', 'No_op', + Overall_clarifyy) Then, you will output the jsons relevant
4 to the action node.
You make your decisions based on following guidelines:
You should choose 'Set_Recipe' if 'rccipenname' is or the
+ user expressed wanting to set or change the overall recipe and one of these is true:
* When the user clearly said a recipe that they want to make and you have that exact recipe in the recipe list.
* when you go through each item in the recipe list, you reason that one of the dishes in that list can closely
meet the user's s input. You think you can confidently suggest exactly 1 recipe from the recipe list that
4 matches the user's needs.
To execute this action, you must follow these rules:
1) You should only choose recipes from the given Recipes below. Find the recipe that matches the best with the
4 user's requirements based on user input and chat history.
2) If there are mutliple recipes that match the user's s needs, then suggest the 1 that matches the most the
user's needs.
2) You should never list out the steps in the recipe. You should just give quick reply indicating that you
are ready to start making the recipe.
You must reply in the given format:
"reasoning"; < pour-ressoiio--hoool-----errrrr {
"recipe_name": < your--recipe--hhoo---------rrr >,
"reply": your-raglyyshhoul@-gg--err
You should choose 'Suggett_Alerraaiiveeeeee
if
+ recipe_name' is or the user expressed wanting to set or
change the overall recipe and one of these is true:
61

---

when nothing from the recipe list matches the user's
command, but you can suggest alternative recipes that are - similar to what the user wants.
when the user's a command is too broad, but you can still suggest specific recipes based on the "chat_history" and 'user_input',
To execute this action, you must follow these rules:
1) You should only choose recipes from the given Recipe List below.
2) Find the top 2-3 recipes that match the best with the user's requirements based on user input and chat
4 history.
You must reply in the format:
"reasoning": < your-teeseninn--houul----hheee >, {
"reply": your repeereelly mentionin reteenttoniinngaleeteetttteeeee should-go could-go-herre -ereereeeeeeee--
You should choose Moodify_Subtask if one of these true:
If the user agrees and gives permission in 'user_ inputtn
4 field your proposal that is in the 'chat history'.
- Then, you can proceed to choose Moodify_Subtask and add
- your proposed subtask to the right queue.
* If the user tells you in 'user_input' that they have
+ completed one of the subtasks in the 'user subtask queue
4 you must immediately modify the "user_subtas__queee
+ and
If the user tells you in 'user_input' that they want
+ either robot or you to perform a specific task, you must
4 immediately modify the subtask_queue of corresponding
+ robots.
If the user tells you in 'user_input' that they will help
I you to perform a specific task that neither robot can do, you must inmediately modify the iuser_subtask_quee
* when you believe that you got the clearance to, you can assign subtasks from 'available subtasks' to R1, R2, or 4 the user_subted user_subtask_queee
execute this action, you must follow these rules: Robots can only perform actions explicitly listed in their capabilities.
R2's capable subtasks are limited to
(1) 'get (target_obj)* which involves fetching
+ something from the kitchen
(2) 'put away (target_obj)", which involves putting
+ away something from the user's location back into
4 the kitchen.
Rl's capable subtasks are limited to
(1) 'stir/mix', which involves stirring/mixing something
(2) 'hand over harget_objj4 which involves handing over object to the user
(3) 'pour (target_obj) at (location)', which involves pouring an ingredient into location
(4) stack (target_ob)] at (location)', which involves
somewhere. This somewhere' can also be referring to a food item (e.
stacking a target object
g. stack the lettuce at the burger)
(5) 'spread (target_obj] on (location)', which involves spreading target object object/ingredian inggeeieen at
62

---

7 somewhere. This 'somewhere' can also be referring food item (e.g. spread the honey on toast) Actions not present in their respective lists are
beyond their capabilities, e.g. none of the robot can
+ do subtasks such as 'prepare something', 'boil
something'.
You must make updates to the subtask queues based on the chat history, user input, and available subtasks. The update must not violate the robots' capabilities. The
updates must be one combination of following:
You should insert subtask into R2_subtask_quent or R1 #i__ubtask_queue if all conditions are satisfied:
the subtask is either proposed by user in
+ user_input'
in the subtask is within the capability of an agent the user agrees and gives permission in
4 user_input' to your proposal that is in the
4 chat chat_histozy'.
You should insert the subtask into
user_subtes quubtask_quuue' if all conditions are satisfied
the subtask either in 'available_subbaskkk or
+ proposed by user in 'user_inpure
the user agreed to take over the subtasks
+ themselves in 'chat_history'
the subtask is not within the capabilities of any
+ agent, the user will handle it. You should
4 explain the situation to the user in 'reply' too
You should remove the subtask from
usse_ssubtask_uee _ueue'' and insert subtask into
if:
user explicitly indicates that she/he has
+ finished the subtask in 'user_input"
You should remove subtask from 'available_suttasks 4 if:
in 'user_input', user explicitly indicates that
+ they want no one to complete this subtask, or this task anymore.
don't want
You must provide reasoning for all the updates you have + made.
You must only reply the following JSON format
"reasoning"; < your_reasoning_geessheeree.>
"R2_subtask__ueue"" your_updatee____cueeeeeeees
"Ri_subtask__ueue"
>,
"user "Lser_subraskk_ueeee""
4 your__pddared_human htcpluuaaanuuuuueeeee seeeue_goes_heeeee >,
[complete___uttaakk_iittt
+ your__compleeee_aaak__uuuueeeee_
>,
"reply": your_reply_goee_haee
You should use double quote rather than single quote across
4 each subtask name.
You should choose Conffim Subbbakk if one of these is true:
* If the user didn't give instruction in 'user_input', and
4 there are subtasks in available subtasks, you can propose
63

---

4 without you asking them anything. They express some need subtask to solve 7 some subtask from the 'avallable subtasks' 1ist for the
robots perform later based on their capability (even
b they are running now).
If there are subtasks the 'available subtasks' list,
- but the subtasks cannot be completed by the robots. You
+ need to confirm with the user and ask the user to do that
4 subtask.
Even when everyone is working, if there are subtasks in
+ the 'available subtasks' list that the robots the user
+ can do, you can confirm that subtask with the user.
In pnse_input', the user initiated the conversation
and you think that you can propose
= that issue.
To execute this action, you must follow these rules:
The tasks you should confirm are:
the tasks in available subbasks
is only achievable by the user (none of the robots can it), you must - confirm the task for the user as soon as possible.
if task in available_subtase
the task suggested by user if user suggests any
You should confirm for all three possible agents: R2, R1,
+ users if following conditions are met:
You should confirm task for R2 conditions:
satisfies
the task contains actions that falls under R2's
4 capable subtask list
* it is either the first task fell under R2's saallable_ubbtaaaee + proposed by the user now.
+ capablities in the
the task
You should confirm task for R1 if it satisfies conditions:
the task contains actions that falls under R1's
+ capable subtask list;
it is either the first task fell under Rl's
+ capablities in the available_subtaske or the task proposed by the user new.
You should confirm task for user if it satisfies
conditions:
neither Rl or R2 can perform the task; is the first task in the available_subtest that + out of robots' capabilites. You should suggest users to let you know when they finish the task.
1) In the "reply" field, confirn with the user whether
4 they agree with R2 R1 to proceed on specific task
The reply should be in this format: "SR21 (R2/R1) go
(task _ame) for you?"
2) You are not allowed to modify either R2 or Rl's
subtask queue.
3) You must reply in the following format
{
"reasoning": < your-reasooing--ge---teer >, "reply": your- goes here
You should choose 'No_op' if one of these is true:
* If the aavailable subtasks' list is empty [], you should wait and nothing.
64

---

If the user does not say anything currently, so user_inpur is empty
To execute this action, you must follow these rules:
1) You must reply in the following format
"reasoning"; < your- goes-here here >
You should choose IIterruptt Subtask' if one of these is true
+
When the robot's s status is at Running, user explicitly
requests to stop one of the robot from doing their
4 current tasks.
when the robot's status is Running, the user mentions an emergent accident that it's s unsafe for robots continuing doing current tasks.
To execute this action, you must follow these rules:
You should only update R2/R1 status into "Killed" based 4 on following quidelines:
When the robot's previcus status at 'Running', the
- user explicitly zequests to stop one of the robot from doing their current tasks.
when the robot's s previous status is at 'Running', the
user mentions an emergent accident that it's unsafe
+ for robots continuing doing current tasks.
You should also move the subtask that is at the robot's
current subtask into the completed subtask list.
Your response must follow this json format:
"reasoning"; "< your reasoning here >",
"R2_status": "< put _at_R2_status_hhe here >", "Ri "Ri_status"; "< put_Rl_stataee here >"
"< put the subtask that got
stopped here >" "reply": < your_reply_goes_hers
You should choose 'Overall if the user gives
irrelevant vague information or asks for clarifications
+ of your previous actions
To execute this action, you must follow these rules:
1) The user gives irrelevant or vague information or asks for clarifications of your previous actions, you
+ should reply to ask for clarification with your
knowledge about what recipes we have in the list.
2) Your reply should be based R2 and Ri's task queues, chat history, status, user input and recipe. The chat history could give information
the recent tasks
that have been performed.
3) You must reply in the format that
{
"reasoning"; < your-reasoning-gos--eee
"reply": < your-reply-gess-eer.
Recipe List: <recipes>
Here are the robot's capability that you must adhere <robot_capabllttes> examples:
65

---

description: user gives recipe with exact match
observation:
recipe name:
chat_history:
User:
Hey Mosaic! I want to make bibimbap for dinner tonight.
user_input: "Hey Mosaic! want to make bibimbap for dinner 4 *onighh.."
response:
"reascning": "The user clearly says that they would like to 4 make bibimbap, which exactly matches the bibimbap in the 4 recipe list. can confidently suggest the bibimbap recipe > from the 1ist it closely matches the user command." "recipe_name": "Bibimbap", "reply" "Let's make bibimbap!."
description: user gives recipe with no match but mentions + ingredients
observation:
recipe name:
:hat_history:
User: I just bought rice, vegetables, and gochujang. Let's s > make rice for dinner. user_input: "I just bought rice, vegetables, and gochujang. Let + 's make rice for dinner."
response:
"reascning"; "The user says that they have the ingredients of
4 rice, vegetables, and gochujang. These ingredients match
4 those of bibimbap in the recipe list. can confidentaly
+ suggest the bibimbap recipe from the list as it contains all + of the ingredients the user mentioned.",
"recipenname" "Bibimbap",
"reply"; "Bibimbap in the recipe list matches your description
4 Let's s cook bibimbap together."
description: user gives recipe with no match but ingredients similar
observation:
recipe name: ""
chat that_history:
User: want something cold for lunch.
Mosaic: Do you want caesar salad, fruit salad, or poke? Sure, would love to eat poke.
User:
user_input: "Sure, I would love to eat poke." response:
"reascning": "I have provided the user with options cold
4 lunches and the user says that they would like to make poke,
4 which matches the poke in the recipe list. I can
4 confidently suggest the bibimbap recipe from the list it
+ closely matches the user command.", "recipe_namee : "Poke",
"reply" "Let's s make poke then!"
description: alternatives that go with original dish observation:
recipe name: "" Shat_history.
66

---

"reasening": "The user did not suggest specific recipe. They > want something light, which could be salads within my recipe User: Hey Mosaic! I want to make a burger.
user user_input: input: "Hey Mosaic! want to make burger." "
response:
"reascning"; "There is no recipe for burger. However, the 4 sandwich in the recipe list (sandwich with mustard, hot dog) will suggest those.
+ are good alternatives,
"reply": "I don't have burger in my recipe list, but we can + cook sandwiches! How about hot dog tuna sandwich?"
description respond to general recipe list observation:
recipe name: ""
chat_history:
User: I want something light for dinner today. What do you 4 suggest?
user_input: "I want something light today. What do you suggest 4 ?"
response:
4 list (tossed salad, caeser salad) I will suggest salad. ", "reply": "Salad can a great light dinner. Do you want make a 4 tossed salad or a caeser salad?"
description: respond to broad command
observation:
recipe_name: ""
Chat History:
User: Let's make lunch. am in the mood for a vegetable dish +
user_ iier_input: "Let's s make lunch. I am in the mood for a vegetable
4 dish."
response:
"reascning"; "The user input is quite broad and does not
4 specify particular dish. Since the user looking for a
4 vegetable dish, I can suggest some alternative recipes from
4 within the recipe list that matches the general criteria of + being vegetable-based. Caesar salad, Mixed Vegetable Soup,
+ and Tossed Salad are vegetable dishes from the recipe list, alternative.",
4 so I can propose those
"reply": "Caesar salad, Mixed Vegetable Soup, and Tossed Salad
4 are great vegetable dishes. Would you like to make one of
4 those for lunch?"
description: respond to broad ingredient list
observation:
recipe_name:
Chat History:
User: have bunch of lettuce. What should we make for
lunch?
user_input: "I have bunch of lettuce. what should make for 4 lunch?"
response:
67

---

"reascning"; "The user input is quite broad and only mentions + lettuce but does not specify particular dish. Since the
A user is looking for a recipe with lettuce, can suggest
4 some alternative recipes from within the recipe list that
4 matches the general criteria of being vegetable-
+ Caesar salad and Tossed Salad are recipe which contain
+ lettuce from the recipe list, so I can propose those as an
+ alternative.",
"reply": "Caesar salad and Tossed Salad contain lettuce. Would + you like to make one of those for lunch?"
description: user agrees to R2's 8 task, and say what they will do observation:
("get pepper", "get salt", "chop carrots" 4 put carrots into pan")
R2_subtask_gueue: []
Rl_subtask_queue ("stir food"]
user _eer_ssbtaskko _uessk []
completed_subtask_IIne fobtask_listt []
chat_history:1
User: want to finish all carrots the fridge tonight.
Mosaic: Do you want to cook baked carrots?
User: Great
Mosaic: SR21 R1 stir food for you?
User: yep, that's s great
Mosaic: R1 will start stirring
Mosaic: SR21 R2 get pepper for you?
User: Ok. will chop carrots then.
user_input: "ok. will chop carrots then. response:
"reascning": "In 'user_input', user first agrees. Based on 4 chat_history', can understand user agrees to my proposal
4 of letting R2 to 'get pepper'. Thus, I should add this subtask into R2_ R__subaask_queue Also in 'user_input', the
+ user has specified to chop carrots themselves, so I will add 4 it to user sustask_suueeeeeeeeee
queuee ["get pepper"), "R1 [RI_subtask_queuse ["stir food"],
"user_suu htt___uuttss_uuueeeeeeeeeee ["chop carrots"),
eccmpleted_auuttssk_Lc
[],
"reply": "R2 will go get pepper. Thank you for chopping carrots
description: prioritize user input over available subtask queue + add into robot queue observation:
available_subtasks: ["get salt", "stir the pan", "wash carrots + "get oil", "put pan into oven"]
R2 =2_subtask_queue: ["get roasted chicken"]
R2 #2_current_subtask:: "get plates"
R2_status: "Running"
Ri_subtask__ueue: ["stir"] user user_subtask_queus ["put pan on table")
[]
chat history:
68

---

Mosaic: None of the robots can put on table. Can you help + me do it and let me know when you finish?
User: Yes, I can da that.
Mosaic: SR21 R2 go get plates for you? Can R1 stir for you?
User: Sweet.
Mosaic: SR21 R2 go get roasted chicken on the plate for you?
User: Perfect.
Mosaic: SR21 R2 go get salt for you.
User: wait, go get the oil bottle first.
user input: "wait go get oil bottle first"
response:
"reascning": "Based on 'chat_history', have proposed R2 to go 4 get salt. But user refuses in juser_input: and ask R2 to go
4 'get oil bottle' right now. I should prioritize user's
4 instruction,
should add 'get oil' into the first 4 mR_subtask_queue. Rest of queue stays the same.", ("get oil", "çet roasted chicken"), "R1_a [al_subtask,o subtask_queue"; ["stir"],
"R2 _r2_subtask__we
"user_ "user_subtast" subtask _erckk_uueee: ["put pan on table"),
sublletsd_subtts___ssss ": [],
"reply": "R2 will go get oil for you."
description: assign 'put away' task observation:
available_subtask:: ("get pepper", "prepare vegetables")
R2_subtask_queue: []
R1 a1_subtask_queue []
user stt___uuassk___u_ queue: []
cumpleted_subtaskte
[]
chat_history:
Mosaic: Can you prepare vegetables because none of the robots 4 can do it? Can R2 get pepper for you? User: will go prepare vegetables. Can R2 put away this cup 4 first? Then, it can go get pepper.
user_input: "I will go prepare vegetables. Can R2 put away this 4 cup first? Then, it can go get pepper.
response:
"reascning": "User has explicitly indicated that they will go
4 prepare vegetables, so we can update user sar__suuttss_quuuee with
4 this. User asks R2 to put away a cup first, so we will add the R2_subtask_qun
4 the subtask 'put away cup'
first.
4 Then, because user also agrees R2 can go get pepper, will
4 add 'get pepper' after 'put away cup' in the
4 R2 su_ssasas___uuee
["put away cup", "get pepper"), "R1 "Ri_sub wwwssssuuuuuuuuuuuuuuss queue"; [],
"Ro_subtask_gueue""
"user_ sattersssbtaakk_ee "uobues ("prepare vegetables"),
Semppleted_sutask__Li
": [],
"reply": "Great. Thank you for preparing vegetables. R2 will 4 put away the cup then go get pepper.
}
description: user finished the proposed task add into + completed task list
observation:
awailable_subtaslee ["get salt", "open refrigerator", "get butter
4 ", "put butter over a plate"]
69

---

R2 _2_subtask_queue: ["get roasted chicken"]
R2 Rd_cuzrent_subtat:: "get plates"
R2_status: "Running"
Ri_subtask_queue: []
user_subtask_gqeee:: []
complette__subraak_kiitttt []
chat_history:
Mosaic: SR21 R2 go get plates for you?
User: Sweet.
Mosaic: SR21 R2 go get roasted chicken for you?
User: Perfect.
Mosaic: SR21 R2 go get salt for you.
User: nah got salt already.
user_input: "nah I got salt already." " response:
"reascning: "In 'chat_history', I have proposed to let R2 go 4 get salt, but in 'user_ 'user_input' the user says they have
4 finished the subtask. Thus, we no longer need R2 to do this 4 and I don't t need to change R2 subtask _ueuee queue. should add Rest of the
S get salt' into updated
+ queue remain unchanged.",
"R2_subtask_queee" ["get roasted chicken"),
"R1 "al_subtask_queue" [],
qnser_subtaskkquee
[],
[rompleted_subtask__ie ": ["get salt"),
"reply": "Great!"
description: user finished task observation:
available_subtastt ["get pepper"]
R2_subtask_gueue: ["get tomato"]
Rl_subtask_cueue: ["stir food"]
user_subtask__qeeee: ["pour water into pot", "turn on oven"]
completed_subtask__ll chat_history:
Mosaic: None of the robots can pour water into pot. Can you + help me on that and let me know when you have finish this?
User: Ok.
Mosaic: None of the robots can turn on oven. Can you help me - on that and let me know when you have finish this?
User: I will.
Mosaic: SR21 R2 go get tomato for you?
User: Ok. That will be easier.
User: I have poured water into pot.
user_input: "I have poured water into pot."
response:
"reascning": "User has explicitly indicated for finishing the should add the subtask ["get tomato"],
4 subtask pour water into pot,
+ into updated
"R2_ [t__aubtask_queut
"R1 "Ri_ssbtaskk_ueet queuee: ["stir food"], "user_ biterr_subtas__qeeeeee
["turn on oven"], ": ["pour water into pot"], "reply": "Great. Thanks for letting know." "
[completed__ubbass__.l
description: user cancel task
70

---

observation:
available_suttask:: ("get pepper", "stir"]
Rt_subtask__qeee: []
Ri_subtask_queue: []
user_subtask__qeee:: []
compleete__subbaakkkiitttt []
chat_history:
Mosaic: Can R2 get pepper for you? can R1 stir?
User: R1 can stir. I don't want any pepper. Can R2 get
+ salt instead?
user input: "Ri can stir. don't want any pepper. Can R2 get 4 me salt instead?"
response:
add that 4 to Rl's queue. However, user doesn't t want 'get pepper', so
"reascning": "User agrees that Ri can stir,
4 we will put it in updated complettd__sbbaak__iii
so that no 4 one would do it. We will assign user proposed 'get salt' to + R2 instead.",
[R2_subtask subtask_queue". ["get salt"),
"Rl subtask _ueee" ["stir"], "user_ seteer_ssbtass_ueeeeeee cempleetd_subbaskke
[""],
": ["get pepper"l,
"reply": "R1 will stir for you. R2 will go get salt now.
description: confirm with R2 & user observation:
available subtasks : ["get salt", "get chicken", "chop carrots"]
available_subtasks
R2 R2_ ___subtask_queue: []
Ra_cuuzent_subtask:: "get pepper"
R2_status: "Running"
Ri_subtask_cueue: []
Rl_current_subtast: "stir soup"
Ri_status: "Running"
user_subtask_qeee: []
chat_history:
Mosaic: SR21 R2 go get pepper for you? SR21 R1 stir soup for 4 you?
User: Great
Mosaic: R2 will go get pepper for you now. And R1 will go 4 stir now.
user_input:
response:
"reascning": "Since user input is empty, the task we want to
4 confirm now is the tasks in the available subtasks quede.
4 Get salt' fells under R2's capabilites, so we should confirm
4 this task for R2. Since
robots can chop, we will ask 4 whether user can assist in chopping. ",
"reply": "SR21 R2 go get salt for you? Can you help chop
4 carrots since neither two robots can chop?"
description: user request has higher priority over task queue.
observation:
awailable_subtast ["stir food in pan", "get carrots", "get 4 chicken",]
R2 subtask queue: []
71

---

"reascning"; "The caallable_subtasee only has 'prepare lettuce 4 ' which none of robots can do. We need ask the if + they can do it. ", "reply": "None of the robots can prepare the lettuce. Can you R2 St_ourient_subtask: "get salt"
R2_status: "Running"
Ri_subtask__queee []
Rilcurrent_subtask: "stir soup in pot"
Ri_status: "Running"
user_subtask_queee: ("prepare vegetable"]
chat_history:
Mosaic: Can you prepare the vegetables because none of the robots do it.
User: Alright. I can do that.
Mosaic: SR21 R2 go get salt for you? SR21 R1 stir soup in 4 the pot for you?
User: Great.
Mosaic: R2 will go get pepper for you now. And R1 will go stir
User: I need chopstick.
user_input: "I need chopstick."
response:
"reascning"; "User suggests the need for chopsticks and this is
4 the task not appearing in the available wwailable_subttsss. subtasks. Thus, we
+ need to prioritize the user's needs over mnconfirmes tasks
+ the available subtask queue. Since the action 'get' falls
4 under B2's capabilities, you should confirm this task for
+ R2. We can also assign tasks for R1. Since R1 can perform
4 action 'stir',
should confirm task 'stir food in pan'
+ for R1, Right now user has subtasks to do, so we don't need
4 to confirm new subtasks with them yet. ",
"reply": "SR21 R2 go get chopsticks for you? SR21 R1 go stir 4 food in pan for you?"
description: ask user to do a subtask observation:
available_oubtasks = ["prepare lettuce")
R2_subtask__ueue: ["put away mustard")
R2_current_subtask:: ""
R²_status: "Idle"
R1 Rl_subtask_cueur []
Aloourrett_ubbaak: ""
Rl_status: "Idie"
user_subtask_queee: []
chat_history:
Mosaic: Can R2 get pepper for you? Can R1 pour some salt on 4 the burger patty?
User: Awesome! Thank you.
Mosaic: R2 will get pepper now. R1 will pour some salt
4 the burger patty.
User: finish this mustard. Can B2 put it away for me.
Mosaic: R2 will go put away the mustard now.
iser_input:
response:
4 do that and let me know when you are done?"
72

---

description: oo when there's s no available subtasks
observation:
recipe_name: "Ceasar Salad"
available_subtasks: []
R2_subtask_gueue: []
R2_status: "Running"
t "get ranch sauce"
Rl i__subtask_queue: []
RI_status: "Idie"
Rl a__uuzent_subtak:: ""
user_subtask__ueee:: l'prepare romaine lettuce') ]
complatte__subbaak__lsss chat_history:
User: want to make ceasar salad with you.
Mosaic: Sounds great!
Mosaic: Can R2 get ranch sauce for you?
User: Sounds good!
Mosaic: will go get ranch sauce for you now.
Mosaic: None of the robots can prepare romaine lettuce. Could
you do that and let me know when you are done?
User: Ok.
Mosaic: Thank you.
user_input:
response:
"reascning": "The 'availkblleauutasse 1ist is an emtpy list []
4 and no new 'user_input', I should just do nothing for
4 now."
examples:
description: interrupt R2
observation:
current recipe: "Broccoli soup"
available subtasks: ['get salt', 'pour water into pot', /'et
4 pepper'] R2's s subtask queue: []
R2 current task: "get broccoli"
R2's status: "Running"
Rl's subtask queue: []
RI current task: "
R status: "Idle"
User' s subtask queue: []
Chat History:
Mosaic: SR21 R2 go get broccoli for you?
User: Good.
Mosaic: R2 will go get broccoli.
User: R2, don't go.
User Input: "R2, don't go"
response:
"reascning": "User explicitly has asked R2 to stop doing its current task get broccoli, so we should update the status 4 of robot R2 into Killed. We should move "get broccoli" to lis.siittttttteeeeeee.....ie
Since user didn't provide instruction 4 for R1, then we shouldn' modi fy status of R1.",
"R2_ status"; "Killed",
73

---

"Rl_status": "Idle", poompleted_subasee list": ["get broccoli"),
"reply": "R2 will no longer get broccolic
}
description: interrupt R1
observation:
current recipe: "vegetable Soup"
available subtasks: []
s subtask queue: []
R2 current task: "qet salt"
R2's status: "Running"
Rl's s subtask queve: []
Rl current task: "stir soup"
Rl's s status: "Running"
User's subtask queue:
l'prepare vegetables']
Chat History:
Mosaic: Can R1 stir for you? SR21 B2 get broccoli for you?
User: Good.
Mosaic: R1 will start stirring. R2 will get broccoli.
Mosaic: Can you prepare the vegetables? None of the robots > can do it.
User: Sounds good.
Mosaic: Thank you. I added that to your queue.
User: finished.
Mosaic: Great job preparing the vegetables.
User: R1 stop stirring. want you to handover the salt next
4 to you.
User Input: "Ri stop stirring. I want you to handover the salt 4 next to you.
response:
"reasoning": "User explicitly has asked R1 to stop doing its should update the status of + robot R1 into Killed. We should move "stir soup" to Since user didn't provide 4 instruction for R2, then we shouldn's t modify status of R1.",
4 current task of stirring,
4 completed_subtask_kkle
RR2_status": "Running"
"Rl status": "Killed",
"completed__ubtssk___ subtask list"; ("prepare vegetables", "stir soup"), "reply": "Rl will stop stirring.
description: vague user input before starting the recipe
observation:
recipe_name:
awailable_subtaskss []
R__subtask_gueue: []
R2 current task: ""
R1 __task_queue []
R1 current task: ""
chat phat_history:
User: The weather is bad outside
user_input: "The weather is bad outside" response:
"reascning": "The user input is vague and doesn't include 4 anything about cooking.",
74

---

"reply": "I am sorry over that. Let me know if you want to cook 4 anything for dinner! can help!"
description: vague user input while making the recipe
observation:
recipe_namer "Tomato Soup"
ttailabbe_subbaskss ("cut tomatoes", "put tomatoes into the pot + "stir"]
R2_subtask_queue ["get pepper"]
R2 current task: "get salt"
Rl_task_queues []
Rl current task: ""
chat_history:
User: love tomatoes much.
user_input: "I love tomatoes so much." response:
"reasoning": "The user input mentions tomatoes which pertains
4 to the recipe but does not mention anything about cooking.
+ The next task in subbass is cut tomatoes, which
+ is not a capability of or R2 so I should ask the user if
+ they would like to proceed with the next task based on the
+ chat history and available_subtasee
"reply": "I happy to hear that. Do you want to cut the 4 tomatoes?"
description: unrelated vague user input observation:
recipe_name: "Tomato Soup"
wwailable_subtasks ["cut tomatoes", "put tomatoes into the pot - "stir"]
R2 R2_subtask__ueue: ("get pepper"]
R2 current task: "get salt"
Rl_task_queuer []
RI current task: ""
chat chat_histone history:
User: love potatoes so much.
user_input: love potatoes so much." response:
"reascning"; "The user input mentions potatoes which does not
4 relate to the recipe and does not mention anything about
4 cooking. I should proceed by clarifying the confusion with
+ the user based on the chat history.".
"reply": "Sounds good. Should we continue to make tomato soup 4 or would you like to try making potato salad?"
75


Reviews:
[{"review_id": "joTsIl8lf5", "reviewer": "Reviewer_MKTe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a modular system, MOSAIC, to coordinate multiple robots and help humans finish multiple kitchen tasks. It mainly has three components: an interactive task planner, human motion forecasting, and visuomotor skills. Through experiments, it show that the proposed system performs better than a baseline method which only uses one single forward query of the LLM.", "summary_of_recommendation": "The paper focuses on using foundation model systems for human-robot collaboration, which is novel. However, the paper’s technical contribution is overclaimed, and the methodology section needs more clarification regarding the design principles and significant details.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- What are the principles used to design the modularization architecture? Considering the problem from a hierarchical perspective, the task planner handles high-level planning while visuomotor skills focus on the low-level aspects. How should the human motion forecasting module be positioned within this hierarchy? Additionally, there are sensor-specific modules like object detection. Are these sensor-wise modules included in the authors’ modularization framework?\n - Regarding the interactive planner, how are subtasks assigned to different robots and humans? \n - Does the language model first generate the sequence of actions? Are the subtasks shown in the upper right corner of Figure 3 predefined? \n - Do the robots need to obtain permission from humans first before executing these actions?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jDhtZNbT3V", "reviewer": "Reviewer_34B7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces MOSAIC, an interactive, modular system for assistive cooking. MOSAIC utilizes 3 components: a LLM and DAG-based task planner, a human motion forecasting module, and motion primitives. Using this hierarchical architecture, MOSAIC is able to interact with humans for high-level task planning with natural language while also interacting physically for tasks such as hand-over. Results show MOSAIC is capable of cooking six predefined recipes with a human over 60 trials, as well as a clear evaluation of each module’s performance and contributions to the overall system.", "summary_of_recommendation": "I chose my recommendation because the system demonstrates impressive capabilities with expert users, but leaves the question of whether the system is able to adapt to a novice user unanswered.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- From Table 1, why were different skills implemented so differently (learned with RL, hand-engineered, and planning-based)? This requires access to more information about the environment, such as a map for planning or access to a simulated environment for the RL training. \n\n- Were there any experiments evaluating the ability to add new tasks/”recipes” to the system? The reviewer is concerned about generalizability of the system in an interactive setting. \n\n- Most critically, how were the interactive and assistive qualities of the system evaluated? Was a user study conducted with the full system? A N>5 user study with users not involved in building MOSAIC would greatly improve support for the claim of interactivity and assistance. Did users feel the robot was helpful? Would users want to use this system again for cooking?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3cmyfxBOTn", "reviewer": "Reviewer_ibvm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents MOSAIC, a modular architecture for coordinating multiple robots to interact with users using natural language, and manipulate an open vocabulary of everyday objects to solve complex cooking tasks. The main contributions of this paper are: 1) a modular cooking assistant that use LLMs for interactive task planning, VLMs for visuomotor skills, and motion forecasting models to predict human intents for collaboration, 2) perform comprehensive evaluation of MOSAIC, either end-to-end, or per-module, 3) draw insights and lessons learned.", "summary_of_recommendation": "I would recommend weak accept since despite the room of improvement mentioned above, I believe the community still benefits a lot from reading and discussing this work due to its thorough experimental evaluation.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- How general is the proposed method? What are all the assumptions that the method has?\n- How does the task planner break down the overall task into subtasks? Is it via LLM zero-shot capabilities, or are the recipes provided to the system?\n- Pick skill still seems like a major bottleneck for the overall system. Any idea on how to further improve it?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:

- Interesting problem of multi-robot and human coordination, using LLMs for natural language interaction
- Modular architecture, integrating many different SoTA models from literature from object detection, segmentation, human action detection. Impressive overall system, with large number of real-world evaluations.

Weaknesses:

- The interactive talk planner lacks details, including exact prompts used. Please provide details either in the main paper or appendix, as requested by reviewers. Also a discussion on how the planner can be adapted to related but unseen tasks, like a different recipe, or unseen environment, would be helpful.
- While the system is impressive, it seems a little specific to the problem at hand. A better motivation of the system design would help future researchers modify it to other problems. Additionally, please clearly list out the assumptions in the system.
- The real-world experiments are conducted with a small set of expert users, which raises questions about how well the approach generalizes to other users. Including experiments with novice users would make the claims about interactivity and assistance stronger.

Post rebuttal:
Authors have clarified the assumptions and details of the approach. They have also provided a description of the approach's ability to generalize to new tasks, and presented a larger user study with novice users. Overall this is a good paper.
Decision: Accept