MOSAIC: Modular Foundation Models for Assistive and Interactive Cooking

Authors: huaxiaoyue  wang
kushal  kedia
juntao  ren
rahma  abdullah
atiksh  bhardwaj
angela  chao
kelly y chen
nathaniel  chin
prithwish  dan
xinyi  fan
gonzalo  gonzalez-pumariega
aditya  kompella
adrian  pace
yash  sharma
xiangwan  sun
neha  sunkara
sanjiban  choudhury
caesar  salad
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 07:07:49

Abstract: We present MOSAIC, a modular architecture for coordinating multiple robots to (a) interact with users using natural language and (b) manipulate an open vocabulary of everyday objects. At several levels, MOSAIC employs modularity: it leverages multiple large-scale pre-trained models for high-level tasks like language and image recognition, while using streamlined modules designed for low-level task-specific control. This decomposition allows us to reap the complementary benefits of foundation models and precise, more specialized models, enabling our system to scale to complex tasks that involve coordinating multiple robots and humans. First, we unit-test individual modules with 180 episodes of visuomotor picking, 60 episodes of human motion forecasting, and 46 online user evaluations of the task planner. We then extensively evaluate MOSAIC with 60 end-to-end trials. We discuss crucial design decisions, limitations of the current system, and open challenges in this domain

Main paper: 
 introduction: 
 Collaborative tasks in household environments present significant challenges for robots. Consider the scenario in Figure 1 , where a human user collaborates with two robots to prepare a meal. We'd like for communication with the system should feel natural to the user. Furthermore, each robot should be able to complete auxiliary tasks across a wide range of objects while fluidly collaborate with the humans. Prior systems in this domain [1, 2, 3, 4, 5] have demonstrated impressive capabilities. However, they either they operate in isolation and lack meaningful collaboration with humans, or employ highly scripted behavior. In this paper, we aim to overcome both of these limitations by designing a system that fluidly collaborates with humans and performs a wide range of tasks.While a single end-to-end model works well for tasks like language understanding where large amounts of data are available, such an approach is difficult for collaborative robots, where less data is available and precise control is important. Our key insight is that by modularizing our architecture, we can segment out parts of the framework that require broad generalization, such as language and image recognition, from the portions that require task-specific control. Furthermore, we can triage end-to-end system failures to a specific component, enabling efficient system improvement.We operationalize this insight to create MOSAIC (Modular System for Assistive and Interactive Cooking): an architecture that applies modularity at multiple distinct levels to significantly improve the overall system's performance. Each module has a well-scoped task which is simpler to complete Figure 1 : MOSAIC cooking in the kitchen. MOSAIC interacts with a user via natural language and controls a tabletop manipulator (R1) and a mobile manipulator (R2) to prepare vegetable soup with the user. and results in fewer overall mistakes. While the principle of modularity has been central to developing robust real-world robotic systems, these modules have historically been robot/task specific. In contrast, our architecture integrates general-purpose pre-trained models to solve robotic tasks. This design choice enables us to build a system that is flexible, interpretable, and scalable.Our contributions can be organized into three groups:1. We architect MOSAIC: a full-stack Modular Cooking Assistant. We present a novel framework for home robots that integrates multiple large-scale pre-trained models. In particular, we use large language models for interactive task planning, vision language models for visuomotor skills, and motion forecasting models for predicting human intents for collaboration. We detail in Section 2 the key design decisions we made to ensure our system is scalable and interpretable.2. We perform a comprehensive evaluation of MOSAIC. We extensively test the limits of our system through 60 end-to-end trials where two robots collaborate with a human user to cook complex, long-horizon recipes (Section 3.2). We also test individual modules with 180 episodes of visuomotor picking (Section 3.4), 60 episodes of human motion forecasting (Section 3.5), and 46 online user evaluations of the task planner (Section 3.3) 3. We analyze both MOSAIC's successes and failures to derive actionable insights for the field. The modular nature of our architecture lends itself to error diagnosis -for each failure, one can clearly pinpoint which component failed and why it did. To this end, we distill key findings from our evaluations into limitations and exciting directions of future work based on our current system. 
approach: 
 We present MOSAIC, Modular System for Assistive and Interactive Cooking, a modular architecture that combines multiple large-scale pre-trained models to solve collaborative cooking tasks. Fig. 2 shows the three main components of MOSAIC: 1) Interactive Task Planner (2.1): a module that interacts with real users via natural language to plan a diverse set of tasks and coordinate subtasks during the cooking process; 2) Human Motion Forecasting (2.2): a module that leverages motion forecasting models to predict human motion such that robots can seamlessly collaborate with humans while maintaining a margin of safety to avoid human-robot collisions; 3) Visuomotor Skill (2.3): a module that generalizes robot skills to a diverse set of kitchen objects and environments.We make a set of simplifying assupmtions in our work, which we detail in Appendix B.The goal of the task planner is to continuously interact with a human user using natural language, delegate subtasks to different robots or the user, and monitor progress. After agreeing upon a task with the user (e.g. "Prepare vegetable soup"), the task planner uses an LLM and an online recipe to represent the task as a direct acyclic graph (DAG) to model temporal dependencies between Figure 2 : MOSAIC System Overview. The Interactive Task Planner module communicates with the user via natural language to decide on a recipe. It assigns subtasks to each robot accordingly. The Human Motion Forecasting module extracts and converts the human's 2D post to 3D coordinates, which it uses to predict future human motion. Separately, a VLM takes image and language as input and produces a 3D grasp pose around the object of interest. Combined, all three are taken by the execution policy of the Visuomotor Skill module to produce a final robot action. different subtasks. From this representation, the task planner assigns and maintains a queue of subtasks for each robot based on the current agents' status and the user's requests. To execute a subtask (e.g. "fetch salt"), the task planner generates a code snippet that issues a series of API calls such as go to("pantry"), pick("pepper"), etc. The initial set of subtasks and their mutual dependencies is generated once after the recipe is decided upon, the task planner can reassign subtasks and accept new ones based on the user's input.While many recent approaches [1, 6, 7, 8, 9] directly use LLMs for task planning, we observe two main challenges. First, due to the large action space, the LLMs tend to violate constraints that the developer specifies even with chain-of-thought prompting [10] . Second, this approach requires all constraints to be specified in one monolithic prompt, which gives the developer little control over the LLMs' behavior and is challenging to debug and scale. To overcome both challenges, we propose an architecture that embeds LLMs within a behavior tree (BT) [11] (as shown in Figure 2 ). The nodes break the entire reasoning process down into easier reasoning process for LLMs to think about, thereby reducing the complexity and potential error rate of the LLMs and making it easy to scale to multiple behaviors. Finally, adding a new behavior is as simple as creating a prompt for that behavior and adding it as an option for other behaviors to invoke. No change to the code is necessary. Further details in Appendix C and exact prompts for all nodes are listed in Appendix I.Seamless and fluid coordination with humans while maintaining a margin of safety requires forecasting human motion. However, accurately forecasting human motion in dynamic environments such as kitchens is challenging, as humans can perform a wide range of motions, such as manipulating various objects in the kitchen or moving between stations. Even with large amounts of training data and a long context window, current state-of-the-art models struggle to accurately predict human motion at all times. We aim to build a forecasting model that generates predictions that sufficiently capture the impact of forecasted human motion during interactions with the robot.Training Pipeline. We first pre-train our model on AMASS [12] a large dataset of human activity, encompassing over 300 subjects and 40 hours of motion capture data. However, AMASS only consists of general single-human movements (e.g. jumping, walking, dancing), which are not representative of a human's motion when working in close proximity with a robot. To this end, we utilize the Collaborative Manipulation Dataset (CoMaD) [13] , a dataset consisting solely of humanhuman interaction episodes in a kitchen setting. For each episode in CoMaD, we identify these periods where both humans are in close proximity and construct a transition dataset. We sample data equally from the transition dataset and the entire CoMaD dataset to train the motion forecaster. This approach helps the forecaster maximize task efficiency by upsampling critical periods of the interaction when the human is likely to approach and interact with the robot.Inference Time: Real-time, Vision-based Forecasting and Planning. Given an RGB-D image of the human, we use MediaPipe [14] to extract the 2D positions of their upper-body joints. These locations are then back-projected to 3D world coordinates using the image depth map and used to generate real-time motion forecasts for robot planning. However, the 3D coordinates obtained using the RGB-D camera are noisier than the high-fidelity motion capture data which the model is trained on. By injecting random Gaussian perturbations into the model's input at training time, we force the forecaster to learn to denoise potentially noisy input and generate smooth forecasts. takes up the highest proportion of calls, and requires a policy that satisfies a tight set of constraints. Thus, we learn the policy via reinforcement learning (RL). go to() assumes access to a map of the environment, and plans a path between the given start and goal.SkillThe visuomotor skills module takes the goal specification from the task planner, forecasts from the forecasting module, and observations from cameras and outputs a series of actions to complete the task. A common approach is end-to-end training on a suite of demonstrations [15, 16, 17, 18, 19, 20, 21] , though good test-time performance generally requires optimal demonstrations to have good coverage over the observation space, which are challenging and time-consuming to collect. Instead, we decouple perception and action generation into a set of individual modules, where different skills have the flexibility to use different combinations as shown in Table 1 . We describe these modules below, with more details in App. D.Object detection. Given image and language input, we obtain a set of bounding boxes from OwlViT [22] and take the bounding-box coordinate with the highest CLIP similar score [23] . We use FastSAM [24] to obtain a more accurate segmentation of the object within the bounding box, and back-project the segmented pixels through the depth camera's point clouds and take the goal location to be its center of mass.Action execution. For simple skills where there are no obstacles in the path between the robot and object-of-interest, we use an IK-based controller and a set of engineered primitives (e.g. a stirring motion), which take as input the object position and human motion forecasts to solve the task.For skills where such collisions may naturally occur during execution, we train an off-the-shelf reinforcement learning algorithm in a simulator that has access to an approximate robot dynamics model and a target grasp position. No action is executed in the simulator if the predicted action results in a collision, and instead gets a negative reward proportional to the distance from the goal. Further details on the motion primitives, simulator, and reward function are in Appendix D.Integration with human motion forecasts. Some skills use the current end-effector position and user's motion forecasts (from Section E) to avoid collisions with the human (e.g. user drops food into the pot while the robot executes stir()) and anticipate future positions (e.g. handover()).3 ExperimentsThis skill uses a map of the kitchen acquired beforehand and the internal localization mechanism of Stretch RE1 to navigate to designated locations around the kitchen. 
setup: 
 In all experiments, the mobile manipulator is a 6-DoF Stretch Robot RE1 [25] , and the tabletop manipulator is a 7-DoF Franka Emika Research 3 [26] . The kitchen has two overhead RGB-D cameras that can perceive the workspace and capture a human's motion. To allow users to interact with the task planner, we use Google's speech-to-text APIs [27] to transcribe user's verbal instructions and its text-to-speech APIs to vocalize the task planner's responses.The goal of end-to-end trials is to categorize how much failure each module has when the entire system is integrated and running together. To this end, we conduct a total of 60 end-to-end trials with Each recipe contains various subtasks involving different robot skills. We report the number of trials that are completed without any errors and the individual subtask completion rate. We also categorize the failure cases. MOSAIC is able to complete 41/60 tasks with an average subtask completion rate of 91.6%.two robots and a user collaboratively making 6 recipes. Each recipe involves a different combination of robot skills and different types of interaction with the user. 1 For example, users may provide vague instructions, interrupt a robot's subtask, and add new subtasks that are not in the recipe. Overall, MOSAIC completes 41/60 (68.3%) collaborative cooking trials of 6 different recipes with an average subtask completion rate of 91.6%.Modularity enforces each sub-module to have a clear input/output contract, allowing one to localize failures and extract transferable insights. We use this to cluster failures into the 6 categories as shown in Figure 3 . Specifically, errors originating from the task planner module usually come due to incorrect transcriptions of the user command from the text-to-speech sub-module. Errors in the perception module of visuomotor skills lead to an incorrect object identification or an insufficiently stable grasp. Likewise, tracking errors arise when the user moves outside the camera's view. At heart, for complex tasks with multiple humans and robots, where failures are inevitable, modularity makes it easy to triage and treat failures. In the following sections, we lift the insights from the end-to-end trials to each module and analyze how to limit errors therein.Since the task planner directly interacts with the users, a frequent failure mode is constraint violation (e.g. acting without permission). Thus, it is crucial for it to exhibit predictable behaviors, especially when planning over long task horizons. To this end, we first quantify the frequency in which each constraint gets violated when the task planner interacts with real users in an online user 2 . We compare the proposed approach, which embeds LLMs within a behavior tree (Tree), against directly calling the LLM once with one prompt (One-Prompt). The monolithic LLM prompt has constraints that it must follow, explanations of what actions to choose in each situation, and in-context examples.In the study, each user is randomly assigned to interact with either Tree or One-Prompt and is asked to analyze if the task planner changes subtasks without the user's permission (Act Without Permission), claimed robots can do subtasks beyond their capabilities (Lying), and did not respond to the user's instruction (Ignore User). We explicitly instruct participants to engage in non-nominal inter- actions with the task planner in hopes of eliciting a constraint violation. We received 26 responses from lab members who are not familiar with the task planner's capabilities and 20 responses from external users on Prolific [28] , a crowd-sourcing website. Quantitatively, LLM with behavior tree violates 47.8% fewer constraints compared to the baseline of using one LLM prompt (Figure 4 ). Furthermore, localizing failure modes via modularity sufficiently scopes down the problem such that we can programmatically evaluate the approaches on unit tests. Specifically we test whether the task planner properly handles a request and chooses the right action as the interaction becomes more complex (e.g. user always disagrees with task planner and reassigns subtasks). Tree remains above 90.0% for its unit test pass rate, while One-Prompt's performance drops from 100% (for easy cases) to 60.0% (for difficult cases), as the number of complex interactions increases. More details are in Table 9 in Appendix H. Overall, The results suggest that compartmentalizing the action space by adding explicit structure to each LLM's reasoning problem significantly helps the task planner to respect constraints.End-to-end trials showed imprecision in the perception module to be a common source of failure for visuomotor skills. Thus, we now focus our analysis on quantifying how the vision component influences policy performance and qualifying common failure cases. We place the mobile manipulator in front of a pantry with increasing number of objects and test the success of pick(), the skill with the highest utilization frequency across all end-to-end runs.Since localizing the error within the vision module, we found that directly using OwlViT [22] leads to rapid deterioration in accuracy when clutter increases, as the pre-trained model had difficulty identifying the correct bounding box from a large set of proposals. To rememdy this, we apply postprocessing via Non-Maximum Suppression [29] and CLIP [23] (abbreviated as OwlViT + CLIP). Table 2 shows OwlViT + CLIP increases skill completion by 70% in medium-clutter regimes, and 60% in high-clutter regimes. Critically, modularity helps to specifically identify the erring module, which allows improvements therein to have system-wide benefit. We find that a cluttered background and poor lighting conditions to lead to a suboptimal set of bounding boxes for CLIP to score, while more specific prompts produce better bounding box proposals.Additionally, figure 5 qualifies three failure cases of the vision module. First, the presence of too many objects (especially similarly-shaped ones such as various seasoning bottles) leads to a suboptimal set of bounding box proposals for CLIP to score. Second, when lighting and/or color blends the object contours into the background, only parts of the object may be included in the bounding box, resulting in a lower CLIP score. Lastly, imprecise prompts produce poor bounding box proposals.Finally, we analyze how using motion forecasts with noisy model inputs during training allow the module to be more robust during end-to-end trials through on-policy evaluations of a 7-DOF Franka robot arm collaborating with a real human user on two tasks.The stir() skill involes the robot stirring a pot while the human periodically adds in vegetables. We measure the time it takes the robot to detect that the human arm is reaching into the pot (TIME TO REACT (MS)), the minimum distance maintained between robot arm and human hand (SAFETY MARGIN (CM)), and the number of times the human hand comes within a minimum threshold of the robot arm (COLLISIONS). In the handover() task, the user asks the robot to pick up and handover objects. We measure the average time to complete the handover (TIME TO GOAL (MS)) and the movement efficiency (PATH LENGTH (CM)), which measures the distance tracked by the robot's end-effector.We compare against two baselines: (1) Current which assumes the current human pose will be its pose across the entire planning horizon, and (2) Forecast (Base) [30] that is not trained on noisy input. Each baseline and skill combination is evaluated 10 times for a total of 60 evaluations.First, we find that using forecasts during the stir() skill significantly improves on all metrics, maintaining a 74% greater SAFETY MARGIN from the human on average and, more importantly, avoids any collisions. In contrast, we observe that the robot reacts very late when using the human's current pose, and results in collisions 20% of the time as shown in Table 3 . In the handover() skill, the robot is 24% slower in completing the task following the current human wrist position compared to using the handover location predicted by our forecaster. Using the forecast, the robot moves directly toward the handover location, finishing the skill with 28% shorter trajectories.Next, we ablate on training with noisy inputs by comparing our forecaster with a baseline approach (Base) [30] that does not train on noisy model inputs. In the stir() task, our forecaster has 23% quicker reaction time to human movements. Further, there is more variability in the performance of the Base forecaster (measured by the variance of each metric) that can be attributed to greater sensitivity to noisy inputs. Similarly, in the handover() skill, the Base forecaster's predictions are often erratic leading to jerky movements by the robot arm. Following them is no better than using the current human position for planning, as measured by task completion time and path length. Overall, we find noise injections to the forecasting model make it more robust to perception errors, while using forecasts improves several key performance metrics of downstream skills.We assume that objects are not occluded for detection and grasping, though they can be next to each other. We also assume that the upper torso of the human is visible to the cameras for tracking and prediction.The place(<loc>) skill is parameterized by the target locations and completed with pre-coded motion primitives. 
discussion and limitations: 
 We decompose the overall problem of interactively cooking with a human user into a set of modules leveraging general-purpose pre-trained models. We localize errors in our system to individual modules and conduct targeted experiments. However, a number of open challenges still remain. First, improvements can be made to the task planner by grounding it with multi-modal input such as cameras and sensors. Further, expanding to new environments in a scalable and flexible manner may Table 3 : Task-Specific Performance Metrics. We evaluate the robot's interactions with the human user on 2 collaborative manipulation tasks. Integrating forecasts into the robot's skills improves fluidity and increases safety margin across all metrics. We observe that relying on the current human pose during REACTIVE STIR-RING is risky and results in collisions. ROBOT-HANDOVER tasks are more efficient using forecasting. require us to revisit previous assumptions and adopting new capabilities. Currently, the system's capabilities remain static after being deployed in an everyday user's household. An exciting area of future research is to continuously learn from real-time human feedback and interactions.We assume access to a library of robot skills that can be invoked with specific input parameters (e.g. pick up("salt"), stir()).Appendix C Interactive Task Planner Details The interactive task planner consists of three main components: a representation of a task and the dependencies of its subtasks, a mechanism to decide on a recipe and assign subtasks to others, and a medium to communicate to robots which skills to use for a given subtask. We implement these using a direct acyclic graph (DAG), a behavior tree, and LLM-generated code (communicated over ROS action services).Task DAG. The task planner represents a task (e.g. "Prepare vegetable soup") as a DAG, whose nodes represent subtasks of that task and whose edges represent dependencies between the subtasks. However, a DAG alone is insufficient for generating a task plan based on the user's instruction because it does not specify exactly which robot should complete which subtask.For a recipe, the DAG is generated by an LLM prompt ahead of time. Concretely, the LLM takes as input the ingredient list and step-by-step instructions, scraped from the recipe website. 3 It outputs a marked-down nested list of the recipe, which is easier for the LLM to reason about and generate in practice. A list item represents a subtask, and the nested structure represents the dependencies, so we can programmatically convert the LLM output into a DAG. The complete LLM prompt is in the supplementary material.The task planner also maintains a done state for each node/subtask in the DAG. To determine the available subtasks, we start from a root node whose done state is set, with outgoing edges to the first subtasks. Then, we follow each outgoing edge until reaching a node whose done state is unset and add it to a set. If no node is found through this process, the recipe has been finished. See Figure 7 for an example of a DAG for Caesar salad. We compare our approach against using one LLM prompt, which tends to violate constraints. Given the observation, the LLM with one monolithic prompt directly assigns subtasks to robots, which violates the constraint that it must confirm with the human before assigning tasks. Meanwhile, because our approach compartmentalizes the action space and reasoning process in a behavior tree, it is able to follow a correct reasoning path and correctly confirm its subtask proposal with the user.A DAG allows us to represent dependencies, such as, sequential: 'do A before B', and AND dependencies: 'do A and B before C'; it currently does not allow OR dependencies (do A or B before C). However, we could still create 32 unique recipes with this limitation.Your goal is to assign subtasks to robots based on what the user has confirmed in the chat history… You should choose a combination of these actions: assign(agent, subtasks), say(msg)For example, if the user has agreed to …, you can choose:assign("R1", ["fetch pepper"]) assign("R2", ["stir"]) say("Great, they will fetch pepper and stir the pot now.")What to do when making recipe?You should choose ONLY ONE of these nodes: ConfirmAssign, AssignSubtask, MarkAsComplete, Interrupt, NoOp You must respect these constraints when making the decision:-You must confirm your subtask assignment with the user before assigning… For example, once the user agreed to your subtask proposal, you should choose:Your goal is to decide which node to go to based on the current situation in chat history… Behavior Tree. We use a behavior tree to decide on a recipe and then assign subtasks to others by designing the tree around the behaviors we expect. Fig. 8 visualizes the behavior tree and shows an example comparing MOSAIC's task planner to the baseline that relies on a monolithic LLM prompt. Each behavior is encapsulated in a node, which represents a call to an LLM with a specific prompt and a pre-defined set of decisions to choose from. It takes as input an observation from the world and outputs arguments for high-level actions. For example, Fig. 9 shows snippets of prompts for different behaviors. The instructions describe the goal, the action space, which part of the observation to focus on, constraints to adhere to, and in-context examples. Assign Subtask is a leaf node that directly assigns subtasks to the robot and speaks to the user. On the other hand, the What to do when making recipe? behavior is a higher-level node that calls other behaviors, e.g. Confirm Assign, Assign Subtask, etc.More specifically, each sample from the observation space consists of: 1. recipe name 2. available subtasks 3. each robot's subtask queue, current subtask, and current status (Idle, Running, or Interrupted)4. user's subtask queue 5. completed subtask queue 6. user's current inputWe define this motion primitive for R2 (Franka arm) holding a tool (such as a ladle) in its arm parameterized by the target utensil where the action takes place (<obj>), for example, a pot. Further, this skill is responsive to the human's movements in the robot's stirring radius. If the human's motion forecasts reach into the robot's workspace, the robot stops stirring and makes space for the human to move in.5. pour(<obj>) Similar to the stir() function, this skill enables R2 to pour an already gripped object such as a salt can into a target receptacle (<obj>), such as a bowl. This process involves the utilization of motion primitives based on the estimated locations of the objects involved. Specifically, in the scenario of pouring salt into a bowl, R2 executes a sequence of actions: it first positions the salt can over the bowl at a calculated tilt angle and then shakes the can to dispense the salt. Following the completion of the pouring action, R2 returns the salt can to its original location on the table.6. handover() R2 (Franka arm) completes handovers quickly and efficiently by directly moving its end-effector towards the forecasted human wrist position. Once the robot's end effector is within a threshold of the human's wrist position, it stops and releases the object into the robot's hand. Finally, the robot arms reset back to its original position.Object Localization. The object localization pipeline first takes as input RGB image and text prompt of the object of interest, which is passed through an OWLViT [69] object detection model that produces k bounding box proposals denoting possible locations of the object. These k bounding boxes are then filtered using non-maximum suppression to remove overlapping boxes. Due to the camera angle and other noise in the environment, we find that the top OWLViT bounding box does not reliably agree with the desired object. Thus, these proposals are refined by feeding each of the images of the cropped bounding boxes and the text prompt to a pre-trained CLIP [23] model to create a CLIP score that measures how aligned each cropped image is with the text prompt 4 .Next, the image, the bounding box with the highest CLIP score, and the text prompt is fed to a pretrained FastSAM [24] model to segment the object located in the bounding box. The point cloud given by the depth camera is used to project all the points inside the segmentation mask into 3D space. All the 3D points of the object are averaged to obtain a final, single 3D point. This 3D point is then fed to the execution module to produce actions for how the robot should move to the object.RL Simulator and Reward Function.The RL agent needs to take the goal prediction and execute a series of actions to reach that goal without collisions. For pick() specifically, consider a pantry that is stocked with items. A desirable trajectory would avoid hitting the pantry boards, hitting neighboring objects, and pushing the object as the gripper approaches. To guide the agent, we create a simulator that, for a given goal point, builds a 3-dimensional set of walls to the sides, back, and bottom of the goal. Invalid actions are those that collide with a wall or violate robot joint states. An episode starts by sampling a start and goal position within some distance reachable by our robot. We evaluate under four different starting configurations: a) Easy, when the gripper is close to the object; b) Realistic, when the gripper is retracted; c) Recovery, when the gripper is in an extended position away from the object; and d) Obstacle, when the object is partially occluded. We see the RL agent trained in simulation successfully reaches the goal without hitting the pantry, despite being reset to states that oblige recovery motions. However, success rate deteriorates as object placements violate the initial assumptions made about the simulator used to train the agent.The observation space is the L 1 norm between the goal and current positions. We then train a Proximal Policy Optimization [70] agent using the implementation from Raffin et al. [71] with the same action space as the teleoperation commands in the demonstration data using the following cost functionEQUATIONwhere O c and O g represent the current and desired endeffector coordinates respectively, and ∥ • ∥ 2 is the Euclidean distance. The main failure case for the agent is violating joint constraints while trying to avoid the walls because the observation space does not include joint states.Comparing Training RL in Simulation with BC and IK. We evaluate the action-execution module of the pick() skill from 4 different types of starting configurations, visualized in Table 4 . The RL agent completes the skill and avoids collision, achieving the highest success rate of 85% across the different configurations.Using an IK-based controller successfully reaches the goal in the Easy configuration where the gripper is directly in front of the object. In other harder settings, such an approach often collides with the pantry, leading to an overall success rate of 32.5%. We also train a behavior cloning (BC) agent using 50 demonstrations of the pick() across the Easy and Realistic configuration. BC achieves 50% accuracy when tested within these two settings, but its performance decreases to 15% in out-of-distribution configurations (Recovery and Obstacle). The RL policy has a perfect success rate across all configurations except for Obstacle, which it completes 40% of the time. This is because we assumed an absence of occlusion while designing our reward function for the RL policy, and thus the Obstacle configuration demonstrates a limitation to learning via hand-designed reward functions. We posit that with sufficient demonstrations, BC is capable of learning a more expressive policy in such situations.Behavioral Cloning Baseline Our BC policy consists of two feed-forward layers with 256 neurons and is trained on 50 demonstration trajectories with variation in the robot arm's starting height and location of the object. At each timestep, the model takes as input the difference between the current end-effector position and the final position (the same as the RL agent), and outputs a 10-dimensional vector of logits, where each dimension corresponds to moving one of the robot's 10 joints. The model is trained using a weighted cross-entropy loss function to account for class imbalances. Onpolicy, a final action is obtained by categorically sampling from the output vector.Model Architecture. We use a Space-Time Separable Graph Convolutional Network (STS-GCN) [58] model architecture for our human-motion forecaster, which encodes the human's joint positions at different timesteps as nodes in a graph. Instead of simply constructing a fully connected graph between all nodes, the model constructs a sparse network without redundant edges across temporal and spatial dimensions. Edges are connected only between the same human joint through consecutive timesteps and between all joints at the same timestep.Experimental Setup. In order to employ our human motion forecasting model for real-time inference, we make use of an RGB-D camera (Intel RealSense D435) pointed at the human's torso. The human pose is represented by the 3D positions of 7 upper body joints (shoulders, elbows, wrists, and neck). We track the 2D human joint locations using MediaPipe [14] on input RGB images and back-project them to 3D world coordinates using the depth map. As discussed in the approach, our method is forced to handle noisy inputs from depth map projections which are out-of-distribution for motion forecasting models trained on high-fidelity motion capture data. We first compare forecasting performance on CoMaD [13] to select a model suitable for predicting human motion in our dynamic kitchen setting. CoMaD is collected via motion capture suits and contains 270 episodes of human-human interactions across 3 different kitchen tasks with an average length of 30 seconds per episode (4+ hours of total data). Then, we conduct experiments injecting various levels of random Gaussian noise into motion capture data at train time to overcome the train-test distribution mismatch and report results on a dataset of human motion tracked by our single-camera setup.Forecasting Metrics. We quantify errors made by the forecaster by measuring both the Average Displacement Error (ADE) on all predicted timesteps and the Final Displacement Error (FDE) of the predicted pose 1-second prediction into the future given 0.4 seconds of pose history. We report metrics on All Joints as well as Wrists specifically, as they are the most relevant joints in the manipulation tasks we roll out. We additionally report forecasting metrics on the CoMaD transition dataset of human motion during short transition windows in which humans come in close contact with one another, denoted by prefix 'T-' (e.g. T-All Joints ADE, T-Wrists ADE). Note that humans are always in very close proximity during the TABLE SETTING task.CoMaD Forecasting Results. Our two baselines are (1) BASE, trained only on AMASS data, and(2) SCRATCH, trained only on CoMaD data. We report results for two more models: (3) FINE-TUNED, pre-trained on AMASS data and fine-tuned on CoMaD data, and (4) FINETUNED-T, pretrained on AMASS data and fine-tuned on CoMaD with upsampling from its transition dataset.Each model is tested on a held-out CoMaD test set of episodes. FINETUNED-T significantly outperforms all other models across every metric for the REACTIVE STIRRING and HANDOVER tasks.On the TABLE SETTING task, FINETUNED only marginally produced lower errors compared to FINETUNED-T, both of which beat out the baselines.We find that upsampling CoMaD transition data where humans are in close contact enables more accurate motion forecasts on kitchen activities. BASE struggles to generate accurate predictions in highly dynamic manipulation tasks, as it was only trained on AMASS [12] data of a single-human and lacks interaction data. SCRATCH is challenged with learning general human motion dynamics from CoMaD, a much smaller dataset compared to AMASS, which is reflected by its higher errors. Ultimately, we find that pre-training forecasting models on large-scale human activity data and fine- tuning on human-human interaction data yields the best performance in close-proximity kitchen manipulation tasks. Our method employs FINETUNE-T for the remaining experiments.Vision-Based Forecasting Results. We attempt to address the train-test distribution mismatch (trained on high-fidelity motion capture data and and tested on human poses estimated by RGB-D cameras) faced by the motion forecasting model when making predictions on our RGB-D based 3D pose tracking system by injecting random Gaussian noise to motion capture inputs at train time, forcing the model to denoise inputs and generate smooth forecasts. Formally, we conduct experiments by doing the following: given the history of human pose (J joints, each in 3D coordinates) over the last K timesteps ϕ ∈ R K×J×3 , add Gaussian noise N ∈ R K×J×3 ∼ N (0, σ 2 I) to obtain ϕ σ = ϕ + N (σ denotes the "noise level" injected into the pose history). Let ξ H ∈ R T ×J×3 denote the human pose in the next T timesteps. Instead of learning a model for P (ξ H |ϕ) as traditional methods do, we learn to model P (ξ H |ϕ σ ). Table 6 shows vision-based forecasting metrics on the REACTIVE STIRRING and HANDOVER tasks for models trained with σ ∈ {0, 0.001, 0.01, 0.1}. We find that when forecasting human motion from our single-camera based 3D pose history, the model learned with hyperparameter σ = 0.01 generates the most accurate predictions across all metrics (ADE and FDE), yielding it most suitable to be integrated into the overall system.One key benefit of a modular system is the ability to localize the failure of an entire end to end run within the specific submodule that failed. We enumerate some of the most common failures observed in our end-to-end experiments below:(A) [Visuomotor Skill] Failed to pick up the object: Sometimes, the VLM selects an incorrect object given the object prompt (further analysis in Section 3.4). Other times, errors in the predicted goal location leads to missed grasps.(B) [Visuomotor Skill] Failed to successfully place the object: Errors in the go to() skill leave the robot too far away from the table to successfully place an object. Releasing the object from an incorrect height also causes it to topple. Experimental Setup. In order to conduct the user study, we build a web-based application to chat with the task planner. The application is intended to virtually simulate a kitchen environment, where the participants see: 1) the chat history with the planner, 2) the complete recipe, 3) the current task queue of each agent, 4) available tasks, and 5) completed tasks (see figure 10 ). The application allows users to interact with the task planner once, prepare a pre-determined recipe, and then answer survey questions based on their experience. We picked 7 recipes: "avocado toast", "sundae", "milkshake", "biryani", "ramen", "stir fried noodles" and "pasta", to assign to participants in the internal study, randomly selecting a mixture of desserts, noodles, and entrées with roughly the same number of nodes in their recipe DAG. Each participant prepared the same recipe twice, one with each planner (One-Prompt and Tree), but was not made aware that the planner was different in the two interactions.We also picked 10 recipes: "mango sticky rice", "eggdrop soup", "pasta salad" and 6 from above, to conduct the external study. We again added a variety of different recipes of similar length. We notably excluded "biryani", as our internal study showed participants from all regions and cultures may not be familiar with this dish, and familiarity of a recipe helps them focus on the interaction. Each approach is evaluated on 5 random unique recipes from beginning to end with varying numbers of non-nominal interactions. Each approach gets run 3 times per recipe. We present the total number of constraint violations across all runs for each difficulty level. Tree has the lowest total number of constraint violations compared to One-Prompt for all difficulty levels. Concretely, Tree makes 80.4% fewer constraint violations compared to One-Prompt for "Easy" tests and 22.0% fewer for "Hard" tests.Therefore, out of the n = 46 interactions, 26 were from 13 internal participants, set up as a withinstudy, and 20 were from unique external participants, set up as an in-between study. We randomize the order in which the two planners are presented in the internal study, and all participants participated in a "pilot" study with a recipe of their choice to develop familiarity with the interface. We do not include results from the "pilot" study due to a large variance in interactions depending on the size of the recipe.Instructions to the Users. We asked all users -both internal and external, to interact with the planner naturally and with at least 3 non-nominal interactions (in order to bring out constraint violations) 1. They should not directly name the recipe they want to make, and instead lead the assistant into suggesting it.2. They must make at least one intervention in the assignment of a task, like "I will handle the stirring"3. They must add at least one task that was not part of the recipe, like "get me eggs" for Ramen.As part of the post-chat survey, we ask the users the following questions:1. How many times has the assistant assigned a task without their permission?2. How many times they were lied to by the assistant?3. How many times did they feel ignored by the assistant?Their chat history is presented to them as they fill out this survey, and they are asked to provide specific instances along with each answer. Three authors then cross-validated the users' answers with the chat history.Full Quantitative Results. Table 8 shows the results of our study on the three metrics we discussed above. We see that while each study by itself shows some trends, both studies put together give us enough data to reject the null hypothesis along two metrics (lying and assigning without confirmation). We also see that the overall frequency of ignoring the user is low in both approaches.We provide examples for how One-Prompt and Tree violate each constraint:• Act Without Permission: The task planner assigns/removes subtasks without user's permissions.• Lying: The task planner claims to do something but does not do it.• Ignore User: It does not respond to the user's instruction.Table 7 lists examples of violations for each of these constraints.Experimental Setup. To systematically test the task planner, we design unit tests that evaluate whether the task planner has correctly handled a user request. In addition to nominal interactions, where the user gives clear instructions and agrees with the task planner's proposal, we identify 4 non-nominal interaction modes and how the task planner should react to those interactions.(A) Vague recipe name: The user says a general category or a general description, so the task planner needs to talk to the user to narrow the options down to one recipe.(B) Recipe that does not exist: The user says that a recipe that the system does not currently support, so the task planner needs to talk to the user to suggest an alternative.(C) Modify subtask assignment: When the user disagrees with the task planner's proposed plan and requests to assign a subtask to another agent, the task planner should comply and assign that subtask to the agent requested by the user.(D) Add subtask outside of the recipe: When the user requests to assign a subtask outside of the recipe, the task planner should comply and assign that subtask to an agent who is capable of performing that subtask.To generate natural interactions during the tests, we create an LLM prompt that mimics an everyday user who provides different instructions based on the interaction mode we set programmatically. The prompt is in the supplementary materials.We create the following categories with increasingly more complex interactions: "Easy" with only one random non-nominal interaction and "Hard" with six. For each difficulty level, we test the approaches on 5 recipes, and for each recipe, we run the entire cooking process 3 times with the same set of non-nominal interactions. This experiment results in 30 runs per approach and an average of 34 chats per run.We measure the average percentage of unit tests passed and analyze the number of times that the task planner has violated the constraints specified in the prompts. The constraints are the same as the ones in the user study ("Act Without Permission", "Lying", "Ignore User").Full Quantitative Results. Table 9 shows that, overall, Tree has a higher percentage of average unit tests passed compared to One-Prompt. When the interaction becomes more complex in "Hard," Tree maintains its unit test pass rate at 94.0 ± 2.30%, while One-Prompt's performance drops from Error Analysis The One-Prompt assigns 'get salt' and 'get pepper' to R2 before user confirmation. This is against the constraint that the One-Prompt cannot directly assign without confirming.The recipe is Corn Soup. There are no available subtasks. The latest user input states that they want to pour corn into the pot instead of R1.Action say("Understood, you will handle pouring the corn into the pot. Please let me know if there's anything else I can assist with!")Error Analysis One-Prompt acknowledges the user and confirms that they will handle pouring the corn into the pot but does not assign this to the user. This is a lie because the user was never assigned the task despite what the task planner claims.The recipe is Pasta Salad. The available subtasks are 'mix bowl', 'prepare vegetables', 'pour salt into bowl', 'pour pepper into bowl', and 'pour pasta into bowl'. The latest user input asks R1 to prepare the vegetables and says that the user will mix the bowl.Action say("Understood. You will prepare the vegetables and mix the bowl. Please let me know once you have finished preparing the vegetables, and I can assist with the next steps for the pasta salad. R1 will remain on standby for now") assign("User", ['prepare vegetables', 'mix bowl'])Error Analysis One-Prompt ignores the user's request to assign 'prepare vegetables' to R1. Though this is outside of R1's capabilities, a message of 'Unfortunately, this is not within R1's capabilities. Would you like to prepare the vegetables?' would have acknowledged the user's instructions without ignoring them. By ignoring the user's request for R1, One-Prompt is against the constraint to reply to all of the user's instructions. 100.0±0.00% to 60.0±9.04. Meanwhile, Figure 11 highlights that although both models' constraint violations increase when the interaction becomes more complex, Tree consistently violates fewer constraints compared to One-Prompt.We include the full content of the prompts used by the interactive task planner and our experiments. Specifically:1. The interactive task planner decides on a recipe and reasons through subtasks through a behavior tree. We provide the prompts for each node in the tree in Section I.1.2. Once a recipe is decided, the LLM generates a nested list which is then processed into a directed acyclic graph (DAG). The prompt to do so is detailed in Section I.2.3. Once the robot needs to execute an assigned subtask, another LLM is used to generate Python code that calls these low-level robot skills. The example template for code generation is in Section I.3.Finally, the monolithic prompt we compare to in the user study is included in Section I.4.This section provides the prompts for all possible nodes to exist in the behavior tree. Note that to add a new behavior, one simply needs to create a prompt for the behavior and add it as an option for other relevant behaviors to invoke.Deciding on a recipe. -description: User gives non-existing recipe but there is an → alternative -observation: | recipe_name: "" chat_history:-User: Hey Mosaic! I want to make corn and avocado salad. user_input: "Hey Mosaic! I want to make corn and avocado salad → ." -response: | { "reasoning": "The recipe has not been decided yet. There is no → recipe for corn and avocado salad. However, the salads in → the recipe list are good alternatives.", "decision": "Suggest_Alternative_Recipe" } -description: User gives broad command. Suggest alternatives ( → specific dish) -observation: | recipe_name: "" chat_history:-User: Let's make dinner. I am in the mood for a vegetable → dish.user_input: "Let's make dinner. I am in the mood for a → vegetable dish." -response: | { "reasoning": "The user input is quite broad and does not → specify a particular dish. Since the user is looking for a → vegetable dish, I can suggest an alternative recipe from the → list that matches the general criteria of being vegetable-→ based. Caesar salad is a vegetable dish from the recipe list → , so I can propose it as an alternative.", "decision": "Suggest_Alternative_Recipe" } -description: User clearly says a specific dish -observation: | recipe_name: "" Chat chat_history:-User: I want to make some some kind of quick, grab-and-go → lunch.- → Find the recipe that matches the best with the user's → requirements based on user input and chat history.2) If there are multiple recipes that match the user's needs, → then suggest the 1 that matches the most to the user's needs → .2) You should never list out the steps in the recipe. You → should just give a quick reply indicating that you are ready → to start making the recipe.3) You must reply in the given format: { "reasoning": < your-reasoning-should-go-here >, "recipe name": < your-recipe-should-go-here >, "reply": < your-reply-should-go-here > } Recipe List: <recipes> examples: -description: user gives recipe with exact match -observation: | recipe name: "" chat_history: -User: Hey Mosaic! I want to make bibimbap for dinner tonight. user_input: "Hey Mosaic! I want to make bibimbap for dinner → tonight." -response: | { "reasoning": "The user clearly says that they would like to → make bibimbap, which exactly matches the bibimbap in the → recipe list. I can confidently suggest the bibimbap recipe → from the list as it closely matches the user command.", "recipe name": "Bibimbap", "reply" : "Let's make bibimbap!." } -description: user gives recipe with no match but mentions → ingredients -observation: | recipe name: "" chat_history:-User: I just bought rice, vegetables, and gochujang. Let's → make rice for dinner. user_input: "I just bought rice, vegetables, and gochujang. Let → 's make rice for dinner." -response: | { "reasoning": "The user says that they have the ingredients of → rice, vegetables, and gochujang. These ingredients match → those of bibimbap in the recipe list. I can confidentaly → suggest the bibimbap recipe from the list as it contains all → of the ingredients the user mentioned.", "recipe name": "Bibimbap", "reply" : "Bibimbap in the recipe list matches your description → . Let's cook bibimbap together." } -description: user gives recipe with no match but ingredients → similar -observation: | recipe name: "" chat_history:-User: I want something cold for lunch.-Mosaic: Do you want caesar salad, fruit salad, or poke? -User: Sure, I would love to eat poke. user_input: "Sure, I would love to eat poke." -response: | { "reasoning": "I have provided the user with options of cold → lunches and the user says that they would like to make poke, → which matches the poke in the recipe list. I can → confidently suggest the poke recipe from the list as it → closely matches the user command.", "recipe name": "Poke", "reply" : "Let's make poke then!" } Suggesting alternative recipes. If the user proposes a recipe that is outside the set of seed recipes, the task planner will communicate with the user to settle on the closest alternative within the set of seed recipes. 2) Find the top 2-3 recipes that match the best with the user's → requirements based on user input and chat history.3) You must reply in the format: { "reasoning": < your-reasoning-should-go-here >, "reply": < your-reply-mentioning-alternative-recipes-should-go-→ here > } Recipe List: <recipes> examples: -description: alternatives that go with a specific dish not in → the list -observation: | recipe name: "" chat_history:-User: Hey Mosaic! I want to make an onion soup. user_input: "Hey Mosaic! I want to make an onion soup." -response: | { "reasoning": "There is no recipe for an onion soup. However, → the soup in the recipe list (mixed vegetable soup, tomato → soup, egg drop soup) are good alternatives, so I will → suggest those. ", "reply": "I don't have onion soup in my recipe list, but we can → cook other soup! How about a mixed vegetable soup, tomato → soup, or egg drop soup?" } -description: respond to a general recipe list -observation: | recipe name: "" chat_history:-User: I want something light for dinner today. What do you → suggest? user_input: "I want something light for dinner today. What do → you suggest?" -response: | { "reasoning": "The user did not suggest a specific recipe. They → want something light, which could be salads within my recipe → list (tossed salad, caeser salad). I will suggest salad.", "reply": "Salad can a great light dinner. Do you want to make a → tossed salad or a caeser salad?" } -description: respond to broad command -observation: | recipe_name: "" Chat History:-User: Let's make lunch. I am in the mood for a vegetable dish → . user_input: "Let's make lunch. I am in the mood for a vegetable → dish." -response: | { "reasoning": "The user input is quite broad and does not → specify a particular dish. Since the user is looking for a → vegetable dish, I can suggest some alternative recipes from → within the recipe list that matches the general criteria of → being vegetable-based. Caesar salad, Mixed Vegetable Soup, → and Tossed Salad are vegetable dishes from the recipe list, → so I can propose those as an alternative.", "decision": "Caesar salad, Mixed Vegetable Soup, and Tossed → Salad are great vegetable dishes. Would you like to make one → of those for lunch?" } -description: respond to broad ingredient list -observation: | recipe_name: "" Chat History:-User: I have a bunch of lettuce. What should we make for → lunch? user_input: "I have a bunch of lettuce. What should we make for → lunch?" -response: | { "reasoning": "The user input is quite broad and only mentions → lettuce but does not specify a particular dish. Since the → user is looking for a recipe with lettuce, I can suggest → some alternative recipes from within the recipe list that → matches the general criteria of being vegetable-based. → Caesar salad and Tossed Salad are recipe which contain → lettuce from the recipe list, so I can propose those as an → alternative.", "decision": "Caesar salad and Tossed Salad contain lettuce. → Would you like to make one of those for lunch?" } Making the recipe. Once the DAG is generated and subtasks are decided on, the task planner needs to communicate with the user to assign (and sometime reassign) subtasks to either the user or robots. The prompt to do that is shown below.version: 1.0.0 node_type: DecisionNode node_name: Execution prompt_description: e2einterrupt prompt_version: 1.4.2 system: | You are a helpful assistant named Mosaic who facilitates two → robots (R2 and R1) to collaboratively help a user cook a → recipe. Your goal is to assign subtasks that are needed for → the current recipe and to monitor the status of the subtasks → .You are a helpful assistant named Mosaic who facilitates two → robots (R2 and R1) to collaboratively help a user cook a → recipe. Your goal is to assign subtasks that are needed for → the current recipe and to monitor the status of the subtasks → .You will receive the current state of the world, which includes → :→ need to confirm with the user and ask the user to do that → subtask. * Even when everyone is working, if there are subtasks in → the 'available subtasks' list that the robots or the user → can do, you can confirm that subtask with the user. * In 'user_input', the user initiated the conversation → without you asking them anything. They express some need → and you think that you can propose some subtask to solve → that issue. -You should choose 'No_op' if one of these is true: * If the 'available subtasks' list is empty [], you should → wait and do nothing.-If the user does not say anything currently, so ' → user_input' is empty -You should choose 'Interrupt_Subtask' if one of these is true → :* When the robot's status is at Running, user explicitly → requests to stop one of the robot from doing their → current tasks. * When the robot's status is Running, the user mentions an → emergent accident that it's unsafe for robots continuing → doing current tasks.Your response must follow this json format: { "reasoning": "< put_your_reasoning_here >", "decision": "< decision >" } Here are the robot's capability that you must adhere to: <robot_capabilities> examples: -description: Just got recipe. -When the robot's previous status is at 'Running', the user → explicitly requests to stop one of the robot from doing → their current tasks.-When the robot's previous status is at 'Running', the user → mentions an emergent accident that it's unsafe for robots → continuing doing current tasks. You should also move the subtask that is at the robot's current → subtask into the completed subtask list.Your response must follow this json format: { "reasoning": "< put_your_reasoning_here >", "R2_status": "< put_R2_status_here >", "R1_status": "< put_R1_status_here >", "completed_subtask_list": "< put the subtask that got → stopped here >" "reply": < your_reply_goes_here The subtasks in the recipes are: "fetch butter", "fetch onion", → "fetch celery", "fetch chicken broth", "fetch vegetable → broth", "fetch chicken", "fetch egg noodles", "fetch carrots → ", "fetch basil", "fetch aregano", "fetch salt", "fetch → pepper", "melt butter", "pour onion into pot", "pour celery → into pot", "stir pot", "cook for 5 minutes" "pour chicken → broth into pot", "pour vegetable broth into pot", "pour → chicken into pot", "pour egg noodels into pot", "poor → carrots into pot", "pour basil into pot", "pour oregano into → pot", "season soup with salt", "season soup with pepper", " → stir pot", "simmer for 20 minutes"We should fetch all the ingredients first, but fetching the → ingredients should happen in parallel.After fetching butter, we can melt butter.After melting butter, we can pour onion and celery into the pot → .After pouring onion and celery into the pot, we can stir the → pot to cook.After cooking, we can pour rest of the ingredients into the pot → .After adding all the ingredients, we can stir the pot again.After stirring, we can leave the pot to simmer For each to-be-executed subtask, the task planner generates a code snippet which issues a seris of API calls. We provide the LLM with examples of what the code should look like, and then programmatically add in the query which includes the subtask name and the current completed_action_functions. The exact template used is shown below: # Python mobile robot high-level task planning script import numpy as np from robot_utils import <robot_api> from env_utils import <env_constants> <header_example_separator> """ get can of corn completed_action_functions: ["go_to('PANTRY')"] """ <query_code_separator> # go_to(PANTRY) # already completed this action pick_up_item(CORN) go_to( → somewhere. This 'somewhere' can also be referring → to a food item (e.g. spread the honey on toast) -Actions not present in their respective lists are → beyond their capabilities, e.g. none of the robot can → do subtasks such as 'prepare something', 'boil → something'.You must make updates to the subtask queues based on the → chat history, user input, and available subtasks. The → update must not violate the robots' capabilities. The → updates must be one or combination of following: * You should insert subtask into R2_subtask_queue or → R1_subtask_queue if all conditions are satisfied:-the subtask is either proposed by user in ' → user_input' or in 'available_subtasks' -the subtask is within the capability of an agent -the user agrees and gives permission in ' → user_input' to your proposal that is in the ' → chat_history'. * You should insert the subtask into ' user_subtask_queue' if all conditions are satisfied → :-the subtask is either in 'available_subtasks', or → proposed by user in 'user_input' -the user has agreed to take over the subtasks → themselves in 'chat_history' -the subtask is not within the capabilities of any → agent, so the user will handle it. You should → explain the situation to the user in 'reply' too → . * You should remove the subtask from ' → user_subtask_queue' and insert subtask into → completed_subtask_list if:-user explicitly indicates that she/he has → finished the subtask in 'user_input' * You should remove a subtask from 'available_subtasks → ' if:-in 'user_input', user explicitly indicates that → they want no one to complete this subtask, or → don't want to do this task anymore.You must provide reasoning for all the updates you have → made. You must only reply in the following JSON format { "reasoning": < your_reasoning_goes_here>, "R2_subtask_queue": < your_updated_R2_queue_goes_here >, "R1_subtask_queue": < your_updated_R1_queue_goes_here >, "user_subtask_queue": < → your_updated_human_queue_goes_here >, "completed_subtask_list": < → your_completed_task_queue_goes_here >, "reply": < your_reply_goes_here > } You should use double quote rather than single quote across → each subtask name. The trials had authors acting as users and involved a total of 4 users between the ages of[20][21][22][23][24][25][26][27][28][29][30].2 The user study was approved by the Institutional Review Board at the University. See Appendix G for experimental setup, user study interface, and survey questions.We used an existing open-source recipe scraper.If k is set too low, the set may not contain a bounding box around the object of interest to be used by CLIP. If k is set too high, the set of bounding boxes may be too noisy, resulting in lower accuracy. We set k = 10 for all experiments. 
related work: 
 Home Robots. Recent research efforts have attempted to provide robots with generalist capabilities to sufficiently adapt to home-like environments [1, 2, 3, 4, 31] . However, many of these works [3, 4] are limited to completing predefined tasks that don't require explicit task planning, e.g. picking a single item. Liu et al. [31] similarly tackle open-vocabulary navigation, but still sidesteps the challenge of a dynamic environment by assuming a static representation of the world after initialization. On the other hand, some works consider multi-arm/multi-robot planning for collaborative tasks [9, 32, 33, 34] . For example, Mandi et al. [9] significantly constrains human-robot collaboration by forcing the human to complete a specific task before the robot proceeds with its own task. In this paper, we aim to overcome these limitations by designing a system that enables multiple robots to fluidly collaborate alongside humans to perform a wide range of tasks.Specific Modules. Our interactive task planner module is similar to work in single-robot settings with clearly defined language goals that generate a list of actions as the plan [8, 35, 36, 37, 38, 39] and synthesize code that calls robot action API [6, 40, 41, 42] ; however, most of these works are non-interactive. In contrast to the most similar interactive work [8] , we are solving a multiagent task planning problem involving two robots and a user and communicating with the user to allocate tasks properly. Our visuomotor skills module is similar to the family of prior work [15, 19, 20, 43, 44, 18, 18, 44, 45, 46] that leverages VLMs for object identification within manipulation tasks. However, in contrast to prior work [47, 48, 42, 49, 45, 46] , we train our action policy using reinforcement learning in simulation where affordances are provided by the VLM and constraints are inherent to the simulator. Collaborative manipulation tasks near humans necessitate human motion prediction, traditionally bypassed by assuming a static human [50, 51] . Advances in neural networks and the availability of extensive human motion datasets [12, 52, 13] have enabled the development of sophisticated RNN and GNN models to predict movements from past joint positions [53, 54, 55, 56, 57, 58, 30] .We first enumerate more related works in Section A, then outline the configuration of our system, as detailed in Section B. Subsequently, we delve into further discussions on each component of our system, covered across Sections C, D, and E. Finally, we present an in-depth analysis of the user study, incorporating details on the experimental setup and supplementary findings, all of which are elaborated in Section G.Appendix A Extended Related Works Modular Architectures. Modularization has been extensively employed to partition complex and long-horizon robotics tasks into more easily addressed sub-components. For example, in the space of task planning, different Large Language Model (LLM) modules are used to improve command interpretation in unseen environments [59] and to produce corrective replanning prompts [60] . OK-Robot [31] focuses on the problem of object retrieval and navigation using separate submodules for mapping, object-detection, path-planning, and grasping. Our approach tackles the combined domains of task planning, visuomotor skill learning, and human motion forecasting. We leverage modularity at multiple levels, e.g. we outperform a single VLM object detection by combining Owl-Vit to detect bounding boxes, and CLIP to select the correct box.Task Planning. A task planner takes as input a high-level task, e.g. cooking a recipe, and generates a plan, e.g. a sequence of sub-tasks, to achieve that goal. Traditional approaches frame this as a search problem and invoke a symbolic planner to solve it [61, 62, 63, 64] . However, using these methods for everyday tasks is challenging because they require pre-defining the search space and lack a natural-language interface to interactively communicate the task. Recent work leverages LLMs for task planning to overcome both of these limitations. In single-robot settings, given a clearly defined language goal, recent work can be categorized as generating a list of actions as the plan [35, 36, 37, 38] , synthesizing code that calls robot action API [6, 40, 41, 42] , or translating to a problem solvable by a classical planner [7, 65] . However, none of these systems interact with humans and coordinate tasks for both humans and robots.Idrees et al. [5] focuses on estimating a user's intent and using question-answering interaction with the user to update that estimate. Then, the robot can use that estimate to suggest how the user can make task progress. However, its interaction is limited to yes-or-no questions, and it does not focus on task planning for multiple agents based on the robot's capabilities and the user's unstructured natural language feedback.Li et al. [8] has the closest task planning framework to our approach, where the LLM takes a specific natural language goal to generate a step-by-step plan before synthesizing robot code for each step. However, because we are solving a multi-agent task planning problem involving two robots and a user, our task planner cannot simply output a list of steps. It must continuously communicate with the user to properly allocate subtasks to suitable agents.Visuomotor Skills. Several recent works study the application of pre-trained vision-language models (VLMs) to robotics [18, 66, 15, 19, 47, 20, 43, 44] . One family of recent work [15, 19, 20, 43, 44, 18] integrate pre-trained VLMs in an end-to-end fashion, e.g. segmenting out regions of interest to assist in action prediction [18, 44] . A second flavor of approach [47, 48, 1, 37] leverages VLMs to recognize affordances and constraints in the environment and provide corresponding execution instructions through language [47] or code [48] . Our model is similar in this aspect, where we distinguish the training objectives of environment perception and action execution. This effectively liberates us from needing a large dataset of humans or robots demonstrations to provide good coverage [18, 15, 19, 20, 21] and from having to worry about embodiment mismatch between large-scale robot learning datasets [67, 68] .Human Motion Forecasting. Collaborative manipulation tasks in close proximity to humans require predicting human motion. This is a challenging problem since human motion is complex and highly variable. A common approach is to sidestep the problem of motion forecasting [50, 51] by considering the human to be static. Instead, recent research is moving towards the use of neural networks and supervised learning to predict future human motion based on a short history of past joint positions [53, 54, 55, 56] . The release of large open-sourced datasets of human motion [12, 52] has made it possible to train large RNN and GNN-based neural network models for human-pose forecasting [57, 58] . Consequently, these datasets have been integrated into robot motion planning, focusing on collaborative manipulation tasks [30, 13] . Closest to this work, ManiCast [30] proposed a framework to learn cost-aware human forecasts. However, this approach relies on a bulky motion camera setup, requiring the user to wear a motion capture suit with markers. In this work, we run our integrated human motion forecasting and planning system in real-time using a single RGB-D camera to track human pose.Kitchen Scene and Robot Placement. The kitchen scene consists of a main kitchen table at the center where all cooking activities are performed. A pantry is placed near the table, which contains a large range of condiments and kitchen staples. There is also a secondary table on the side of the center table meant for serving up the final dishes. Our robot system includes two robots (R1 and R2).• R1 (Franka Emika Research 3 [26] ) is a tabletop 7-of manipulator stationed at one end of the kitchen tables at the center of the scene.• R2 (Hello Robot Stretch RE1 [25] is a mobile manipulator that can navigate around the kitchen area, capable of fetching and putting away condiments and kitchenware as required by the user.Camera Placement. For the tabletop manipulator (R1), the perception stack includes two Intel Realsense D435i RGB-D cameras placed above the center kitchen table. Both cameras are placed at opposite ends of the table and at an angle such that they capture the entirety of the tabletop as well as the human user. Integrating both camera perspectives enhances the visibility of objects and human poses within a cluttered kitchen setting, effectively mitigating occlusion issues. The mobile manipulator (R2) uses an onboard Intel Realsense D435i RGB-D head-camera for perceiving objects.Computational Details. In addition to the onboard computing capabilities of the robots, our setup includes five personal computers (PCs) dedicated to running various system modules. These PCs are connected to the same network, utilizing the Robot Operating System (ROS) for communication.For tasks that demand real-time neural network inference, we employ onboard GPUs, (NVIDIA GeForce RTX 3060). Detailed information about each PC's role and configuration is provided:• C1: Connected with a Bluetooth microphone and speaker, this PC runs the Speech-Text system for communicating with the user and the Interactive Task Planner that utilizes GPT-4 API calls.• C2: Used for running neural network models related to the perception (object detection) and control (RL agent) of R2. This PC also communicates with C1 to allocate subtasks to R2.• C3: This PC forms the perception stack for R1, including running neural network models for object detection and pose estimation.• C4: This PC runs the human forecasting model using the pose estimates computed by C3. This PC also computes motion plans for R1 based on the predicted object pose and human forecasts. Further, it communicates with C1 to allocate subtasks to the R2.• C5: This PC is installed with a real-time kernel to send joint commands to R2 at 1 kHz frequency as recommended by the robot manufacturers.System Assumptions. In accordance with the setup above, we make a set simplifying assumptions in our work:1. Access to a set of seed recipes: A recipe contains a set of subtasks with temporal dependencies. We seed the system with an initial set of recipes, but the user has the freedom to make modifications on the fly (e.g. adding an ingredient).2. Access to a map: We assume that our system has mapped the kitchen ahead of time, so it is aware of where ingredients and tools are stored and how to navigate to different locations. 
chat history: 
 The recipe name can be empty if the recipe has not been decided yet. The available subtasks are populated by the DAG. The robot and user subtasks are all populated by the behavior tree's highlevel actions; the robot additionally has a current subtask and status field updated over a ROS action server as the robots complete their subtasks. When subtasks are completed, the completed subtask queue is updated. Finally, the user's currently spoken input is stored and later appended to the chat history along with the task planner's messages.The high-level actions include• say(msg)• set recipe(name)• assign(agent, subtasks)• mark complete(subtasks)• interrupt(agent)• no op() say(...) allows the task planner to communicate to the user with a message. assign(...) will assign a list of subtasks to an agent (robot or human). mark complete(...) will set a list of subtasks as completed. interrupt(...) will stop a robot from doing its current subtask. no op() does nothing.The tree consists of various nodes that each query an LLM that either outputs (1) a decision for the next node to run or (2) arguments for the high-level actions to take. Each node is associated with a prompt that is used when querying the LLM. If a node's query response is malformed (e.g. bad JSON) or invalid (e.g. bad decision or arguments), the node is rerun. Each node only requires the observation as input, so we can run each node simultaneously to parallelize the LLM queries and draw a path from the root to a leaf based on the decisions made.The tree runs a cycle to take high-level actions whenever the observation differs from the past observation. This gives the user time to respond to the task planner's questions. To receive user input and respond to the user, we use speech-to-text and text-to-speech modules, respectively. The tree runs indefinitely until the script is terminated.Code Generation. Whenever the task planner assigns a subtask to a robot, it must be converted into a sequence of low-level skills the robot is capable of. We do this by using an LLM to generate code that the robot runs.When the task planner assigns a subtask to a robot, it is first added to the robot's subtask queue. A thread dedicated to the robot checks to see if there are any subtasks in the queue, pops it to add to the current subtask, and sets the status to Running. A separate prompt for code generation is used to query an LLM to generate code for the provided subtask. An example of generated code includes from robot_utils import <robot_api> from env_utils import <env_constants> pick_up_item(LADLE) place_item_at(POT) stir()where <robot api> includes all low-level robot skills like pick up item(...) and where <env constants> includes enums for objects in the environment. Each line of code executing a robot skill sends a ROS action to the robot to execute said skill. When the robot finishes executing its current skill, it communicates that it has finished to the task planner, which can, in turn, send another skill. This continues until the entire subtask is finished, in which case the robot's current subtask is cleared, and the robot's status is set to Idle. If the robot is interrupted, its current subtask is also cleared, but its status is set to Interrupted.Skill Library. The task planner has access to a number of robot skills represented as function calls that are parameterized by object positions and target locations. For each skill, the positions of the objects are estimated using an open-vocabulary object detection model, OWL-ViT (more details in the next section), given text prompts provided by the task planner. For navigation, we store mapped locations to real-world coordinates, assuming the kitchen scene does not change its configuration between runs.We enumerate below the set of low-level skills performed by the two robots in this paper:1. pick(<obj>): Both robots share the same object detection module to complete the pick(<obj>) task to get bounding boxes and a 3D grasp-pose around the object of interest. R1 (Franka arm) moves directly to the grasp pose using an inverse kinematicsbased joint impedance controller. R2 (Stretch robot) is tasked with picking up objects from a cluttered pantry. To avoid hitting the pantry and surrounding objects, the robot uses a reinforcement learning policy trained in simulation to execute actions. 
 
 Appendix:  


Decision: Strengths:

- Interesting problem of multi-robot and human coordination, using LLMs for natural language interaction
- Modular architecture, integrating many different SoTA models from literature from object detection, segmentation, human action detection. Impressive overall system, with large number of real-world evaluations.

Weaknesses:

- The interactive talk planner lacks details, including exact prompts used. Please provide details either in the main paper or appendix, as requested by reviewers. Also a discussion on how the planner can be adapted to related but unseen tasks, like a different recipe, or unseen environment, would be helpful.
- While the system is impressive, it seems a little specific to the problem at hand. A better motivation of the system design would help future researchers modify it to other problems. Additionally, please clearly list out the assumptions in the system.
- The real-world experiments are conducted with a small set of expert users, which raises questions about how well the approach generalizes to other users. Including experiments with novice users would make the claims about interactivity and assistance stronger.

Post rebuttal:
Authors have clarified the assumptions and details of the approach. They have also provided a description of the approach's ability to generalize to new tasks, and presented a larger user study with novice users. Overall this is a good paper.

Reviews: review: title: title: Review
paper_summary: paper_summary: The paper proposes a modular system, MOSAIC, to coordinate multiple robots and help humans finish multiple kitchen tasks. It mainly has three components: an interactive task planner, human motion forecasting, and visuomotor skills. Through experiments, it show that the proposed system performs better than a baseline method which only uses one single forward query of the LLM.
main_review: main_review: Strength:
 - The paper focuses on an important problem, which is human-robot coordination. The topic is also of interest to CoRL attendees.

Weakness:
 - The paper claims that modularization is its main technical contribution. However, several prior studies have already addressed modularized systems and methods to tackle complex, long-horizon tasks. For instance, Voxposer first constructs affordance maps with different LLM and then generates codes. This makes the paper’s technical contribution in this area less significant. However, I do agree that using LLM systems for human-robot interaction is novel.  
 - The interactive task planner, which aims to allocate tasks to two robots and humans in the experiment, lacks significant details. It is unclear how the subtasks are allocated, whether the LLM first generates the overall plan, and how frequently the plan is regenerated. 
 - The experiment setting is unclear. It will be better to use one running example to show how the method works. And it will be better to add more descriptions for each task.
questions: questions: - What are the principles used to design the modularization architecture? Considering the problem from a hierarchical perspective, the task planner handles high-level planning while visuomotor skills focus on the low-level aspects. How should the human motion forecasting module be positioned within this hierarchy? Additionally, there are sensor-specific modules like object detection. Are these sensor-wise modules included in the authors’ modularization framework?
 - Regarding the interactive planner, how are subtasks assigned to different robots and humans? 
 - Does the language model first generate the sequence of actions? Are the subtasks shown in the upper right corner of Figure 3 predefined? 
 - Do the robots need to obtain permission from humans first before executing these actions?
limitations: limitations: 3
review_summary: review_summary: The paper focuses on using foundation model systems for human-robot collaboration, which is novel. However, the paper’s technical contribution is overclaimed, and the methodology section needs more clarification regarding the design principles and significant details.
title: title: MOSAIC Review
paper_summary: paper_summary: This paper introduces MOSAIC, an interactive, modular system for assistive cooking. MOSAIC utilizes 3 components: a LLM and DAG-based task planner, a human motion forecasting module, and motion primitives. Using this hierarchical architecture, MOSAIC is able to interact with humans for high-level task planning with natural language while also interacting physically for tasks such as hand-over. Results show MOSAIC is capable of cooking six predefined recipes with a human over 60 trials, as well as a clear evaluation of each module’s performance and contributions to the overall system.
main_review: main_review: MOSAIC builds an impressive system for interactive situated cooking with humans. Human users are able to interact with MOSAIC through natural language with the LLM-based task planner, as well as physically with the human motion forecasting model. This setting is particularly challenging due to the safety concerns when interacting closely with humans, which MOSAIC addresses with a trained motion forecasting model that stops the robot when a human is approaching a close proximity. This is a large systems effort, with many real robot evaluations validating the efficacy of each individual component. The paper was well written and the supplemental video was appreciated to visualize the interactive rollout of MOSAIC. 

While MOSAIC provides some impressive demonstration rollouts with expert author users, a larger user study with non-expert users -- users who were not involved with building MOSAIC -- would be needed to validate the assistive claim of the system. Is MOSAIC able to be assistive for a more general audience with different requirements? Are novice users able to learn to use the system effectively?
questions: questions: - From Table 1, why were different skills implemented so differently (learned with RL, hand-engineered, and planning-based)? This requires access to more information about the environment, such as a map for planning or access to a simulated environment for the RL training. 

- Were there any experiments evaluating the ability to add new tasks/”recipes” to the system? The reviewer is concerned about generalizability of the system in an interactive setting. 

- Most critically, how were the interactive and assistive qualities of the system evaluated? Was a user study conducted with the full system? A N>5 user study with users not involved in building MOSAIC would greatly improve support for the claim of interactivity and assistance. Did users feel the robot was helpful? Would users want to use this system again for cooking?
limitations: limitations: 3
review_summary: review_summary: I chose my recommendation because the system demonstrates impressive capabilities with expert users, but leaves the question of whether the system is able to adapt to a novice user unanswered.
title: title: Interesting robot systems for (multiple) human robot interaction for everyday kitchen tasks
paper_summary: paper_summary: The paper presents MOSAIC, a modular architecture for coordinating multiple robots to interact with users using natural language, and manipulate an open vocabulary of everyday objects to solve complex cooking tasks. The main contributions of this paper are: 1) a modular cooking assistant that use LLMs for interactive task planning, VLMs for visuomotor skills, and motion forecasting models to predict human intents for collaboration, 2) perform comprehensive evaluation of MOSAIC, either end-to-end, or per-module, 3) draw insights and lessons learned.
main_review: main_review: The paper has a few noteworthy strengths. First of all, the authors attempt to tackle some of the most ambitious robot-human collaboration tasks in the real world, with natural language interface and open vocabulary. The proposed approach seems like a well-designed, modular system that pieces together several SOTA models across AI fields (e.g. object detection, LLM task planning, etc). The combination of behavior trees with LLMs for task planning is fairly interesting. The most impressive aspect of the paper is that it includes comprehensive end-to-end experimental evaluation across 6 different long-horizon tasks, as well as per module evaluation with ablation studies and detailed analysis. It’s appreciated that each of the error types from the robots are carefully broken down, repeated, analyzed and remedied as much as possible.

However, the paper is not without room for improvement. First of all, one may be concerned about the generality of the approach (e.g. what if the recipe is unknown, what if no map is given for navigation, what if some skills are difficult to learn with RL or motion planning). Honestly, a final success rate of 68% is great, but are we really 68% there in terms of a helpful robot assistant in our kitchen? In the limitation section, it’s mentioned that “expanding to new environments in a scalable and flexible manner may require us to revisit previous assumptions and adopt new capabilities”. It would be helpful if the authors can explicitly list out all the (hidden) assumptions of the proposed method, i.e. what breaks if the method is deployed in a brand new kitchen with a brand new user companion. Lastly, it’s a bit unclear whether this paper suits the CoRL venue because to the best of my knowledge, the only components trained by the authors are the pick skill policy and the human forcaster, whereas all other components are off-the-shelf pretrained models or engineered solutions (e.g. LLMs, OWL object detector, motion planners, etc)
questions: questions: - How general is the proposed method? What are all the assumptions that the method has?
- How does the task planner break down the overall task into subtasks? Is it via LLM zero-shot capabilities, or are the recipes provided to the system?
- Pick skill still seems like a major bottleneck for the overall system. Any idea on how to further improve it?
limitations: limitations: 3
review_summary: review_summary: I would recommend weak accept since despite the room of improvement mentioned above, I believe the community still benefits a lot from reading and discussing this work due to its thorough experimental evaluation.