Multi-Task Interactive Robot Fleet Learning with Visual World Models

Authors: huihan  liu
yu  zhang
vaarij  betala
evan  zhang
james  liu
crystal  ding
yuke  zhu
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 20:26:03

Abstract: Recent advancements in large-scale multi-task robot learning offer the potential for deploying robot fleets in household and industrial settings, enabling them to perform diverse tasks across various environments. However, AI-enabled robots often face challenges with generalization and robustness when exposed to real-world variability and uncertainty. We introduce Sirius-Fleet, a multi-task interactive robot fleet learning framework to address these challenges. Sirius-Fleet monitors robot performance during deployment and involves humans to correct the robot's actions when necessary. We employ a visual world model to predict the outcomes of future actions and build anomaly predictors to predict whether they will likely result in anomalies. As the robot autonomy improves, the anomaly predictors automatically adapt their prediction criteria, leading to fewer requests for human intervention and gradually reducing human workload over time. Evaluations on large-scale benchmarks demonstrate Sirius-Fleet's effectiveness in improving multi-task policy performance and monitoring accuracy. We demonstrate Sirius-Fleet's performance in both RoboCasa in simulation and Mutex in the real world, two diverse, large-scale multi-task benchmarks. More information is available on the project website: https://ut-austin-rpl.github.io/sirius-fleet

Main paper: 
 introduction: 
 In recent years, there have been significant advancements in developing robots capable of performing various tasks [1, 2, 3] . The rapid progress in generalist robots holds great potential for deploying robot fleets [4, 5, 6] in households and industrial environments where the robots operate under a generalist multi-task policy. Despite these research advances, robots face challenges with generalization and robustness when deployed in real-world environments, which are often diverse and unstructured. These challenges undermine the safety and reliability of robot systems and limit their applicability in practical scenarios.To address these challenges, a series of works have been developed on interactive imitation learning (IIL) [7, 8, 9, 10, 11, 12, 13] and interactive fleet learning (IFL) [14, 5, 6] . Prior work on humanin-the-loop learning [7, 13, 15, 10] has proposed involving humans in real-time monitoring and correction to ensure trustworthy deployment. However, these methods often require continuous human supervision. To reduce the high human workload, runtime monitoring approaches [16, 17, 18, 19, 12, 20, 21] have been proposed. These approaches automatically monitor robot performance, identify anomalies, and ask for human control when needed. Methods like out-of-distribution (OOD) detection [20, 21, 18] and failure detection [12] have been introduced to identify anomaly cases of robot execution. However, these methods have primarily been used in single-task settings, limiting their effectiveness for large-scale, multi-task fleet deployment.We introduce SIRIUS-FLEET, a multi-task interactive robot fleet learning framework. SIRIUS-FLEET consists of a multi-task policy and a runtime monitoring mechanism. SIRIUS-FLEET enables the multi-task policy to be deployed across large robot fleets with the runtime monitoring mechanism, which requests human intervention when anomalies are predicted. SIRIUS-FLEET tackles two challenges. First, SIRIUS-FLEET enables a multi-task policy to be continually updated throughout long-term deployment. Multi-task policy training enables knowledge sharing across tasks and, therefore, improves generalization. Second, SIRIUS-FLEET builds a runtime monitoring mechanism that efficiently supervises multiple tasks simultaneously. To address the first challenge, we train a multi-task policy that can be constantly fine-tuned with deployment data. For the second challenge, we employ a visual world model as the backbone for runtime monitoring, enabling the sharing of learned representations across downstream anomaly predictors for various tasks.A key challenge in runtime monitoring is to effectively predict future action outcomes, as it allows the system to preempt potential failures before they occur. Recent advancements in world models [22, 23, 24, 25] have shown the capability of simulating future scenarios and predicting future task outcomes. Inspired by this, we develop a visual world model trained on diverse robot trajectories performing a large variety of tasks, which enables the prediction of future task outcomes and helps prevent potential failures. The visual world model is trained by reconstructing image frames from input observations, which allows it to capture fine-grained visual details necessary for precise manipulation. The learned embeddings from the world model are then shared across downstream anomaly prediction tasks. We train two distinct types of anomaly predictors: failure prediction and out-of-distribution (OOD) prediction. The two predictors complement each other -failure prediction predicts failures similar to those identified by humans previously, and OOD prediction captures cases when the robot is in novel, unfamiliar scenarios. The predictors are trained using the frozen embeddings from the visual world model and are continuously fine-tuned during deployment. Unlike prior work that uses fixed prediction thresholds, SIRIUS-FLEET automatically adjusts the anomaly predictors' criteria based on task performance and human feedback. This adaptive threshold aligns with the robot's evolving level of autonomy, resulting in more effective runtime monitoring.We evaluate our multi-task interactive robot fleet learning framework on large-scale benchmarks in simulation and real-world environments. Our key findings are: 1) our runtime monitoring system effectively supervises diverse multi-task scenarios with > 95% success rates in overall system performance, 2) the multi-task policy continually improves over time by leveraging deployment data, 3) our anomaly predictors for runtime monitoring outperform baseline methods in accuracy and reduction of human workload. In summary, our contributions are:1. A framework for multi-task interactive robot fleet learning. The multi-task robot policy efficiently improves over deployment through runtime monitoring and human interaction;2. A runtime monitoring mechanism based on a visual world model backbone with taskadaptive anomaly prediction thresholds;3. A demonstration of the high performance of our multi-task fleet learning system in both simulation and real-world environment, achieving on average > 95% success rates in system performance. 
related work: 
 Multi-Task Robot Learning. Recent advancements in multi-task robot learning have seen the development of agents capable of performing diverse tasks across various domains [2, 26, 27, 1, 28, 29, 30, 31] These advances have been driven by innovations in policy architectures [1, 32, 33, 34] , the availability of large-scale datasets [35, 36, 37, 38] , and new robotics benchmarks [32, 39] . Despite this progress, many of these models are deployed as static, one-off implementations, which limits their robustness and generalization in real-world, unstructured environments. In contrast, SIRIUS-FLEET is the first framework for robot manipulation that enables iterative improvement of multi-task policies through human-in-the-loop interaction during deployment.Robot Fleet Learning. The recent large-scale robot deployment and data collection efforts [40, 41, 42] have increased interest in robot fleet learning [43, 4, 5, 44, 45, 6] . This area of research addresses key challenges such as resource allocation [43, 14] , decentralized and federated learning [5, 44, 45] , and system management [6] . However, two significant challenges remain: supervising large robot fleets with minimal human oversight and enabling multi-task policy improvement over time using deployment data. SIRIUS-FLEET addresses these challenges by combining runtime monitoring with continuous policy updates in a multi-task, interactive fleet learning framework. This ensures that policies improve iteratively based on deployment data and human feedback.Interactive Imitation Learning. Human-in-the-loop methods have been introduced to ensure safe and trustworthy deployment by allowing robots to learn from human interventions during task execution [10, 9, 7, 13, 15, 8, 11] . To reduce the burden of constant human oversight, runtime monitoring techniques have been developed to identify anomalies during task execution [12, 17, 46, 16, 47, 48] . Two main areas of focus are unsupervised out-of-distribution (OOD) detection [18, 49, 19, 50, 20, 21] , and failure detection, which can be done either by binary classification [51, 52, 53] or by learning risk functions from trajectory data [12, 14] . Recent advancements in foundation models have also led to the use of Large Language Models (LLMs) and Vision Language Models (VLMs) to identify anomalies for policies based on these models [54, 55, 56] . While prior work has primarily focused on task-specific dynamics models for single-task settings, SIRIUS-FLEET introduces a visual world model trained on diverse datasets before deployment. This model can predict anomalies across various tasks, making SIRIUS-FLEET scalable without additional training during deployment.3 SIRIUS-FLEET: Multi-Task Interactive Robot Fleet Learning 3.1 Background 
problem formulation: 
 We formulate multi-task interactive robot fleet learning as a finite-horizon Markov Decision Process (MDP), where N robots operate in N independent MDPs. The i-th robot operates in its respective i-th MDP, defined as M i = (S, A, T , H i , µ 0 i , R i ), where S is the state space, A is the action space, T : S ×A → S is the transition dynamics, H i is the horizon length, µ 0 i is the initial state distribution, and R i : S × A → R is the reward function. In sparse-reward settings, R i is replaced with a goal predicate g i : S → {0, 1}. The collection of MDPs, {M i } N i=1 , can be reformulated as a single unified MDP with shared state space S, action space A, and transition function T . Data from all robots are aggregated to learn a unified multi-task policy π(a | s, g i ), which maximizes the expected return: max π J(π) = E st,at∼π,µ0 H t=1 g i (s t ) . We consider an interactive learning framework [7, 13, 17] for a fleet of robots [4] , where learning and deployment happens iteratively with a human policy π H in the loop. Each robot bootstraps its initial policy π 0 via behavioral cloning from a set of human demonstrations, D 0 . Starting from deployment round i = 1, each robot executes tasks using policy π i with runtime monitoring, supervised by anomaly predictors E i . During policy execution, E i determines whether the current state may lead to anomalies. Upon predicting a potential anomaly, the system signals the human to monitor the process. While monitoring, the human can choose to actively intervene [57] and take control if necessary, allowing the human policy π H (s, g) to override the robot policy π i (s, g). The resulting trajectories τ = (s t , a t , r t , c t ) are added to the data buffer D ′ of the current deployment round, where c t indicates whether timestep t is controlled by human action. The data buffer is then updated as D i+1 = D i ∪ D ′ , which is used to train the next-round policy π i+1 and anomaly predictors E i+1 .We present SIRIUS-FLEET's runtime monitoring mechanism, which supervises multiple tasks simultaneously across diverse environments during deployment. The system is designed to meet three key goals: 1) generalization-the anomaly predictors are built on shared embeddings from the visual world model, allowing them to be used across different tasks; 2) task adaptability-it adjusts dynamically to the evolving progress of each task during deployment; and 3) failure preemption-it predicts anomalies before they occur, enabling timely prediction and intervention. To achieve this, we train a visual world model that simulates task progress and supports anomaly prediction across various tasks, as illustrated in Figure 2 .Inspired by recent advances in world models [22, 23, 24, 25] , we train a visual world model on diverse robot trajectory frames to predict future task outcomes and prevent potential failures. The visual world model, trained by reconstructing image observations, predicts future latent embeddings. This world modeling approach is effective for several reasons: 1) pixel reconstruction is a readily available form of supervision, allowing the model to learn without manual annotations; 2) reconstructing image frames helps the model capture the fine-grained visual details necessary for precise manipulation tasks, and 3) it helps to develop the model's ability to predict changes in the robot's visual environment over time.SIRIUS-FLEET trains an autoregressive visual world model W as the backbone for downstream anomaly prediction. The world model W = (E γ , D λ , T ψ ) consists of an encoder E γ , a decoder D λ , and a conditional next state prediction model T ψ . In training, E γ encodes image observation x t at timestep t into latent embedding z t . The decoder D λ reconstructs z t to image xt and minimizes the image reconstruction L2 loss. T ψ inputs the history of T timesteps of embeddings (z t-T , z t-T +1 , . . . z t ), and outputs ẑt+1 . It autoregressively predicts ( ẑt+1 , ẑt+2 , . . . ẑt+L ) for L steps into the future using the last T timesteps of embeddings and reconstructed embeddings, and minimizes the embedding reconstruction loss between ( ẑt+1 , ẑt+2 , . . . ẑt+L ) and (z t+1 , z t+2 , . . . z t+L ). E γ and D λ are implemented with UNet [58, 33] , and T ψ uses conditional Variational Autoencoder (cVAE) [59] . T ψ is jointly trained with E γ , D λ on the same latent space.We use a stochastic latent space rather than a deterministic one since the stochastic latent space of cVAE supports multiple future sampling and facilitates better prediction [60, 17] . We use transformer architecture [61] for the encoder, decoder, and prior network for the cVAE in T ψ .The visual world model captures changes in task outcomes over time. Its learned representation can be used for the anomaly predictors to predict future anomalies. Since individual tasks vary, we train task-specific anomaly predictors on the frozen representation to predict failures and OOD anomalies. Out-of-Distribution (OOD) Prediction. We identify OOD states using k-means clustering. The frozen visual world model W generates embeddings from sampled trajectories in the data buffer. We use Principal Component Analysis (PCA) to reduce the embedding dimensions to l and calculate c k-means centroids for each task. To predict OOD for an embedding z, we reduce its dimensions, find the nearest centroid, and calculate its L2 distance d. A state is identified as OOD if d exceeds the task threshold α g . α g is determined by d θ g , which is the distance of the top θ g percentile of the distances to the nearest centroids from the validation latent embeddings. For efficiency, we perform task-specific k-means clustering on image embeddings for the specific task without training additional models.Adaptive Decision Boundaries. Anomaly predictors should adjust their anomaly prediction threshold as robot performance improves during deployment. In a multi-task setting, the varying task performances across different tasks pose additional challenges. We adjust anomaly predictors by loosening the decision boundaries for high-performing tasks and tightening them for low-performing tasks. For failure prediction, we finetune it using the most recent round of deployment data D ′ , with human intervention labels reflecting the updated human perceived risk.The decision boundary expands for OOD prediction as the robot encounters more in-distribution states. We sample from D i , the aggregated deployment data for all rounds, for k-means clustering. For each new embedding z, we use a threshold α g for its distance d to its nearest centroid, which depends on θ g . We update the prediction threshold α g based on the human intervention ratio p H , which acts as a proxy for policy performance. We fit the exponential decay function, θ g = a + be cp H , and obtain the parameters by fitting the curve to a set of calibration trajectories. Empirically, a = 95.2, b = -17.7 and c = -3.2. We found this function to be robust across all rounds and all tasks from simulation to real-world experiments, and we used the same function and set of hyperparameters throughout the simulation and real-world experiments. At deployment, the embedding z is computed from observations, and the distance d from the nearest centroid is compared to α g . If d > α g , the sample is identified as OOD. Figure 3 (Top) shows the decision boundary, while Figure 3 (Bottom) shows the fitted curve for θ g .Continual Model Improvement. While the visual world model W is trained once and frozen, the policy and the anomaly predictors are continually finetuned over deployment: the policy is updated with the data from previous rounds using weighted sampling, the failure classifier for failure prediction is finetuned with the most recent intervention labels, and the K-means clustering in OOD prediction expands its latent space coverage after each round.Anomaly Predictors at Runtime. During deployment, the visual world model predicts each task's future embeddings over L steps. We generate N possible future scenarios by sampling from the cVAE latent space and making L-step predictions N times. Each predicted future embedding is then assessed individually by the anomaly predictors. For failure prediction, we compute the average failure score across future embeddings. For OOD prediction, we calculate the average distance of each future state to its nearest cluster centroid and compare the final average distance against the OOD threshold.Multi-Task Policy Training. We train a Transformer-based [61] multi-task policy, as shown in Figure 4 . The policy inputs image observations, robot proprioceptive data, language task goals, and output robot actions. The design follows that in RoboMimic [32] and RoboCasa [39] . 
experiments: 
 In our experiments, we aim to address the following questions: 1) How well does SIRIUS-FLEET continue to improve its policy during deployment with effective runtime monitoring? 2) How does SIRIUS-FLEET's runtime monitoring performance compare with baseline methods? 3) Are the anomaly prediction made by SIRIUS-FLEET at the right moments, qualitatively?To measure the effectiveness of SIRIUS-FLEET's multi-task fleet learning, we evaluate how both the policy and the runtime monitoring perform and evolve over time. Following prior works [62, 13, 17] , we evaluate the system in a human-in-the-loop setting through rounds of iterative deployment.Evaluation Setting. Our evaluation spans three rounds of policy updates and runtime monitoring. Full human supervision is required in the first round since no anomaly predictors have been trained yet. Human supervision is only requested in the next two rounds when an anomaly is predicted. After each round, the collected deployment data and intervention labels are used to train the policy and anomaly predictors for the next round. The initial policy is trained with 50 human demonstrations per task over 1200 epochs. We collect 100 rollouts per task for each round to ensure consistent data size and finetune the policy for 400 epochs.Evaluation Metrics. We evaluate SIRIUS-FLEET's policy using Autonomous Performance, which measures the policy's ability to achieve its goals without human intervention. After each round, the policy is finetuned on the newly collected data, and we evaluate its success rate without runtime monitoring to assess performance improvements over time. To evaluate runtime monitoring, we use Combined Policy Performance (CPP) and Return of Human Effort (ROHE), following [12, 17] . CPP measures the overall system success rate under monitoring, reflecting the effectiveness of human-robot collaboration in preventing anomalies. ROHE assesses the efficiency of human intervention by comparing the policy's success rate against the amount of human intervention effort, calculated as the ratio of policy performance to the number of interventions:Normalized ROHE = Eτ [ Tτ t=0 r τ t ] 1+ H T .The goal is to maximize performance while minimizing human workload.Evaluation Environments. For simulation, we use RoboCasa [39] , a visually diverse benchmark with various objects, layouts, and scenes. RoboCasa contains both a large dataset using MimicGen [38] , an automated trajectory generation method (5k trajectories per task), and a human demonstration dataset (50 human demonstrations per task). The visual world model is trained on 20 task suites of MimicGen data, and the policy is trained on 12 task suites of human demonstration data. We use Mutex [27] for real-world experiments, a commonly used benchmark for multi-task learning. The visual model is trained on 50 tasks, and the policy is trained on 10 tasks across 5 task suites. We use an OSC controller with 7D action space (x-y-z position, yaw-pitch-roll orientation, and gripper).System Performance Over Time. Figure 6 presents the performance of SIRIUS-FLEET over several deployment rounds. The autonomous policy improves consistently, showing a 13% increase in simulation and 45% in real-world settings. SIRIUS-FLEET maintains high combined policy performance, exceeding 95% in both environments, demonstrating the effectiveness of runtime monitoring and timely human intervention. Additionally, SIRIUS-FLEET shows improved ROHE performance Since these baselines are designed for singletask environments, we evaluated them on five tasks from different categories in RoboCasa, training separate models for each task. All baselines used fixed detection thresholds, while ThriftyDAgger adjusts thresholds based on a target intervention ratio. For fairness, we applied the same intervention ratio across all comparisons. As shown in Figure 7 , SIRIUS-FLEET consistently outperforms the baselines in policy performance and ROHE. The fixed thresholds used by the baselines were ineffective in handling varying task distributions, resulting in excessive interventions or decreased performance across tasks.Ablations. We conducted ablation studies to explore the following key questions: 1) How important is the design of the multi-task visual world model, as opposed to the single-task design? 2) What are the individual impacts of the Failure Prediction and OOD Prediction components? 3) How critical is the design of the multi-task policy? The results of these studies are detailed in Appendix 6.2. 
conclusion: 
 We introduce SIRIUS-FLEET, a framework for multi-task interactive robot fleet learning that combines a multi-task policy with runtime monitoring. Driven by a visual world model and task-adaptive anomaly predictors, SIRIUS-FLEET improves policy performance and reduces human intervention through timely anomaly prediction.Limitations. SIRIUS-FLEET is best suited for quasi-static manipulation tasks, where anomalies can be easily corrected with teleoperation. However, applying SIRIUS-FLEET to dynamic tasks could be challenging. Also, our experiments involve a small group of five human operators; future work could expand this by conducting large-scale human studies to understand the effects of diverse human interventions on runtime monitoring and policy training. Lastly, our evaluations used a single embodiment (Franka Emika Panda arm). Future research can extend SIRIUS-FLEET to crossembodiment fleet learning, making it more generalizable across different robot platforms [36] .6 Appendix 
table of contents: 
 • Ablations (Section 6.2)• Implementation Details (Section 6.3)• Qualitative Analysis and Discussion (Section 6.4)• Additional Details on Tasks (Section 6.5)Multi-Task World Model vs. Single-Task World Model. We empirically show how the multi-task world model is more generalizable than the single-task version in terms of future latent state prediction accuracy. Accurate future latent state prediction is essential for effective error prediction, as the downstream error predictors directly use the predicted future states. We compared the performance of SIRIUS-FLEET's multi-task world model against six single-task world models in terms of mean squared error (MSE) for future latent state prediction. As shown in Table 1 , the multi-task world model consistently outperforms the single-task models across all tasks. This indicates that the multi-task world model is crucial for providing more accurate state predictions.Ablation of Failure and OOD Prediction. We examine the impact of OOD and failure prediction components on human intervention overlap accuracy: We have a human operator fully supervise the robot execution of a policy and can intervene whenever an unsafe state is observed. We then apply the learned error predictors to the collected trajectories and compare failure states identified by the error predictors with those intervened by the human operator. This shows how the error predicted aligns with humans' risk assessment.As shown in Table 3 , SIRIUS-FLEET's combined use of OOD and failure prediction significantly outperforms using either component alone across all tasks. This suggests that both accurate OOD prediction and failure prediction are critical to the overall performance of the policy, as they complement each other in identifying potential errors during deployment.Multi-task Policy vs. Single-task Policy. We show that training on multi-task settings improves the overall performance of robot policy than training on individual single tasks. We show the policy success rate (%) for 7 different tasks in Figure 8 , where we compare the multi-task policy trained on all tasks with single-task policies trained on individual tasks. For most tasks, SIRIUS-FLEET's multi-task policy performs better than the single-task policy trained on that specific task. This aligns with the observation in prior works [31, 36, 2] that multi-task training helps the policy generalize better.We present the details of hyperparameters for training the world model, policy, and error predictors in the following table : • World Model Architecture: Table 3 ;• Policy Architecture: Table 4 ;• Training: Table 5 ;• Failure Prediction: Table 6 ;• OOD Prediction: Table 7 . Can SIRIUS-FLEET catch important errors and give meaningful intervention timings? We conduct a qualitative analysis of where the error predictors predict error during robot execution using active human monitoring. Specifically, we have a human operator who fully supervises the robot's policy execution and can intervene whenever an unsafe state is observed. We then apply the learned error predictors to the collected trajectories and compare failure states identified by the error predictors with those intervened by the human operator. This shows how the error predicted aligns with humans' risk assessment.We show examples of the human intervention region and errors predicted by SIRIUS-FLEET and the baselines in Figure 9 to 14. We use the same trajectory for all methods. The three instances humans gave interventions are: 1) the robot is not aiming at the object it is grasping; 2) the robot pauses at grasping; 3) the arm is too stiff and does not bend down. We visualize if the errors given by SIRIUS-FLEET and the baselines are aligned with the intervention humans gave.The green area indicates times of actual human intervention; the blue plot indicates the error value predicted by the different methods. We also show the video frame at the corresponding timings to illustrate the failure modes captured (or missed) by the different methods.Note that SIRIUS-FLEET, PATO, and ThriftyDAgger each have two separate components, so we visualize each individually. PATO and ThriftyDAgger share the same ensemble uncertainty component, so we visualize one of them.As shown in Figure 9 and 10, our method can capture a similar error pattern to that the human judges. MoMaRT (Figure 11 ) is prone to predicting false positives due to pixel changes. PATO (VAE part, Figure 12 ) often predicts false negatives; PATO (ensemble part, Figure 13 ) often has high oscillations and extreme values. ThriftyDAgger's (Figure 14 ) risk Q function is data-intensive to train and often has generalization errors; it cannot accurately reflect the task progress and local failure modes.  
 
 Appendix: Single Task Multi Task We show the policy success rate (%) for different tasks, comparing the multi-task policy trained on all tasks with single-task policies trained on individual tasks. SIRIUS-FLEET 's multi-task policy gives overall better performance. Note that for CoffeeSetupMug and PnPCounterToSink, the single-task policy gives 0% success rate. CloseDoorSingleHinge 3.3 × 10 -4 1.5 × 10 -4 PnPCabToCounter 3.1 × 10 -4 1.7 × 10 -4 TurnOnMicrowave 1.7 × 10 -4 1.3 × 10 -4 TurnOnSinkFaucet 3.2 × 10 -4 1.2 × 10 -4 CoffeeSetupMug 4.4 × 10 -4 1.8 × 10 -4Unnecessary: object is already grasped Necessary: not aiming at the object Necessary: arm too stiff; does not bend down Figure 14 : ThriftyDAgger: Risk Q-Function. The y-axis represents the q-value of the current state-action pair, where a higher q-value means less risk of failure. The video frame visualization is omitted here since no clear error modes have been predicted. For the initial BC policy, we train the BC transformer policy for 2000 epochs. For each subsequent round, we finetune the initial BC policy on the newly aggregated data for another 800 epochs. We use this finetuned BC policy checkpoint to perform runtime monitoring experiments.fixed set of scenes and objects. For example, The CoffeeServeMug environment can contain a combination of diverse backgrounds, layouts, coffee machine types, and coffee mug types, making it a challenging environment for generalization. As noted in Section 4.1, We use all 20 suites to train the visual world model and 12 of them for policy learning and runtime monitoring. The list of 10 tasks for policy learning and runtime monitoring experiments is shown in Figure 16 . For the initial BC policy, we train the BC transformer policy for 1000 epochs. For each subsequent round, we finetune the initial BC policy on the newly aggregated data for another 400 epochs. We use this finetuned BC policy checkpoint to perform runtime monitoring experiments. 


Decision: This paper proposes a framework for multi-task interactive robot fleet learning. The reviewers find the idea and the results promising. However, the reviewers have raised several concerns including a lack of clarity on various aspects in the approach, questions about the novelty with respect to the state of the art, lack of clarity in the evaluations and experiment settings, lack of ablations, details on the implementation are missing, the writing has to be significantly improved, unclear when the proposed approach requires human intervention, among others. Each of the reviewers has listed several questions to be addressed in the rebuttal.
Post-rebuttal: Most of the reviewers' concerns have been sufficiently addressed. Please address the remaining suggestions in the final version.

Reviews: review: title: title: Use of learned World Model to predict failure and OOD is a promising idea. Shows good performance with respect to compared methods. Lack of clarity in terms of novelty. Lack of clarity in technical details. Ablations needed in experiments.
paper_summary: paper_summary: This paper proposes a human-in-the-loop interactive, multi-task, continuously improving learning system for a fleet. There are two main components. First, a visual World Model is used to predict future visual states. The latents of this World Model predictions are used to predict whether a particular robot execution may be failing or out-of-distribution. Based on this the human supervisor is prompted to attend and intervene if necessary. The data generated is used to train an imitation learning policy for the next round (i.e., continuously improving).
main_review: main_review: Strengths:

The use of a visual World Model to predict failure and to identify out-of-distribution executions is a promising idea. 

The paper is clearly presented at the high-level. (But important lack of clarity exists at the level of technical details.)

The presented set of results show good performance. 


Weaknesses:

*Lack of clarity in technical details*

A variety of details were not clear for me. 

The paper says "We train a failure classifier F on the frozen latent space for each specific task." This implies the tasks are discrete and are known beforehand? But the "language embedding" input in Fig. 2 implies task is specified as a language input, so it is not a predefined set of discrete tasks?

If the failure classifier can be trained for each task, why not also train other components for each task? Would the World Model perform better if it was task-specific?

Is the OOD prediction also task-specific? Not mentioned?

In the evaluation, how are the three rounds selected? How different are they from each other?

Not clear how the Evaluation Metrics are calculated. For example, how is the Autonomous Performance calculated, if, over all 3 rounds, the system required human intervention? Or is the Autonomous Performance calculated independent of the 3 rounds, on some other dataset after the 3 rounds are over? Or are they only calculated over the timesteps that were autonomous? These were not clarified at all.

Who were the human controllers in the experiments? Were they the authors? How was it ensured that human controllers were not biased to favor the proposed method?

What are the actions of the policy? Joint actions? End-effector motion? The text says "motor action" which implies joint space actions. Figure 2 shows end-effector motions. Not clear.

*Lack of clarity in terms of novelty*

I am not saying the work is not novel. But the Related Work section was not clear enough in its language to convince me that the work is novel. It does not say clearly: Is this the first time a World Model is used to predict failure/OOD? Is this the first time a multi-task policy is learned through interactive imitation learning? If yes, these should be clearly stated. If no, what are these other works and how does this work relate to those?

Related to the above, the significance of the "fleet" in this work is not perfectly clear. It could have very well been a single robot performing multiple tasks one after the other. There does not seem to be any reasoning on the performance of the whole fleet in the current round when making a decision about which robot to attract the human's attention to. Or is there? 

*Ablations needed in experiments*
It is necessary to evaluate the significance of the particular design decisions. What would happen if the World Model was trained on particular tasks? What would happen if the policy were trained on particular tasks?
questions: questions: I have listed my questions above.
limitations: limitations: 1
review_summary: review_summary: Use of learned World Model to predict failure and OOD is a promising idea. Shows good performance with respect to compared methods. Lack of clarity in terms of novelty. Lack of clarity in technical details. Ablations needed in experiments.
title: title: Good motivation, lacks ablations and insights on what aspect of the model works.
paper_summary: paper_summary: This work proposes a framework for multi-task interactive training of robots. The goal is to automate the failure / OOD detection to seek human help. The framework consists of two parts: (1) pre-training the multi-task world model, and (2) enabling continual policy training via detecting failures and asking for human intervention. The work introduces an automatic way of detecting failures (based on a classifier) and OOD (based on k-means). The experiments are shown on a wide variety of tasks: 12 tasks in simulation, and 10 tasks on real-world Panda robot.
main_review: main_review: **Strengths**

1. An important work with good motivation -- multitask policy learning is already of immense interest to the robot learning community and learning with human interventions (and that too minimal interventions) is a topic of close interest to the robot learning audience.

**Weaknesses**

1. Claims: The authors claim that their work proposed "A runtime monitoring mechanism that is based on visual world model backbone and comes 67 with task-adaptive error prediction thresholds;" -- which even [1] does. 

2. On similar lines to comment 1 -- Is it an accurate statement to say that this work is an extension of [1] to multi-task policies? The related work section needs to have a dedicated comparison with this as the core components -- dynamics model, OOD detection, and failure detection are the same. The claim on L141-144 appears to be the same as the Runtime monitoring system in [1]. 


3. I would like to see what aspect of the proposed model is the bottleneck for policy training. It is unclear from the combined results if an accurate world model or accurate error prediction or accurate OOD detection is crucial. Ablations on that end should be included to provide insights so as to what component of the model is crucial (or are they all equally crucial). Refer to [1] for some idea on ablations.


4. Time analysis: What is the wall-clock time taken for FIREWORK compared to the baselines (ThriftyDagger, PATO, and MoMaRT)?


5. Similar to Fig. 12 (in appendix), I would like to see the # human interventions and # failures plot for rounds 2 and 3. This is primarily to see if these quantities decrease/increase from rounds 2 --> 3.

6. There are hardly any implementation details mentioned either in the main paper or in the appendix. The model is highly non-trivial with different components -- hence without the implementation details or pseudocode it will be incredibly difficult to reproduce this work.

7. [Overall writing of paper]: The writing of paper needs to improve. Specifically basing the narrative on top of [1] by introducing [1] as a preliminary could be one option. I found it relatively hard to read the paper (generally speaking). A lot of things were much clearer to me after going through [1]. I leave it upto the authors to revisit the writing.

8. [Very Minor concern] The real-world experiments have been pushed to the appendix. I'd highly recommend authors to move it to the main manuscript. 

---
**References**

[1] Model-based runtime monitoring with interactive imitation learning, Huihan Liu et al., ICRA 2024.


-----

----

**POST REBUTTAL UPDATE**

I thank authors for their rebuttal experiments. Most of my concerns are addressed and hence I'm increasing the score to **Weak Accept**. I'm still not satisfied with their reply regarding the comparison between [1]. 

- Huihan Liu et al's [1] work is extendable to a multi-task policy and world model, specifically no assumption in [1]'s framework particularly prohibits from extending to a multi-task setting.The authors need to be clear about that and say that they are in fact extending the work (no harm in extending an existing work, in my humble opinion that by no means reduces a work's novelty). 

- I would not suggest authors to use MSE as a metric to evaluate world models. MSE is particularly deceiving metric as shown in [2]. The correct way to show that a world model is better than other in a robot learning context is to evaluate it under a policy. Also, I'm little suspicious regarding the experiment results in Table 1 -- was the same amount of data given to the single-task WM? It does not make sense why a multi-task WM does better when compared to a dedicated WM for that task unless there's some difference in the data provided to both the models or some other disparity.

Again, since my other concerns were addressed I've decided to bump up the score to **Weak Accept** and I hope that the authors would carefully consider the post-rebuttal feedback and incorporate them in their work.

**Additional References**

[2] Lambert, Nathan, et al. "Objective mismatch in model-based reinforcement learning." L4DC 2020.
questions: questions: I would like the authors to address the concerns raised in the **Weaknesses** section and perform the experiments asked there (comments 3, 4, 5). In addition to that here are 2 questions on the design choice of the model and baselines that I have.

9. The authors mention that the world model is frozen. Why was this a design choice? This inherently assumes that the trajectories on which the world model is trained are the complete set of optimal trajectories, however, that need not be true. Also, when there is a deviation in the trajectory and the human intervenes and describes the right trajectory, the dynamics model needs to be aware of this deviated-but-successful trajectory. Am I missing something here? 

10. For the baselines is the same architecture of the world model used? (Authors do mention that the policy architecture is the same.)
limitations: limitations: 3
review_summary: review_summary: As it stands, there are several experiments that I've asked for and have some concerns on the core claims and implementation details. Based on this I am currently voting for a weak reject of the paper, however I will give the final recommendation after look at other reviews, authors' rebuttal and the discussion during rebuttal.
title: title: Firework Review
paper_summary: paper_summary: Firework is a framework for multi-task interactive robot fleet learning with a human in the loop and runtime monitoring. The approach uses a visual world model to predict task outcomes and is leveraged as an error predictor.  This approach is compared to MoMaRT, PATO & ThriftyDAgger, and shows improvements in terms of Combined Policy Performance and Return of Human Effort (ROHE).
main_review: main_review: Quality:

This work presents an iterative approach to training and managing an interactive robot fleet. It is well motivated and shows improvements over SOTA baselines.

Clarity:

For Figure 6. Top figure for Turn Off Microwave, what is the y axis referring to?  This figure could use some additional context to describe what conclusions a reader should draw from it. In Sec 5 Analysis and Discussion end with a sentence “We present more model analysis and discussion in the appendix. ” It was unclear to me what section of the appendix this is referring to.

Fig 6 Bottom Figure is a nice illustration, it would be helpful to have a similar set of overlays for the other baseline methods as well (MoMaRT, PATO, ThriftyDAgger), as well as snapshots of camera views at the moments of high probability failure to help the reader understand what is happening. This figure could be greatly expanded on (potentially in Appendix) to help the reader contrast your method to the other baselines.

Originality:

This work’s strength is that it shows improvements over SOTA baselines, and is a nice implementation of individual framework components, not that the problem or approach is highly novel.


Significance:

This work has shown to be SOTA for policy learning with human interventions for an interactive fleet. Hopefully the code can be released so that others can benchmark against it.
questions: questions: Figure 6 is not clear as written and is a missed opportunity to highlight how your method compares to the baselines.
limitations: limitations: 3
review_summary: review_summary: This work has shown to be SOTA for policy learning with human interventions for an interactive fleet. The work could be made stronger by more clearly showing when this approach requires human interventions compared to baselines.