Multi-Task Interactive Robot Fleet Learning with Visual World Models
Authors: Huihan Liu
Yu Zhang
Vaarij Betala
Evan Zhang
James Liu
Crystal Ding
Yuke Zhu
Venue: CoRL 2024 (2024)
Keywords: Robot Manipulation
Interactive Imitation Learning
Fleet Learning
TL;DR: We proposed a framework for multi-task interactive robot fleet learning with visual world models.

Abstract: Recent advancements in large-scale multi-task robot learning offer the potential for deploying robot fleets in household and industrial settings, enabling them to perform diverse tasks across various environments. However, AI-enabled robots often face challenges with generalization and robustness when exposed to real-world variability and uncertainty. We introduce Sirius-Fleet, a multi-task interactive robot fleet learning framework to address these challenges. Sirius-Fleet monitors robot performance during deployment and involves humans to correct the robot's actions when necessary. We employ a visual world model to predict the outcomes of future actions and build anomaly predictors to predict whether they will likely result in anomalies. As the robot autonomy improves, the anomaly predictors automatically adapt their prediction criteria, leading to fewer requests for human intervention and gradually reducing human workload over time. Evaluations on large-scale benchmarks demonstrate Sirius-Fleet's effectiveness in improving multi-task policy performance and monitoring accuracy. We demonstrate Sirius-Fleet's performance in both RoboCasa in simulation and Mutex in the real world, two diverse, large-scale multi-task benchmarks. More information is available on the project website: https://ut-austin-rpl.github.io/sirius-fleet

Multi-Task Interactive Robot Fleet Learning with Visual World Models
Huihan Liu
Yu Zhang
Vaarij Betala
Evan Zhang
James Liu
Crystal Ding
Yuke Zhu
The University of Texas Austin
Abstract: Recent advancements in large-scale multi-task robot learning offer the potential for deploying robot fleets in household and industrial settings, en- abling them perform diverse tasks across various environments. However, AI- enabled robots often face challenges with generalization and robustness when posed to real-world variability and uncertainty. We introduce a multi-task interactive robot fleet learning framework to address these chal- lenges. SIRIUS-FLEET monitors robot performance during deployment and in- volves humans to correct the robot's actions when necessary. We employ vi- sual world model predict the outcomes of future actions and build anomaly predictors predict whether they will likely result in anomalies. As the robot autonomy improves, the anomaly predictors automatically adapt their prediction criteria, leading to fewer requests for human intervention and gradually reduc- ing human workload over time. Evaluations on large-scale benchmarks demon- strate STRIUS-FLEET'S effectiveness in improving multi-task policy performance and monitoring accuracy. We demonstrate STRIUS-FLEET'S performance in both RoboCasa in simulation and Mutex in the real world, two diverse, large-scale multi-task benchmarks. More information is available the project website: https: autt-aussinn in- rpl. github. io/ sirriis--iii fleet
Keywords: Robot Manipulation, Interactive Imitation Learning, Fleet Learning
Introduction
In recent years, there have been significant advancements in developing robots capable of perform- ing various tasks [1, 2, 3]. The rapid progress in generalist robots holds great potential for deploying robot fleets [4, 5, 6] in households and industrial environments where the robots operate under generalist multi-task policy. Despite these research advances, robots face challenges with gener- alization and robustness when deployed in real-world environments, which are often diverse and unstructured. These challenges undermine the safety and reliability of robot systems and limit their applicability in practical scenarios.
To address these challenges, a series of works have been developed on interactive imitation learning (IIL) [7, 8, 10, 11, 12, 13] and interactive fleet learning (IFL) [14, 5, 6]. Prior work on human- loop learning [7, 13, 15, 10] has proposed involving humans in real-time monitoring and correction to ensure trustworthy deployment. However, these methods often require continuous human supervision. To reduce the high human workload, runtime monitoring approaches [16, 17, 18, 19, 12, 20, 21] have been proposed. These approaches automatically monitor robot performance, identify anomalies, and ask for human control when needed. Methods like antoof-distrbbution (OOD) detection [20, 21, 18] and failure detection [12] have been introduced to identify anomaly cases of robot execution. However, these methods have primarily been used in single-task settings, limiting their effectiveness for large-scale, multi-tate task fleet deployment.
We introduce SIRIUS- -LEETT, a multi-task interactive robot fleet learning framework. SIRIUS- FLEET consists of a multi-task policy and a runtime monitoring mechanism. SIRIUS-FiN FLEET enables
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Visual World Model Training & Inference
Multi-Task Interactive Fleet Learning
Visual World Model
Multi-Task Fleet Deployment
Prodict latent embeeddings
Anomaly?
Human
Control
Anomally Predictors
Open
Microwave
Policy
past video frames
finatunad
finatuned
Up
Serving
Pretrained large diverse datasets
Coffen
Place
Model Update
Overview of SIRIUS-FLEET. Our framework of multi-task interactive robot fleet learn- ing consists of two stages: 1) Visual World Model Training & Inference, where we pre-train visual world model on diverse datasets to predict future latent embeddings from past visual obser- vations, and 2) Multi-Task Interactive Fleet Learning, where the pre-trained world model is used to supervise multi-task robot fleet deployment. During deployment, anomaly predictors monitor task performance in real time, soliciting human feedback when necessary. The policy and anomaly pre- dictors are continuously fine-tuned with deployment data, improving task performance over time.
Figure I:
the multi-task policy to be deployed across large robot fleets with the runtime monitoring mecha- nism, which requests human intervention when anomalies are predicted. SIRIUS-FLEET tackles two challenges. First, SIRIUS- -LEET enables a multi-task policy be continually updated through- long-term deployment. Multi-task policy training enables knowledge sharing across tasks and, therefore, improves generalization. Second, HIRIUS-FLEEE builds runtime monitoring mechanism that efficiently supervises multiple tasks simultaneously. To address the first challenge. train multi-task policy that can be constantly fine-tuned with deployment data. For the second challenge, we employ a visual world model as the backbone for runtime monitoring, enabling the sharing of learned representations across downstream anomaly predictors for various tasks.
A key challenge runtime monitoring is to effectively predict future action outcomes, as it allows the system to preempt potential failures before they occur. Recent advancements in world mod- els [22, 23, 24, 25] have shown the capability of simulating future scenarios and predicting future task outcomes. Inspired by this, we develop a visual world model trained on diverse robot trajec- tories performing large variety of tasks, which enables the prediction of future task outcomes and helps prevent potential failures. The visual world model trained by reconstructing image frames from input observations, which allows capture fine- mme-grained visual details necessary for precise manipulation. The learned embeddings from the world model are then shared across downstream anomaly prediction tasks. We train two distinct types of anomaly predictors: failure prediction and out-of-distribution (OOD) prediction. The two predictors complement each other failure prediction predicts failures similar to those identified by humans previously, and OOD prediction captures cases when the robot is in novel, unfamiliar scenarios. The predictors are trained using the frozen embeddings from the visual world model and are continuously fine-tuned during deploy- ment. Unlike prior work that uses fixed prediction thresholds, STRIUS-FLEET automatically adjusts the anomaly predictors' criteria based task performance and human feedback. This adaptive threshold aligns with the robot's evolving level of autonomy, resulting in more effective runtime monitoring.
We evaluate our multi-task interactive robot fleet learning framework on large-scale benchmarks in simulation and real-world environments. Our key findings are: 1) our runtime monitoring system effectively supervises diverse multi-task scenarios with > 95% success rates in overall system per- formance, 2) the multi-task policy continually improves over time by leveraging deployment data, 3) anomaly predictors for runtime monitoring outperform baseline methods in accuracy and reduction of human workload. In summary, our contributions are:
1. A framework for multi-task interactive robot fleet learning. The multi-task robot policy efficiently improves over deployment through runtime monitoring and human interaction;

---

2. A runtime monitoring mechanism based on visual world model backbone with task- adaptive anomaly prediction thresholds;
3. A demonstration of the high performance of our multi-task fleet learning system in both simulation and real-world environment, achieving on average > 95% success rates in sys- tem performance.
2 Related Work
Multi-Task Robot Learning. Recent advancements in multi-task robot learning have seen the
development of agents capable of performing diverse tasks across various domains [2, 26, 27, 1, 28, 29, 30, These advances have been driven by innovations in policy architectures [1,32, 33, 34], the availability of large-scale datasets [35, 36, 37, 38), and new robotics benchmarks [32, 39J. Despite this progress, many of these models are deployed static, one-off implementations, which limits their robustness and generalization in real-world, unstructured environments. In contrast, SIRIUS- FLEET is the first framework for robot manipulation that enables iterative improvement of multi-task policies through human-in-the-loop interaction during deployment.
Robot Fleet Learning. The recent large-scale robot deployment and data collection efforts [40, 41, 42] have increased interest in robot fleet learning [43, 4, 5, 44, 45, 6]. This area of research addresses key challenges such as resource allocation [43, 14], decentralized and federated learning [5, 44, 45], and system management [6]. However, two significant challenges remain: supervising large robot fleets with minimal human oversight and enabling multi-task policy improvement over time using deployment data. PIRIUS-FLEET addresses these challenges by combining runtime monitoring with continuous policy updates in a musk, interactive fleet learning framework. This ensures that policies improve iteratively based on deployment data and human feedback.
Interactive Imitation Learning. Human-in-the-loop methods have been introduced ensure safe and trustworthy deployment by allowing robots to learn from human interventions during task execution [10, 9, 7, 13, 15, 8, 11]. To reduce the burden of constant human oversight, runtime monitoring techniques have been developed to identify anomalies during task execution [12, 17, 46, 16, 47, 48). Two main areas of focus are unsupervised out-of-distribution (OOD) detec- tion [18, 49, 19, 50, 20, 21], and failure detection, which can be done either by binary classification [51, 52, 53] or by learning risk functions from trajectory data [12, 14]. Recent advancements in foundation models have also led to the use of Large Language Models (LLMs) and Vision Lan- guage Models (VLMs) to identify anomalies for policies based these models [54, 55, 56). While prior work has primarily focused on task-specific dynamics models for single-task settings, SIRIUS- FLEET introduces a visual world model trained on diverse datasets before deployment. This model can predict anomalies across various tasks, making SIRIUS-FLEES scalable without additional train- ing during deployment.
SIRIUS-FLETT: Multi-Task Interactive Robot Fleet Learning
3.1 Background
3.1.1
Problem Formulation
We formulate task interactive robot fleet learning as a horizon Markov Decision Process (MDP), where robots operate in N independent MDPs. The i-th robot operates in its respective i-th MDP, defined as M; = (S, A, T, H,, p., R2), where S is the state space, A is the action space, T SX A S is the transition dynamics, H; is the horizon length, n is the initial state distribution. and R₂ S x A + R is the reward function. In pparse-reware settings, R& is replaced with goal predicate 9, S - 13. The collection of MDPs, J.M.NN. can be reformulated as a single unified MDP with shared state space S, action space A, and transition function T. Data from all robots are aggregated to learn a unified multi-task policy π (a 8, 9.9;), which maximizes the expected
return: max, (TT = Egr Po
Etz (6()))
3

---

Visual World Model
Predicted Future Embeddings
Anomaly Predictors
Frozen Pretraining
Finetuned Over Time
Transformer cVAE
Failure Detection
T--T+1
Pr
Failurr Classiffer
EnT
Minimizing
Z/z
Zpei
Re
End
EnI
Einc
OOD Detection K-means Clustering
Training Only
Figure 2: Model Architecture. The visual world model comprises UNet-based encoder and decoder combined with vAE- and Transformer-basee prediction model. This architecture allows the world model to predict future embeddings from the current state. The learned representations are then used for anomaly predictions, including failure and OOD prediction.
3.1.2 Multi-Task Interactive Fleet Deployment
We consider an interactive learning framework [7, 13, 17] for a fleet of robots [4], where learning and deployment happens iteratively with human policy н in the loop. Each robot bootstraps its initial policy mo via behavioral cloning from a set of human demonstrations, D Starting from deployment round 1, each robot executes tasks using policy म, with runtime monitoring, supervised by anomaly predictors E,. During policy execution, E, determines whether the current state may lead to anomalies. Upon predicting a potential anomaly, the system signals the human to monitor the process. While monitoring, the human can choose to actively intervene [57] and take control if necessary, allowing the human policy T T (s.g) to override the robot policy (8, The resulting trajectories т (st, Qr, Tt, ct) are added to the data buffer D' of the current deployment round, where e indicates whether timestep is controlled by human action. The data buffer is then updated Di+1 D UD', which used train the next-round policy mTч and anomaly predictors E
3.2 Runtime Monitoring for Multi-Task Fleet Deployment
We present SIRIUS-FLEET'S runtime monitoring mechanism, which supervises multiple tasks si- multaneously across diverse environments during deployment. The system is designed to meet three key goals: 1) generalization-the anomaly predictors are built on shared embeddings from the vi- sual world model, allowing them to be used across different tasks; 2) task adaptability- adjusts dynamically to the evolving progress of each task during deployment; and 3) failure preemption-in predicts anomalies before they occur, enabling timely prediction and intervention. To achieve this, we train a visual world model that simulates task progress and supports anomaly prediction across various tasks, as illustrated in Figure?
3.2.1 Training the Visual World Model
Inspired by recent advances world models [22, 23, 24, 25], we train a visual world model on diverse robot trajectory frames to predict future task outcomes and prevent potential failures. The visual world model, trained by reconstructing image observations, predicts future latent embeddings. This world modeling approach is effective for several reasons: 1) pixel reconstruction is readily available form of supervision, allowing the model to learn without manual annotations; 2) recon- structing image frames helps the model capture the fine-grained visual details necessary for precise manipulation tasks, and 3) it helps to develop the model's ability to predict changes in the robot's visual environment over time.
MIRIUS-FLERS trains an autoregressive visual world model W the backbone for downstream anomaly prediction. The world model W = (F.y, Dx, Tx) consists of an encoder E,y, decoder

---

Da, and conditional next state prediction model Ty. In training, E, encodes image observa- tion 3, at timestep t into latent embedding zt- The decoder Dx reconstructs to image & and minimizes the image reconstruction L. loss. To inputs the history of timesteps of embed- dings (z It. 2t), and outputs z(411 It autoregressively predicts (2f+1 22, ze4 for L steps into the future using the last T timesteps of embeddings and reconstructed em- beddings, and minimizes the embedding reconstruction loss between (26-1; 2442,.... 2 L) and (#1+1, Fe+2 (ttL)) E-y and D2 are implemented with UNet [58, 33], and T' uses conditional Variational Autoencoder (cVAE) [59]. Tp is jointly trained with E,, D> on the same latent space.
We use a stochastic latent space rather than a deterministic one since the stochastic latent space of cVAE supports multiple future sampling and facilitates better prediction [60, 17]. We use trans- former architecture [61] for the encoder, decoder, and prior network for the cVAE in Tg-
3.2.2 Building the Downstream Anomaly Predictors
The visual world model captures changes in task outcomes over time. Its learned representation can be used for the anomaly predictors to predict future anomalies. Since individual tasks vary, we train spec-fc anomaly predictors on the frozen representation to predict failures and OOD anomalies.
Failure Prediction. We train failure clas- sifier Fr for each task using the frozen im- age embeddings from the visual world model. Failure labels are from trajectories т = (st, ae, Tt, q)) in the last round D' (see Sec- tion 3.1.2), where e marks human interven- tions (human). The trajectory segment be- fore each intervention labeled fail- ures (failure) [13, 17]. The classifier is trained using cross-entropy loss LF - -E y1 log(ū)) with balanced sampling, where y/s E {rollout, failure, human].
Embedding x
Out-of-distribution (1 - 3 of data th cluster
In-distribution
eg of data st this cluster
Adaptive Intervention Fitted Curve
100
Percentile (%)
The failure classifier is a small, computation- ally efficient model trained on frozen world model embeddings (training time 1.5 hour).
85
Original Data Fitted Exponential Decay
80
0.0
0.2
0.6 Autonomous [= Pa)
0.4
0.8
1.0
Out-of-Distrbbution (OOD) Prediction. We identify OOD states using k-means clustering. The frozen visual world model W generates embeddings from sampled trajectories the data buffer. We use Principal Component Anal- ysis (PCA) to reduce the embedding dimen- sions to l and calculate c k-means centroids for each task. To predict OOD for an embedding 2, we reduce its dimensions, find the nearest cen- troid, and calculate its L2 distance d. A state is identified as OOD if exceeds the task threshold og-- Og determined by dg. which is the distance of the top By percentile of the distances the nearest centroids from the validation latent embed- dings. For efficiency, we perform task-specific k-means clustering on image embeddings for the specific task without training additional models.
Figure 3: Adaptive Decision Boundaries. Top: OOD Prediction Boundary. The threshold f,y, de- termined by the human intervention ratio, sets the distance threshold ag. sample is identified as OOD if its embedding's distance d from the clus- ter centroid exceeds ag. Bottom: Fitting function for optimal 0g based on the human intervention ratio pH. The x-axis shows 1 PH. representing the autonomous rollout ratio.
Adattive Decision Boundaries. Anomaly predictors should adjust their anomaly prediction thresh- old as robot performance improves during deployment. multi-task setting, the varying task per- formances across different tasks pose additional challenges. We adjust anomaly predictors by loos- ening the decision boundaries for high-performing tasks and tightening them for low-performing tasks. For failure prediction, finetune it using the most recent round of deployment data D', with human intervention labels reflecting the updated human perceived risk.
5

---

The decision boundary expands for OOD prediction as the robot encounters more in-distribution states. We sample from D' the aggregated deployment data for all rounds, for k-means clustering. For each new embedding 2, we use threshold Og for its distance d to its nearest centroid, which depends on 0g. We update the prediction threshold Og based on the human intervention ratio pn. which acts as a proxy for policy performance. We fit the exponential decay function, fg = a + beePH and obtain the parameters by fitting the curve to set of calibration trajectories. Empirically, - 95.2, b -17.7 and and all tasks from simulation to real-world experiments, and we used the same function and set of hyperparameters throughout the simulation and real-world experiments.
-3.2. We found this function to be robust across all rounds
At deployment, the embedding observations, and the distance d from the nearest centroid is compared to Og. If d Oig. the ple is identified as OOD. Figure (Top) shows the decision boundary, while Figure 3 (Bottom) shows the fitted curve for fg-
computed from
Multi-Task Policy Finetuned Over Time
Robot Action
GMM
Transformer
Joint Gripper
FILM
FILM
3.2.3 SIRIUS-FLEHS in Operation
Embodding
Language
Resnet-18
Resnit-16
Continual Model Improvement. While the visual world model W is trained once and frozen, the pol- icy and the anomaly predictors continually fine- tuned over deployment: the policy is updated with the data from previous rounds using weighted sam- pling, the failure classifier for failure prediction is finetuned with the most recent intervention labels, and the K-means clustering in OOD prediction ex- pands its latent space coverage after each round.
"press stop to the microwave"
Workspace Left Camera
Workspaco Right Camera
Camera
Figure 4: Policy Architecture. The multi- task policy is Transformer that processes images, proprioceptive data, and task lan- guage embeddings. It uses Gaussian Mix- ture Model (GMM) to output robot actions.
Anomaly Predictors at Runtime. During deploy- ment, the visual world model predicts each task's 's future embeddings over L steps. We generate N possible future scenarios by sampling from the cVAE latent space and making L-step predictions N times. Each predicted future embedding is then assessed individually by the anomaly predictors. For failure prediction, we compute the average failure score across future embeddings. For OOD prediction, we calculate the average distance of each future state its nearest cluster centroid and compare the final average distance against the OOD threshold.
Multi- Musti-Task Policy Training. We train Transformer-base [61] multi-task policy, as shown in Figure 4. The policy inputs image observations, robot proprioceptive data, language task goals, and output robot actions. The design follows that in RoboMimic [32] and RoboCasa [39].
4 Experiments
In our experiments, we aim to address the following questions: 1) How well does SIRIUS-FLEET
continue to improve its policy during deployment with effective runtime monitoring? ? 2) How does SIRIUS-FLEET'S runtime monitoring performance compare with baseline methods? 3) Are the anomaly prediction made by SIRIUS-FLEET the right moments, qualitatively?
4.1 Evaluation Setup
To measure the effectiveness of STRIUS-FLEET'S multi-task fleet learning, we evaluate how both the policy and the runtime monitoring perform and evolve over time. Following prior works [62, 13, 17], we evaluate the system in a numan-in-the-looy setting through rounds iterative deployment.
Evaluation Setting. Our evaluation spans three rounds of policy updates and runtime monitoring. Full human supervision is required in the first round since anomaly predictors have been trained

---

Coffee Serve Mug
Coffee Set Up Mug
Olf Microwavve
Turn On Microweve
Turn On Sink Foucer
Tum Off Sink Foucet
Door Single Hinse
Close Door Single Hinge
PrP Counter
PriP Cab Counter
PrP Counter to Sink
Open Door Double Hinge
put the blue the basket
put the red howl the
pull out the the oven
the book the back compartment caday
blue
ovan tray
put the bread on the white plate
the pinnk mug om the White plate
put the bowl with hot dogs on the white plate
put the book in the back compartment of the caddy
put the red cup front compartment of the cacdy
Figure 5: RoboCasa Simulation Tasks and Mutex Real-World Tasks. We evaluate policy learning and runtime monitoring using 12 tasks from the RoboCasa benchmark in simulation and 10 tasks from the Mutex benchmark in real-world environments.
yet. Human supervision is only requested in the next two rounds when an anomaly is predicted. Af- ter each round, the collected deployment data and intervention labels are used to train the policy and anomaly predictors for the next round. The initial policy is trained with human demonstrations per task over 1200 epochs. We collect 100 rollouts per task for each round to ensure consistent data size and finetune the policy for 400 epochs.
Evaluation Metrics. We evaluate SIRIUS- -FLEET' policy using Autonomous Performance, which measures the policy's ability to achieve its goals without human intervention. After each round, the policy is finetuned on the newly collected data, and we evaluate its success rate without runtime monitoring to assess performance improvements over time. To evaluate runtime monitoring, we use Combined Policy Performance (CPP) and Return of Human Effort (ROHE). following [12, 17]. CPP measures the overall system success rate under monitoring, reflecting the effectiveness of human-robot collaboration in preventing anomalies. ROHE assesses the efficiency of human intervention by comparing the policy's 's success rate against the amount of human intervention effort, calculated as the ratio of policy performance to the number of interventions: Normalized ROHE E. ri] The goal is to maximize performance while minimizing human workload.
1+
Evaluation Environments. For simulation,
RoboCasa [39], visually diverse benchmark with various objects, layouts, and scenes. RoboCasa contains both a large dataset using MimicGen [38], an automated trajectory generation method (5k trajectories per task), and human demonstra- tion dataset (50 human demonstrations per task). The visual world model is trained on 20 task suites of MimieGen data, and the policy is trained on 12 task suites of human demonstration data. We use Mutex [27] for real-world experiments, commonly used benchmark for multi-task learning. The tasks across task suites. We use OSC controller with 7D action space (x-y-z position, yaw-pitch-roll orientation, and gripper).
visual model is trained on 50 tasks, and the policy is trained
4.2 Evaluation Results
System Performance Over Time. Figure 6 presents the performance of SIRIUS-FLEET over sev- eral deployment rounds. The autonomous policy improves consistently, showing a 13% increase in simulation and 45% in real-world settings. SIRIUS-FLER maintains high combined policy perfor- mance, exceeding 95% in both environments, demonstrating the effectiveness of runtime monitoring and timely human intervention. Additionally, CRRUS-FFLEE' shows improved ROHE performance

---

RoboCasa (Sim)
1.0
Mutex (Real)
1.0
(Sim)
1.0
Mutex (Real)
Aver Average Success Rate
0.8
Aver Average Success Pate
0.8
Normalized ROHE
Normalized ROHE
0.8
0.772
0.6
0.6
0.713
0.6
0.646
0. 0.093
0.6
0.675
0.4
0.4
0.4
0.300
0.4
5.272
Compined Polkcy
Comtinne folcy Autonemeus Polkyy
0.2
0.2
Autorimmuu
0.0
0.0
0.0
2 Rounds
3
Rounds
Rounds
Figure 6: System Performance Results. SIRIUS-FLEET shows consistent improvement throughout deployment for Combined and Autonomous Policy Performance and for Return of Human Effort (ROHE). The runtime monitoring ensures high combined policy performance and ROHE over time.
over time. The continuous fine-tuning of anomaly predictors and adaptive thresholds leads to fewer predicted anomalies and reduced human intervention.
Baseline Comparison. We compare SIRIUS- FLEET'S multi-task runtime monitoring against three baseline methods: MoMart [20], PATO
Combined Policy Performance
Return Human Effort
Success Rate (%)
160
1.0
80
[21], and ThriftyDAgger [12]. MoMart uses VAE reconstruction loss for OOD detection, PATO combines ensemble policy variance with VAE reconstruction for predicting future im- age goals, and ThriftyDAgger merges OOD detection and failure detection using risky Q- function values.
ROHE
0.6
0.4
0.2
MoMaRT
PATO
ThriByDApper
Sirie Fleet
MoMaRT
PATO
TaittyApper
Sinuss Fleet
Figure 7: Baseline Comparison. SIRIUS-FLEET surpassed the baselines in both combined policy performance and ROHE.
Since these baselines are designed for single- task environments, we evaluated them on five tasks from different categories in RoboCasa, training separate models for each task. All basclines used fixed detection thresholds, while ThriftyDAgger adjusts thresholds based target intervention ratio. For faimesss applied the same intervention ratio across all comparisons. As shown in Figure 7, STRIUS-FLEET consistently outperforms the baselines in policy performance and ROHE. The fixed thresholds used by the base- lines were ineffective in handling varying task distributions, resulting in excessive interventions or decreased performance across tasks.
A blations. We conducted ablation studies explore the following key questions: 1) How important is the design of the multi-task visual world model, opposed to the single-task design? 2) What are the individual impacts of the Failure Prediction and OOD Prediction components? 3) How critical is the design of the multi-task policy? The results of these studies are detailed in Appendix 6.2.
Conclusion
We introduce SIRIUS-FLEET. a framework for multi-task interactive robot fleet learning that com- bines multi-task policy with runtime monitoring. Driven by visual world model and adaptive anomaly predictors, SIRIUS-FLERH improves policy performance and reduces human intervention through timely anomaly prediction.
Limitations. SIRIUS-FLEET is best suited for quasi-static manipulation tasks, where anomalies can be easily corrected with teleoperation. However, applying SIRIUS-FLEET to dynamic tasks could be challenging. Also, our experiments involve a small group five human operators; future work could expand this by conducting large-scale human studies to understand the effects of diverse human interventions on runtime monitoring and policy training. Lastly, our evaluations used single embodiment (Franka Emika Panda arm). Future research can extend SIRIUS-FLEE" to cross- embodiment fleet learning, making it more generalizable across different robot platforms [36].

---

Acknowledgments
We thank Zhenyu Jiang, Rutav Shah, Yifeng Zhu, and Shuijing Liu for providing helpful feed- back on this manuscript. We also thank Xixi Hu, Zhenyu Jiang, Bo Liu, Yue Zhao, and Lizhang Chen for their fruitful discussions. We thank Soroush Nasiriany, Abhishek Joshi, and the Robo- Casa team for their support in the simulation environments and rendering. This work was partially supported by the National Science Foundation (FRR2145283) EFRI-23 18065), the Office of Naval Research (N00014-22-1-2204, N00014-2441-25550, and the DARPA TIAMAT program (HR0011- 24-9-0428).
References
[1] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manju- nath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, Singh, S. Sontakke,
Stone, Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong. F. Xia, Xiao, Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-1: Robotics transformer for real-world control scale, 2022.
[2] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg. T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess,
Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A generalist agent, 2022.
[3] D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. Vint: A foundation model for visual navigation, 2023.
[4] R. Hoque, L. Y. Chen, Sharma, K. Dharmarajan, Thananjeyan, P. Abbeel, and K. Gold- berg. Fleet-dagger: Interactive robot fleet learning with scalable human supervision. In Con- ference Robot Learning, pages 368-380. PMLR, 2023.
[5] L. Wang, K. Zhang, A. Zhou, M. Simchowitz, and R. Tedrake. Robot fleet learning via policy merging, 2024.
[6] M. Müller, S. Brahmbhatt, A. Deka, Q. Leboutet, D. Hafner, and V. Koltun. Openbot-fleet A system for collective learning with real robots, 2024.
[7] A. Mandlekar, D. Xu, R. Martín-Martín, Y. Zhu, L. Fei-Fei, and S. Savarese. Human-in-the- loop imitation learning using remote teleoperation. In arXiv preprint arXiv:2012.06733, 2020.
[8] E. Chisari, T. Welschehold, J. Boedecker, W. Burgard, and A. Valada. Correct me if i am wrong: Interactive learning for robotic manipulation. In RAL, volume 7, pages 3695-3702, 2021.
[9] Spencer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang, P. Ramadge, and Srinivasa. Learning from interventions: Human-robot interaction as both explicit and implicit feedback. In 16th Robotics: Science and Systems, RSS 2020. MIT Press Journals, 2020.
[10] M. Kelly, C. Sidrane, K. tive imitation learning with human experts. 20/9 International Conference on Robotics and Automation (ICRA), May 2019, org/13 10 1109/ 1109/ICRA. ICRA, 19.999999.....
and M. J. Kochenderfer. Hg-dagger: Interac-
URL http: / //xxx doi.
[11] R. Hoque, A. Balakrishna, C. Putterman, M. Luo, D. S. Brown, D. Seita, B. Thananjeyan, E. Novoseller, and K. Goldberg. Lazydagger: Reducing context switching in interactive im- itation learning. In 2021 IEEE 17th International Conference Automation Science and Engineering (CASE), pages 502-509. IEEE, 2021.

---

[12] R. Hoque, A. Balakrishna, E. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. arXiv preprint arXiv:2/09.08273, 2021.
[13] H. Liu, S. Nasiriany, L. Zhang, Z. Bao, and Y. Zhu. Robot learning on the job: Human-in- the-loop autonomy and learning during deployment. In Robotics: Science and Systems (RSS), 2023.
[14] R. Hoque, L. Y. Chen, S. Sharma, K. Dharmarajan, B. Thananjeyan, P. Abbeel, and K. Gold- berg. Fleet-dagger: Interactive robot fleet learning with scalable human supervision, 2022.
[15] Q. Li, Z. Peng, and B. Zhou. Efficient learning of safe driving policy via human-ai copilot optimization, 2022.
[16] E. Yel and N. Bezzo. Fast uu-timme monitoring, replanning, and recovery for safe autonomous system operations. In 2019 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1661-1667, 2019. 10i:10..100/1RR0O 10001100//0004097.20000 98688998.
[17] H. Liu, S. Dass, R. Martin-Martin, and Y. Zhu. Model-based runtime monitoring with interac- tive imitation learning. In IEEE International Conference on Robotics and Automation (ICRA), 2024.
[18] M. Salehi, H. Mirzaei, D. Hendrycks, Y. Li, M. H. Rohban, and M. Sabokrou. A unified survey on anomaly, novelty, open-set, and -utttf-disttibbtion detection: Solutions and future chal- lenges. CoRR, abs/2110. 14051, 2021. URL https:/ //aaaiivvvooo. org. abs 22100210100 11110.1455..
[19] C. Richter and N. Roy. Safe visual navigation via deep learning and novelty detection. Robotics: Science and Systems, 07 2017.
[20] Wong, A. Tung, A. Kurenkov, A. Mandlekar, Fei-Fei, S. Savarese, and R. Martin-Martin Error-aware imitation learning from teleoperation data for mobile manipulation. In Proceed- ings of the 5th Conference Robot Learning, volume 164 of Proceedings of Machine Learn- ing Research, pages 1367-1378, 2022.
[21] S. Dass, K. Pertsch, H. Zhang, Y. Lee, J. J. Lim, and S. Nikolaidis. Pato: Policy assisted teleoperation for scalable robot data collection, 2022.
[22] T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, C. Ng. R. Wang, and A. Ramesh. Video generation models world simulators. 2024.
[23] Bruce, M. Dennis, A. Edwards, Parker-Holder, Y. Shi, E. Hughes, M. Lai. A. Mavalankar, R. Steigerwald, C. Apps, Y. Aytar, S. Bechtle, F. Behbahani, S. Chan, N. Heess, L. Gonzalez,
S. Osindero, S. Ozair, S. Reed, J. Zhang, K. Zolna, J. Clune, N. de Freitas, S. Singh, and
T. Rocktäschel. Genie: Generative interactive environments, 2024.
[24] J. Xiang, G. Liu, Y. Gu, Q. Gao, Y. Ning, Y. Zha, Z. Feng, T. Tao, S. Hao, Y. Shi, Z. Liu, E. P. Xing, and Z. Hu. Pandora: Towards general world model with natural language actions and video states. 2024.
[25] A. Hu, I.. Russell, H. Yeo, 7. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. Corrado. Gaia-l: A generative world model for autonomous driving, 2023.
[26] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. In Fortieth Interna- lional Conference Machine Learning, 2023.
[27] R. Shah, R. Martin-Martin, and Y. Zhu. Mutex: Learning unified policies from multimodal task specifications. In 7th Annual Conference Robot Learning, 2023. URL https:// openreview net Eorum? cd-pwqiqgaa..
10

---

[28] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess,
A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee, Levine, Lu, H. Michalewski, Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. R1-2: Vision-language-action models transfer web knowledge robotic control, 2023.
[29] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning, 2022.
[30] K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauza, T. Davehev, Y. Zhou, A. Gupta, A. Raju, A. Laurens, C. Fantacci, Dalibard, M. Zambelli, M. Martins, R. Pevce- viciute, M. Blokzijl, M. Denil, N. Batchelor, T. Lampe, E. Parisotto, K. Zolna, S. Reed, S. G. Colmenarejo, J. Scholz, A. Abdolmaleki, O. Groth, J.-B. Regli, O. Sushkov, T. Rothörl, J. E. Chen, Y. Aytar, D. Barker, J. Ortiz, M. Riedmiller, J. T. Springenberg, R. Hadsell, F. Nori, and N. Heess. Robocat: A self-improving generalist agent for robotic manipulation, 2023.
[31] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, J. Luo, Y. L. Tan, L. Y. Chen, P. Sanketi, Q Vuong, T. Xiao, D. Sadigh,
C. Finn, and S. Levine. Octo: An open source generalist robot policy, 2024.
[32] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkamni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martin What matters in learning from offline human demonstrations for robot manipulation, 2021.
[33] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models, 2020.
[34] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion, 2024.
[35] F. Ebert, Y. Yang. K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and
S. Levine. Bridge data: Boosting generalization robotic skills with cross-domain datasets, 2021.
[36] Open X-Embodiment Collaboration et al. Open X-Embodiment: Robotic learning datasets and RT-X models. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan, 2024.
[37] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, et al. Droid: A large-scale in-the-wild robot manipulation dataset, 2024.
[38] A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In 7th Annual Conference on Robot Learning, 2023.
[39] S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS), 2024.
[40] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine. QT-Opt: Scalable deep reinforcement learning for vision- based robotic manipulation. In CoRL, 2018.
[41] A. Team, J. Aldaco, T. Armstrong, R. Baruch, Bingham, S. Chan, K. Draper, D. Dwibedi, C. Finn, P. Florence, S. Goodrich, W. Gramlich, T. Hage, A. Herzog, J. Hoech, T. Nguyen, I. Storz, B. Tabanpour, L. Takayama, J. Tompson, A. Wahid, T. Wahrburg, S. Xu,
11

---

S. Yaroshenko, K. Zakka, and T. Z. Zhao. Aloha 2: An enhanced low-cost hardware for bimanual teleoperation, 2024.
[42] M. Ahn, Dwibedi, C. Finn, M. G. Arenas, K. Gopalakrishnan, K. Hausman, Ichter, A. Irpan, N. Joshi, R. Julian, S. Kirmani, I. Leal, Lee, S. Levine, Y. Lu, Leal, Maddineni, K. Rao, D. Sadigh, Sanketi, P. Sermanet, Q. Vuong. S. Welker, Xia, T. Xiao, P. Xu, S. Xu, and Z. Xu. Autort: Embodied foundation models for large scale orchestration of robotic agents, 2024.
[43] G. Swamy, S. Reddy, S. Levine, and A. D. Dragan. Scaled autonomy: Enabling human opera- tors to control robot fleets, 2020.
[44] O. Akcin, P.-h. Li, S. Agarwal, and P. Chinchali. Decentralized data collection for robotic fleet learning: A game-theoretin approach. In K. Liu, D. Kulic, and J. Ichnowski, edi- tors, Proceedings of The 6th Conference on Robot Learning. volume 205 of Proceedings of Machine Learning Research, pages 978-988. PMLR, 14-18 Dec 2023. URL https: proceedi aggse. ngs.
m1 m.r... wwwppsssssee....... /vnesss/220 2205 aal//akiinne n2.a.. html.
[45] O. Akein, O. Unuvar, O. Ure, and S. P. Chinchali. Fleet active learning: submodular max- imization approach. In J. Tan, M. Toussaint, and K. Darvish, editors, Proceedings of The 7th Conference Robot Learning, volume 229 of Proceedings of Machine Learning Re- seurch, pages 1378-1399, PMLR, 06-09 Nov 2023. URL httptt::: httttps://proccedinggg mlr. press/v2291 2ttsss//22/aakc222aaaatas html.
[46] K.-C. Hsu, H. Hu, and J. F. Fisac. The safety filter: A unified view of safety-critical con- trol in autonomous systems. 2023. URL
http:/ppp..smmatccccchhooooooo/o orgg
CozpusID:26169777
[47] R. Sinha, E. Schmerling, and M. Pavone. Closing the loop on runtime monitors with fallback- safe mpc. In Proc. IEEE Conf. on Decision and Control, 2023.
[48] K. Nakamura, R. Tian, and A. Bajesy. Not all errors are made equal: A regret metrie for detecting system-level trajectory prediction failures. In 8th Conference on Robot Learning, 2024.
[49] R. Sinha, A. Sharma, S. Banerjee, T. Lew, R. Luo, S. M. Richards, Y. Sun, E. Schmerling, and M. Pavone. A system-level view on out-of-distribution data in robotics, 2022.
[50] K. Menda, K. Driggs-Campbell, and M. J. Kochenderfer. Ensemblelagger: A bayesian ap- proach to safe imitation learning. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5041-5048. IEEE, 2019.
[51] A. Diryag, M. Mitić, and Z. Miljković. Neural networks for prediction of robot failures. Pro- ceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineer- ing Science, 228(8):1444-1458, 2014.
[52] C. Gokmen, D. Ho, and M. Khansari. Asking for help: Failure prediction behavioral cloning through value approximation. ArXiv, abs/2302.04334, 2023.
[53] A. Xie, F. Tajwar, A. Sharma, and C. Finn. When to ask for help: Proactive interventions in autonomous reinforcement learning, 2022.
[54] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, Varley, Z. Xu, D. Sadigh, A. Zeng, and A. Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners, 2023.
[55] Z. Liu, A. Bahety, and S. Song. Reflect: Summarizing robot experiences for failure explanation and correction, 2023.
12

---

[56] L. Guan, Y. Zhou, D. Liu, Y. Zha, H. B. Amor, and S. Kambhampati. "task success" is not enough: Investigating the use video-language models as behavior critics for catching unde- sirable agent behaviors, 2024.
[57] A. Hu, G. Corrado, N. Griffiths, Z. Murez, C. Gurau, H. Yeo, A. Kendall, R. Cipolla, and
J. Shotton. Model-based imitation learning for urban driving, 2022.
[58] O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation, 2015.
[59] K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. In C. Cortes, N. Lawrence, Lee, M. Sugiyama, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.,
2015. URL https::/pptcceeddnnnnn wew.reirdii.nnnnnen..e.........e wwwweips...pppppppeeeeeeee wwtppcppppppplceeeeeee fi wfffflelfffffe wwt.sseeepppee/11./5..1. file/ http://55222eaaaac67777222222222
[60] Wu, A. Escontrela, D. Hafner, Abbeel, and K. Goldberg. Daydreamer: World models for physical robot learning. In Conference on Robot Learning, pages 2226-2240. PMLR, 2023.
[61] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need, 2023.
[62] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In CoRL, pages 879-8999 99389,9390 2018.
13

---

Appendix
6.1 Table of Contents
Ablations (Section 6.2)
Implementation Details (Section 6.3)
Qualitative Analysis and Discussion (Section 6.4)
Additional Details on Tasks (Section 6.5)
6.2 Ablations
Muski-Task World Model Single-Task World Model. We empirically show how the multi-task world model is more generalizable than the single-task version in terms of future latent state pre- diction accuracy. Accurate future latent state prediction is essential for effective error prediction, as the downstream error predictors directly use the predicted future states. We compared the perfor- mance of HIRIUS-FLEETS multi-task world model against six single-task world models in terms of mean squared error (MSH) for future latent state prediction. As shown in Table 1, the multi-task world model consistently outperforms the single-task models across all tasks. This indicates that the multi-task world model crucial for providing more accurate state predictions.
Ablation of Failure and OOD Prediction. We examine the impact of OOD and failure prediction components on human intervention overlap accuracy: We have human operator fully supervise the robot execution of a policy and can intervene whenever an unsafe state is observed. We then apply the learned error predictors to the collected trajectories and compare failure states identified by the error predictors with those intervened by the human operator. This shows how the error predicted aligns with humans' risk assessment.
As shown in Table 3, SIRIUS-FLEETS combined use of OOD and failure prediction significantly outperforms using either component alone across all tasks. This suggests that both accurate OOD prediction and failure prediction are critical to the overall performance of the policy, as they com- plement each other in identifying potential errors during deployment.
Multi-task Policy vs. Single-task Policy. We show that training multi-task settings improves the overall performance of robot policy than training on individual single tasks. We show the policy success rate (%) for 7 different tasks in Figure 8, where we compare the multi-task policy trained on all tasks with single-task policies trained on individual tasks. For most tasks, SIRIUS-FLEET'S multi-task policy performs better than the single-task policy trained on that specific task. This aligns with the observation in prior works [31, 36. 2] that multi-task training helps the policy generalize better.
Task
Single Task
Multi Task
CloseDoorS wngesiiilleeee nlleliine inge
3.3 10-1
1.5 x 10-1
PnPCabToCounteer
3.1 x 10-4
1.7 x 10-4
TurnOnMicrowave
1.7 10
1.3 10-4
TurnOnSinkFaucet CoffeeSetupMug
3.2 10-4
1.2 x 10-4
4.4 x 10-4
1.8 x 10-4
Table 1: Multi-task World Model vs. Single-task World Model Comparison. We show the
multi-task world model is compared with 6 different world models trained on 6 single tasks. The multi-task world model consistently gives more accurate future latent prediction than the world models trained with single- task data.
MSE loss (L) for future latent states prediction, where SIRIUS-FLEET
14

---

Task
STRIUS-FLEET (OOD)
STRIUS-FLEET (Failure)
SIRIUS-FLEFT
CoffeeSetupMig
85.1
87.0
99.4
PnPCounterToCab
43.1
56.2
73.7
TurnOffMicrowave
54.9
50.0
74.5
TurnoffSinkFaucet.
11.3
38.8
45.0
OpentoorDoublettinge
42.8
63.3
78.4
Table 2: Human Intervention Overlap Accuracy (%) for Different Tasks. STRIUS-FIERR (using
OOD) + Failure combined) achieves better performance than using OOD prediction only or Failure prediction only.
Multi-Task Single-Task Policy
70
Success Rate (%)
60
Single-Task
50
Multi-Task
30
20
10
PnPCabToCounter
PoPCounterToCab
CoffeeSetupMug
PnPCounterToSink
CloseDoofSingleHlinge
TumonSinkfaucet
TumonMiccroave
Average
Tasks
Figure 8: Multi-task Policy Single-task Policy Comparison. We show the policy success rate (%) for different tasks, comparing the multi-task policy trained on all tasks with single-task policies trained on individual tasks. SIRIUS-FLEET 's multi-task policy gives overall better performance. Note that for Co CoffeeSttupMng and PrPCounterToSink, the single-task policy gives 0% success rate.
15

---

Table 3: Hyperparameters: World Model
Group
hyperparameter
Value
UNet Encoder
image output activation
Sigmoid [workspace, wrist]
image observations
observation fusion method
concat
input channels
3
output channels latent channels
3
4
block output channels
[32, 64]
layers per block activation function
SiLU
normalization number groups
32
Dynamics Model
history length
10
number of futures sampled
20
GMM Prior
latent dimension learnable number of GMM nodes
16
True
Transformer Architecture for VAE Encoder, Decoder, Prior
context length embed dimension number of layers number of heads embedding dropout attention dropout block output dropout sinusoidal embedding activation causal
20
512
6
8
0.1
0.1
0.1
False
GeLUU
False
Implementation Details
We present the details of hyperparameters for training the world model, policy, and error predictors in the following table:
World Model Architecture: Table 3;
Policy Architecture: Table 4;
Training: Table 5:
Failure Prediction: Table 6;
OOD Prediction: Table 7.
16

---

Table 4: Hyperparameters: Policy
Group
Hy Hyperparameter
Value
Transformer Policy
context length embed dimension mumber of layers number of heads embedding dropout attention dropout block output dropout sinusoidal embedding activation causal
20
512
6
0.1
0.1
0.1
False
GeLU
False
GMM Head
number of modes min std std activation low noise eval
5
0.005
Softplus
True
Image Encoder
feature dimension backbone class
64
backbone pretrained
False
backbone input coord conv pool class pool number of keypoints pool learnable temperature pool temperature pool noise std
False
SpatialSoftmax
32
False
1.0
0.0
Image Augmentation
class crop height crop width number of crops positional encoding
CropRandomizer
116
116
False
Table 5: Hyperparameters: Training
Group
Hyperparameter
Value
World Model
optimizer type
Adam
initial learning rate
0.0001
learning rate decay factor
0.1
epoch schedule
[]
scheduler type
Constant
Policy
optimizer type
AdamW
initial learning rate
0.0001
learning rate decay factor epoch schedule
1.0
[100]
scheduler type
Constant with warmup
17

---

Table 6: Hyperparameters: Failure Prediction
Group
Hyperparameter
Value
Transformer Architecture
context length embed dimension
10
512
number of layers number of heads embedding dropout attention dropout block output dropout
8
0.1
0.1
0.1
sinusoidal embedding activation causal
False
GeLU
False
Failure Predictor
predict on future threshold count evaluation index use probability
Truc
[7,8,9]
False
Table 7: Hyperparameters: OOD Prediction
Group
Hyperparameter
Value
OOD Predictor
number of future steps
20
predict on future distance metric train k-means percentile
True
k-means
False
see Section 3.2.2
18

---

6.4 Qualitative Analysis and Discussion
Can SIRICS-FLER catch important errors and give meaningful intervention timings? We conduct a qualitative analysis where the error predictors predict error during robot execution using active human monitoring- Specifically, we have human operator who fully supervises the robot's pol- icy execution and intervene whenever an unsafe state is observed. We then apply the learned error predictors to the collected trajectories and compare failure states identified by the error predic- tors with those intervened by the human operator. This shows how the error predicted aligns with humans' risk assessment.
We show examples of the human intervention region and errors predicted by STRIUS-FLEET and the baselines Figure 9 14. We use the same trajectory for all methods. The three instances 1) the robot is not aiming at the object it is grasping; 2) the robot
humans gave interventions pauses at grasping; 3) the arm is too stiff and does not bend down. We visualize if the emrors given by STRIUS-FI.E and the baselines are aligned with the intervention humans gave.
The green area indicates times of actual human intervention; the blue plot indicates the error value predicted by the different methods. We also show the video frame at the corresponding timings to illustrate the failure modes captured (or missed) by the different methods.
Note that FLEIT, PATO, and ThriftyDAgger each have two separate components, so we visualize each individually. PATO and ThriftyDAgger share the same ensemble uncertainty compo- nent, so we visualize one of them.
As shown in Figure 9 and 10, our method capture similar error pattern to that the human judges. MoMaRT (Figure 11) is prone predicting false positives due to pixel changes. PATO (VAE part, Figure 12) often predicts false negatives; PATO (ensemble part, Figure 13) often has high oscillations and extreme values. ThriftyDAgger's (Figure 14) risk Q function is data-intensive to train and often has generalization errors; it cannot accurately reflect the task progress and local failure modes.
19

---

Unnecessary: the object
Necessary: paused grasping
Pn.P Counter To-Cab
1.0
Failure Probability
0.8
B
0.6
0.4
0.2
Predicted Fallure Probability
Human Intervention
0.0
50
100
150
200
Timestep
Necessary: not aiming at the object
Necessary: arm too stiff; does not bend down
Figure 9: SIRIUS-FLETT: Failure Only. The y-axis shows the predicted failure probability from the failure classifier, where higher value indicates a greater likelihood of failure.
Necessary: not aiming at the object
Necessary: arm too stiff; does not bend down
Counter Cab
0.04
0.03
Distance
0.02
0.01
Predicted Distance to Centroid
Distance Threshold
Human Intervention
0.00
50
100
Timestep
Unnecessary: object already grasped
Figure 10: SIRIUS-FLEET: OOD Only. The y-axis represents the predicted distance to the cen- troid. It is predicted OOD if this distance exceeds threshold (indicated by the yellow dotted line).
20

---

Unnecessary: just picked up the object
Unnecessary; normal movement to the cabinet
PnPCounter ToCab
VAE Reconstruction Error
0.07
B
0.06
0.05
0.04
0.03
www
0.02
VAE-Score
VAE AaE-Threshald
0.01
Human Intervention
0.00
150
200
Timesteps
Unnecessary: normal movement to the object
Figure 11: MoMaRT. The y-axis represents the VAE reconstruction error (MSE loss) the current image observation. It is predicted as an error if this loss exceeds threshold (indicated by the yellow dotted line).
PnPCounterToCab
VAE-Score
0.020
VAE-Threshold Human Intervention
0.015
0.010
0.000
0
50
100
150
200
Timesteps
Unnecessary: normal movement the cabinet
Figure 12: PATO: VAE Only. The y-axis represents the variance the VAE reconstruction of the future image. It is predicted error if this value exceeds threshold (indicated by the yellow dotted line).
21

---

Unnecessary: normal movement to the object
Unnecessary: normal movement to the cabinet
PnPCounter
0.14
Ensemble Variance
0.12
Ensemble-Scae Score
0.08
Ensembie-Threshold
0.06
Human Intervention
0.04
0.02
Mwn MW
0.00
MAMa
50
100
150
200
Timesteps
Necessary: not aiming at the object
Necessary: arm too stiff; does not bend down
Figure 13: PATO: Ensemble Only. The y-axis represents the variance of the ensemble policies. It predicted as an error this value exceeds a threshold (indicated by the yellow dotted line).
PnPCounterIBCab
PnPCounterloCab
0.043
Q-Iniv
m
T-Threshoid
Intervention
Q-Value
Q-Value
0.041
Wy
0.040
0.4
0.03
0.2
0-Threshold
Human Intervention
033
100
100
150
Timesteps
Timesteps
Figure 14: ThriftyDAgger: Risk Q-Function. The y-axis represents the q-value of the current state-action pair, where a higher value means less risk failure. The video frame visualization is omitted here since clear error modes have been predicted.
22

---

6.5 Additional Details on Tasks
6.5.1 Real robot experiments
Dataset. We use the Mutex [27] dataset for real robot experiments. The Mutex dataset is diverse multi-task dataset containing 50 different tasks from 8 task suites (where one task suite is distinct set of objects and receptacles), specified with natural language. More details on the Mutex dataset can be found the original Mutex [27] paper. We use the original 50 tasks for training the visual world model; for human workload consideration, we sample 10 of the tasks (among 5 task suites) for multi-task policy learning and runtime monitoring deployment. The list of 10 tasks for policy learning and runtime monitoring experiments is shown in Figure 15.
put the blue cup in the basket
put the red bowl the basket
pull out the tray of the oven and put the bowl with hot dogs the tray
the book in the back
compartment of the caddy
put the blue mug on the oven tray
put the bread on the white plate
the pink mug the white plate
put the bowl with hot dogs on the white plate
put the book in the back compartment the caddy
put cup the front compartment the caddy
Figure 15: Mutex Real World Tasks, We use 10 tasks covering 5 task suites in the Mutex dataset for policy learning and runtime monitoring.
Training, To train the visual world model, we use 30 demonstrations for each of the 50 tasks. To train the multi-task policy, 30 demonstrations per task are used to bootstrap the initial BC policy; then, for each of the human-in-the-logy deployment round for runtime monitoring (see Evaluation Setting Section 4.1), robot rollouts per task are collected for each round.
For the initial BC policy, we train the BC transformer policy for 2000 epochs. For each subsequent round, we finetune the initial BC policy on the newly aggregated data for another 800 epochs. We use this finetuned BC policy checkpoint to perform runtime monitoring experiments.
Evaluation. We use 1 seed and the checkpoint at a fixed epoch (epoch 2000 for the initial policy, epoch 800 for the finetuned per-round policy) to evaluate the real robot policy. We evaluate all 10 tasks, conduct 20 trials per task for 200 trials, and report the average success rate across tasks. We report the per-task autonomous policy performance in Figure 17 and per-task combined policy performance in Figure 18 for reference.
6.5.2 Simulation
Dataset. We use RoboCasa [39] as our simulation environments, which contains a diverse range of tasks We note one important difference in the environment definition in RoboCasa: the environments defined by group of tasks that has similar task semantics, rather than by single task of
23

---

fixed set of scenes and objects. For example, The Co CoffeeServeMug environment can contain combination of diverse backgrounds, layouts, coffee machine types, and coffee mug types, making it a challenging environment for generalization. As noted Section 4.1, We use all 20 suites to train the visual world model and 12 of them for policy learning and runtime monitoring. The list of 10 tasks for policy learning and runtime monitoring experiments shown Figure 16.
CofferServeMug
Mug
TurnOriMierowswe
TurnOnMisroweve
TumnnnSinkFuoett
TurnOfiSinkFaucat
CloseDoorSinglehlngge
PnPCounterTeCab
PnPCebioCounter
PnPCounterToSink
OpenDoorDoouleetinge
RoboCasa Simulation Tasks. We use 12 tasks in the RoboCasa benchmark for policy learning and runtime monitoring.
Figure 16:
50 demonstrations for each of the 20 tasks suites. To train the multi-task policy, 50 demonstrations per task suite are used to bootstrap the initial BC policy; then, for each of the human-in-the-logg deployment round for runtime monitoring (see Evaluation Setting in Section 4. 4.1), 100 robot rollouts per task are collected for each round.
Training. To train the visual world model,
For the initial BC policy, train the BC transformer policy for 1000 epochs. For each subsequent round, finetune the initial BC policy the newly aggregated data for another 400 epochs. We use this finetuned BC policy checkpoint to perform runtime monitoring experiments.
Evaluation. We use 2 seeds and checkpoints at fixed epoch (epoch 1000 for the initial policy, epoch 400 for the finetuned per-round policy) to evaluate the simulation policy. We evaluate all 12 tasks, conduct 50 trials per task for a total of 600 trials, and report the average success rate across tasks. We report the par-task autonomous policy performance in Figure 19 and per-task combined policy performance in Figure 20 for reference.
24

---

Mutex: Autonomous Policy Performance Per Task
Round
0.8
Round
0.7
Round
0.6
Scores
0.5
0.4
0.0
pull out the tray the oven and put the bowl with hot dogs the tray
put the blue cup in the basket
put the red bowl the basket
put the book in the back compartment the caddy
put the blue mug the oven tray
put the bowi with hot dogs on the white plate
put the bread the white plate
But the red cup in the front compartment the caddy
put the pink mug on the white plate
put book the back compartment of the caddy
Average
Tasks
Figure 17: Mutex: Autonomous Policy Performance Per Task. The autonomous policy success
rate improves over three rounds of deployment for the Mutex tasks.
25

---

Mutex: Combined Policy Performance Per Task
1.0
Round Round
0.9
0.8
Scores
0.7
0.6
0.5
pull out the tray the oven and put the bawl with hot dogs on the tray
put the blue cup in the basket
put the red bowl the backet
put the book the back compartment of the caddy
put the bread on the white plate
the bowl with hot degs on the white plate
the red cup in the front compartment of caddy
put the pink mug on the white plate
nut the book in the back compartment of the caddy
Average
put the blue mug the oven
Tasks
Figure 18: Mutex: Combined Policy Performance Per Task. The combined policy success rate improves over three rounds of deployment for the Mutex tasks.
26

---

Robocasa: Autonomous Policy Performance Per Task
Round
Round Round
0.8
0.6
0.2
CloseDoorSingleHlnng
0.0
OpenD Dabbbegrtau
openDnorSingleHinge
TumonsinkFaucet
TumonMlcrowave
Funorttin Faucet
TumorMicrowave
PnPCounter Caab
oounter
PrPCountertosint fasik
CoffeeSetupMug
CoffaeServeMug
Average
Figure 19: RoboCasa: Autonomous Policy Performance Per Task. The autonomous policy suc- cess rate improves over three rounds of deployment for most of the RoboCasa tasks.
27

---

Robocasa: Combined Policy Performance Per Task
Round
Round
0.9
0.8
Scores
0.6
PhPCounterfecab
0.5
CoffeeSetupMug
PrPCounter/osinn
PnPCab Toconter
CoffeeServeMug
OpenDearDrubleHingg Huiieeiige
TumomHlcrowave
TunorisinkFarcet
tumonMicrowave
OnenDoorSingletingg
Fauceet Faucee
SingeannSnngletiing
Average
Tasks
Figure 20: RoboCasa: Combined Policy Performance Per Task. The combined policy success rate improves over three rounds of deployment for most of the RoboCasa tasks.
28


Reviews:
[{"review_id": "AaZfwEwJQY", "reviewer": "Reviewer_sGwF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a human-in-the-loop interactive, multi-task, continuously improving learning system for a fleet. There are two main components. First, a visual World Model is used to predict future visual states. The latents of this World Model predictions are used to predict whether a particular robot execution may be failing or out-of-distribution. Based on this the human supervisor is prompted to attend and intervene if necessary. The data generated is used to train an imitation learning policy for the next round (i.e., continuously improving).", "summary_of_recommendation": "Use of learned World Model to predict failure and OOD is a promising idea. Shows good performance with respect to compared methods. Lack of clarity in terms of novelty. Lack of clarity in technical details. Ablations needed in experiments.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "I have listed my questions above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oWCT2o24kZ", "reviewer": "Reviewer_9gCY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes a framework for multi-task interactive training of robots. The goal is to automate the failure / OOD detection to seek human help. The framework consists of two parts: (1) pre-training the multi-task world model, and (2) enabling continual policy training via detecting failures and asking for human intervention. The work introduces an automatic way of detecting failures (based on a classifier) and OOD (based on k-means). The experiments are shown on a wide variety of tasks: 12 tasks in simulation, and 10 tasks on real-world Panda robot.", "summary_of_recommendation": "As it stands, there are several experiments that I've asked for and have some concerns on the core claims and implementation details. Based on this I am currently voting for a weak reject of the paper, however I will give the final recommendation after look at other reviews, authors' rebuttal and the discussion during rebuttal.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "I would like the authors to address the concerns raised in the **Weaknesses** section and perform the experiments asked there (comments 3, 4, 5). In addition to that here are 2 questions on the design choice of the model and baselines that I have.\n\n9. The authors mention that the world model is frozen. Why was this a design choice? This inherently assumes that the trajectories on which the world model is trained are the complete set of optimal trajectories, however, that need not be true. Also, when there is a deviation in the trajectory and the human intervenes and describes the right trajectory, the dynamics model needs to be aware of this deviated-but-successful trajectory. Am I missing something here? \n\n10. For the baselines is the same architecture of the world model used? (Authors do mention that the policy architecture is the same.)", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6CjBqrWfGo", "reviewer": "Reviewer_eiin", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Firework is a framework for multi-task interactive robot fleet learning with a human in the loop and runtime monitoring. The approach uses a visual world model to predict task outcomes and is leveraged as an error predictor.  This approach is compared to MoMaRT, PATO & ThriftyDAgger, and shows improvements in terms of Combined Policy Performance and Return of Human Effort (ROHE).", "summary_of_recommendation": "This work has shown to be SOTA for policy learning with human interventions for an interactive fleet. The work could be made stronger by more clearly showing when this approach requires human interventions compared to baselines.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Figure 6 is not clear as written and is a missed opportunity to highlight how your method compares to the baselines.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework for multi-task interactive robot fleet learning. The reviewers find the idea and the results promising. However, the reviewers have raised several concerns including a lack of clarity on various aspects in the approach, questions about the novelty with respect to the state of the art, lack of clarity in the evaluations and experiment settings, lack of ablations, details on the implementation are missing, the writing has to be significantly improved, unclear when the proposed approach requires human intervention, among others. Each of the reviewers has listed several questions to be addressed in the rebuttal.
Post-rebuttal: Most of the reviewers' concerns have been sufficiently addressed. Please address the remaining suggestions in the final version.
Decision: Accept