Fleet-DAgger: Interactive Robot Fleet Learning with Scalable Human Supervision
Authors: Ryan Hoque
Lawrence Yunliang Chen
Satvik Sharma
Karthik Dharmarajan
Brijen Thananjeyan
Pieter Abbeel
Ken Goldberg
Venue: CoRL 2022 Oral (2022)
Keywords: Fleet Learning
Interactive Learning
Human Robot Interaction

Abstract: Commercial and industrial deployments of robot fleets at Amazon, Nimble, Plus One, Waymo, and Zoox query remote human teleoperators when robots are at risk or unable to make task progress. With continual learning, interventions from the remote pool of humans can also be used to improve the robot fleet control policy over time. A central question is how to effectively allocate limited human attention. Prior work addresses this in the single-robot, single-human setting; we formalize the Interactive Fleet Learning (IFL) setting, in which multiple robots interactively query and learn from multiple human supervisors. We propose Return on Human Effort (ROHE) as a new metric and Fleet-DAgger, a family of IFL algorithms. We present an open-source IFL benchmark suite of GPU-accelerated Isaac Gym environments for standardized evaluation and development of IFL algorithms. We compare a novel Fleet-DAgger algorithm to 4 baselines with 100 robots in simulation. We also perform a physical block-pushing experiment with 4 ABB YuMi robot arms and 2 remote humans. Experiments suggest that the allocation of humans to robots significantly affects the performance of the fleet, and that the novel Fleet-DAgger algorithm can achieve up to 8.8x higher ROHE than baselines. See https://tinyurl.com/fleet-dagger for supplemental material.


Reviews:
[{"review_id": "yWr_ZLkkeJH", "reviewer": "Reviewer_tye5", "summary": "Fleet-DAgger addresses the following question. How can a fleet of independent robots, each operating the same policy, most efficiently learn a task, given the availability of a small number of humans who can temporarily take over control from the policy and perform interventions? The authors call this Interactive Fleet Learning (IFL).\n\nThe main contributions of the paper are as follows:\n* A framework for doing IFL. The framework’s simulation code and results are open-sourced\n* An allocation algorithm (CUR) for allocating supervision (human or otherwise) to robots\n* A clear set of metrics by which to compare IFL algorithms, including return on human effort (ROHE)\n* A detailed comparison of CUR with relevant baselines and an extensive set of ablations which shed light on the contributions of different components of the algorithm.\n* An instantiation of IFL on a physical system consisting of 4 robots + 2 humans.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* This work is both novel and relevant. Given that behavior cloning (BC), has been demonstrated to be an effective method for robot learning (e.g. What Matters in Learning from Offline Human Demonstrations for Robot Manipulation, Mandlekar et al, 2021) and the benefits of scaling up learning have been clearly demonstrated in other areas of ML, I can see the importance of research into effective and scalable fleet learning growing. Further I have not seen much work in fleet learning to date and so I think this work could be an important step in that direction. Finally, since human time is costly (e.g Back to Reality for Imitation Learning, Johns, 2021 (CoRL blue sky)), I think this work makes a valuable contribution by proposing a method which attempts to maximize return on human effort.\n* The paper is well written throughout and the motivation is especially clear.\n* The literature review appears thorough (although I am not an expert on fleet learning)\n* The selection of baselines (4 different baselines) is thorough and relevant.\n* I especially appreciate the detailed set of ablations in the supplementary materials which shed light on the different components of the CUR algorithm. The terminology (U, UR, CUR etc) is also really clear.\n   * I appreciate that there are space constraints but it would be great to see a brief summary of the main conclusions from these ablations in the main paper.\n* The experiment selection is thorough covering 4 tasks (3 in simulation and 1 on physical robots)\n* The framework appears quite general and also reproducible. The authors open source both the code and results for simulation.\n* Results are demonstrated on a fleet of physical robots (4 robots + 2 humans)\n* The benefit of CUR over baselines in simulation appear to be substantial. However I have a few questions detailed below.\n\nWeaknesses\n* The warm start appears to have a significant impact on CUR performance (Figure 5) however it is not clear if the baselines were also allowed a warm start.\n* ROHE is not calculated for BC. The authors say this is because it is “an offline algorithm” (line 256) however it is not clear to me why this justifies not calculating it. It appears that it could be calculated given the definition of ROHE and would be a valuable comparison.\n* The benefit of CUR over baselines on the physical robot appears marginal\n* The success rate on the physical robot seems very high for even the poorest performing algorithm. If I understood this correctly there are 250 timesteps and 4 robots making a maximum of 1000 cumulative successes (and also 1000 total actions across all robots and humans (line 300)) . The worst performing policy (Random) scored 500 cumulative successes implying that a robot succeeded at the task every 2 actions. This seems high.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In my opinion, this paper currently stands on the border of a weak and strong accept. The problem setting is both novel and relevant, the paper is well written, the experiments are well chosen and thorough. I think the work makes a valuable contribution to the community by establishing a framework for IFL, defining a useful set of metrics, establishing a set of baselines and shedding light on what factors affect learning effectiveness through a thorough set of experiments and ablations.\n\nThere are some issues with the paper which I highlight in the weaknesses and below. The main one being that I am currently unsure how much of an improvement CUR represents over existing algorithms because of (a) the benefit of the warm start for CUR and it is unclear if other algorithms use this and (b) not calculating ROHE for BC and c) the marginal benefit of CUR on the physical robot\n\nHowever for me the strengths clearly outweigh the weaknesses and I don’t think the impact of this works rests on how well CUR performs.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "* It would be helpful if the authors could unify the terminology between the baselines and the ablations. The C, U, R, UR, etc notation is really clear and it strikes me that all of the baselines, except for the random baseline, could be described in this way.\n   * E.g. behavior cloning seems equivalent to C, Fleet-Ensemble DAgger seems equivalent to U.C, Fleet-Thrifty DAgger is a little different but could be described as UG.C\n* Please could the authors calculate ROHE for BC\n* Please could the authors clarify if other algorithms had a warm start\n* Please could the authors clarify the cumulative success rate on the real robot\n* I am curious what the data augmentations are (301). It would be helpful if this could be clarified in the appendix.\n* Line 265 “we use reinforcement learning agents fully trained with PPO”. What does fully trained mean? It would be helpful if the scores per environment could be given for these trained agents. For example, how many cumulative successes would the supervisor PPO agent achieve in the same 10,000 timesteps.\n* Line 163: What does “high priority” mean here. For example could it ever be that a human does not switch to assist an unassisted robot with a higher priority than the currently assisted robot?\n* Line 97 “a small fleet of robots can be difficult for a single human supervisor to optimally control” It is not clear to me why scaling the number of supervisors would help here. More concretely it is not clear there is a difference between 4 robots and 1 human and 20 robots and 5 humans, since each has to operate 4 robots on average. Please could the authors clarify this.\n* Appendix Algorithm 1: line 15, should this be N not M?\n"}}, {"review_id": "xjd-8en9pQ", "reviewer": "Reviewer_3zng", "summary": "This paper presents interactive fleet learning, in which multiple robots interactively query and learn from multiple humans. The new setting applies DAGGER on a fleet of robots and is tested in simulators and the real world. The key contribution is a novel objective to measure return on human efforts during training and different ways of assigning priorities based on constraints and uncertainty. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\nThe paper considers a very practical problem for robot fleets in the industry where minimizing human efforts and maximizing throughput is of critical importance. \n\nCompared to related works on execution time, fleet-dagger focuses on training time, and it is more scalable as a robot-gated algorithm. \n\nThe details on the formulation are novel and meaningful, such as hard reset time R takes more than 1 timestep, and T_t for minimum operation time. The consideration of constraints is also very practical (such as dropping the box in the allegro hand example). \n\nThe software and protocol are also contributions.\n\nWeakness:\n\nThe multi-human and multi-robot setting is novel, but the humans are homogeneous and robots share a single policy for the same task, which is limited and not much different from single robot-human setting except the parallel implementation (parallel MDPs). Additionally, there might be a communication bottleneck to share all data among a large fleet. \n\nThe algorithm baselines are mostly from previous works and CUR seems to be a combination with additional constraint prediction. Since these functions can be trained, assigning in order based on this score seems not that difficult. It would be interesting to consider the execution-time human-robot allocation methods as well. Moreover, some methods require pretrain goal critic or safety critic. One question is how hard it is in practice to train such an additional safety function, compared to the policy learning.\n\nFleet-dagger only focuses on the interactive learning process. But it would be very important to evaluate the trained policy as well. Does the final trained policy differ among these different methods, or only the training process is sped up? Are there some policy visualizations or generalization analyses?\n\nSince the title is fleet robotics, it would be great to have more robots and humans (beyond 4 arms and 2 robots) with more complex tasks (beyond pushing) on the real-world experiments.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a well-thought and clearly written formulation for an important real-world robotic problem. The protocol and codebase are also very useful for the community in general. There are still several weaknesses regarding formulation and algorithms and would love to hear about the response from the authors. Overall I recommend acceptance of this paper. \n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Not sure if I understand ROHE correctly. The definition seems to be dependent on M and N. If we increase N, then the numerator would get bigger since it is a sum instead of a mean, but the denominator might not change. If we increase M, the denominator would become large (if no idle humans), but the behavior on the numerator is harder to analyze because more humans might increase the overall training performance. Overall the ratio seems to complicate the definition. \n\nMore details on how to pretrain goal critics and safety critics are very helpful.\n\nFor the simulation experiment, why is CUR having a higher idle time (isn’t it a bad thing), and why in the real world it is not the case.\nMore results on test performance and visualization are helpful.\n\nSuggest moving figure 3 to earlier part of the paper.\n\nVariance can be useful to provide for multiple runs on the training process, especially with humans in the loop.\n"}}, {"review_id": "pxl818OKoe", "reviewer": "Reviewer_oMQS", "summary": "The authors formalize and provide research tools (ie, benchmark domains and metrics) for the important problem of \"interactive fleet learning,\" in which in which a set of remote (human) operators interact with a fleet of deployed robots over time and the goal is for the fleet to use the interaction data to reduce its dependence upon the human operators. The authors additionally propose and provide experimental results for a set of algorithms that they refer to as Fleet-DAgger which serve to determine which humans should interact with which robots at what times. The authors find in experiments that the proposed \"CUR\" scheme leads to better fleet learning. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) The paper articulates well an important problem in the robot learning community, ie, that of interactive fleet learning (IFL). Moreover, the paper introduces a new—and seemingly valuable—set of benchmark tasks that instantiate this problem \n\n(+) The paper proposed an intuitive and attractive \"return on human effort\" (RHOE) metric for IFL solutions. \n\n(+) The proposed method for supervisor allocation, ie, CUR, performs well in a strong experimental evaluation. \n\n \n\n(-) While I appreciate the overall definition of IFL in the introduction (ie, \"a robot fleet learning via interactive requests for human supervision\"), the Fleet-DAgger approach(es) presented in the paper focus on the particular sub-problem of \"supervisor allocation.\" While this is of course of critical importance to IFL, the paper could and should do a better job articulating that the contribution here is on the allocation part of the problem and not the learning part. For example, this could be made clearer in the contributions list in the introduction, and also in Section 4, where the underlying learning algorithm that is (presumably) re-used from other work doesn't appear to be discussed at all. \n\n(-) Related to the point above, it's not clear as to why the authors have chosen the term \"Behavior Cloning\" for the first method detailed in Section 4.2. In what sense is this related to the term as it is usually understood in the machine learning community (ie, using supervised learning to find a policy that imitates from demonstrations)? This isn't clear to me at all since this appears to again be a solution to the supervisor allocation problem rather than the learning problem. Moreover, having the term \"behavior cloning\" on the results plots (eg, Figure 2) makes it seems as though _learning_ approaches are being presented rather than _supervisor allocation_ approaches. \n\n(-) Because this paper may serve to define IFL for other researchers in the future, I think it's important that the authors highlight a few other limitations of the way that the problem has been studied here. In particular, it strikes me that this is a type of \"federated learning\" problem (perhaps something from that community should be cited?), in the context of which the assumptions about communication/synchronization (ie, that timesteps are synchronized across all robots and that they share the same \\pi_{\\theta_t} for all t) seem like pretty big leaps. For example, the authors cite in the introduction that the IFL problem shows up for deployed fleets of autonomous vehicles. I agree, but certainly these assumptions would be completely invalid in this case. How would that impact the results and conclusions that the authors present here? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper both helps formalize/advance research in an important problem area and proposes and evaluates an interesting family of algorithms that may help the community make progress in this area. That said, there are several ways in which the paper itself can and should be improved before publication in a venue like CoRL.\n\nPOST-DISCUSSION UPDATE: I think the authors have adequately addressed my concerns during the discussion phase, and I think the revised manuscript is greatly improved. As such, I'm increasing my score from \"Weak Accept\" to \"Strong Accept.\"", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "(see weaknesses identified above)", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "owbXzwdN6yK", "reviewer": "Reviewer_45kd", "summary": "Review for Fleet-DAgger: Interactive Robot Fleet Learning with Scalable Human Supervision\nThe paper studies the problem of allocating multiple human teleoperators to a fleet of robots. In a formal framework called Interactive Fleet Learning (IFL), robots query the humans to improve their behaviour. Robots are not acting as swarm, but rather do the same tasks in parallel and can share their data obtained from interacting with humans among each other. Finally, the setting is continual learning, human help is not only available in training but also during execution to further improve the robot’s policies. The authors provide a benchmark toolkit for IFL, as well as a novel IFL algorithm adapting DAgger. The work is evaluated in simulation and in hardware experiments.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper is very well written, clearly structured and studies a relevant problem to the robot learning community. Related work is reviewed in great detail, and the problem statement is thorough. The paper makes two main contributions (IFL formalism and the C.U.R. algorithm). The simulation experiments show strong results. However, I have a technical question on the objective and some concerns about the hardware experiments.Please see detailed comments below:\n\nMajor comments:\n\n•\tThe Fleet-Dagger family that unifies different algorithms is nice and provides a useful building block for designing or improving IFL algorithms\n\n•\tresolved -- About the objective in eq (2): Isn’t this objective trivially maximized by having the humans spent 0 time teleoperating? This should be clarified.\n\n•\tresolved -- Physical Experiment in Section 6.3: How many trials were carried out for this? How were human participants selected? Why are there no error bands in Figure 4? Especially with the smaller margin between algorithms, error bands / SD would be very helpful to evaluate if there is sufficient performance difference to draw any conclusions. \n\nMinor comments:\n•\tresolved -- Fig 2: The step like behaviour for CUR and Fleet-Ensemble-Dagger in the Anymal experiment are peculiar. Is there any explanation for this?\n\n•\tresolved -- A few typos, e.g. line 207 misses a space before ‘In addition’ \n\n•\tresolved -- Reference 2 is not well formatted  \n\nLimitation Section:\n\n•\tThe limitation section is adequate.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper makes valuable contributions with a unified problem formulation, the family of Fleet-Dagger algorithms, the novel CUR algorithm and a strong simulation study in a new benchmark testbed. The work is evaluated in thorough simulations, and its practicality is demonstrated in a set of hardware experiments.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "All issues have been addressed during the rebuttal phase.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (oral)