ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning
Authors: Ryan Hoque
Ashwin Balakrishna
Ellen Novoseller
Albert Wilcox
Daniel S. Brown
Ken Goldberg
Venue: CoRL2021 Oral (2021)
Keywords: Imitation Learning
Fleet Learning
Human Robot Interaction

Abstract: Effective robot learning often requires online human feedback and interventions that can cost significant human time, giving rise to the central challenge in interactive imitation learning: is it possible to control the timing and length of interventions to both facilitate learning and limit burden on the human supervisor? This paper presents ThriftyDAgger, an algorithm for actively querying a human supervisor given a desired budget of human interventions. ThriftyDAgger uses a learned switching policy to solicit interventions only at states that are sufficiently (1) novel, where the robot policy has no reference behavior to imitate, or (2) risky, where the robot has low confidence in task completion. To detect the latter, we introduce a novel metric for estimating risk under the current robot policy. Experiments in simulation and on a physical cable routing experiment suggest that ThriftyDAgger's intervention criteria balances task performance and supervisor burden more effectively than prior algorithms. ThriftyDAgger can also be applied at execution time, where it achieves a 100% success rate on both the simulation and physical tasks. A user study (N=10) in which users control a three-robot fleet while also performing a concentration task suggests that ThriftyDAgger increases human and robot performance by 58% and 80% respectively compared to the next best algorithm while reducing supervisor burden. See https://tinyurl.com/thrifty-dagger for supplementary material.

ggriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning
Ryan Hoque', Ashwin Balakrishna', Ellen Novoseller", Albert Wilcox!, Daniel S. Brown', Ken Goldberg'
Abstract: Effective robot learning often requires online human feedback and inter- ventions that cost significant human time, giving rise to the central challenge in interactive imitation learning: is it possible to control the timing and length of interventions both facilitate learning and limit burden on the human super- visor? This paper presents ThriftyDAgger, an algorithm for actively querying a human supervisor given desired budget of human interventions. ThriftyDAgger uses a learned switching policy to solicit interventions only at states that are suf- ficiently (1) novel, where the robot policy has no reference behavior to imitate, or (2) risky. where the robot has low confidence in task completion. To detect the latter, we introduce novel metric for estimating risk under the current robot policy. Experiments in simulation and on physical cable routing experiment suggest that ThrietyDAgee'' intervention criteria balances task performance and supervisor burden more effectively than prior algorithms. ThriftyDAgger can also be applied at execution time, where achieves a 100% success rate on both the simulation and physical tasks. A user study (N 10) in which users con- trol a three-robot fleet while also performing a concentration task suggests that ThriftyDAgger increases human and robot performance by 58% and 80% respec- tively compared to the next best algorithm while reducing supervisor burden. See dagger for supplementary material.
bttps://tinyur// ctw/thriiiitttteocooeeet.e ttrifty- taaggeeedggerrr
Keywords: Imitation Learning, Fleet Learning, Human Robot Interaction
1 Introduction
Imitation learning (IL) [1, 2, 3] has seen success in a variety of robotic tasks ranging from autonomous driving [4, 5, 6] to robotic manipulation [7, 8, 9, 10, 11]. its simplest form, the human provides an offline set of task demonstrations to the robot, which the robot uses to match human behavior. However, this offline approach can lead to low task performance due to a mismatch between the state distribution encountered in the demonstrations and that visited by the robot [12, 13], resulting in brittle policies that cannot be effectively deployed in real-world applications [14]. Interactive imitation learning, in which the robot periodically cedes control to human supervisor for corrective interventions, has emerged as a promising technique to address these challenges [15, 16, 17, 18). However, while interventions make it possible to learn robust policies, these interventions require significant human time. Thus, the central challenge in interactive IL algorithms to control the timing and length of interventions to balance task performance with the burden imposed on the human supervisor [19, 18]. Achieving this balance is even more critical the human supervisor must oversee multiple robots at once [20, 21, 22], for instance supervising fleet of self-driving taxis [6] or robots in a warchouse [23]. Since even relatively reliable robot policies inevitably encounter new situations that must fall back human expertise, this problem is immediately relevant to contemporary companies such as Waymo and Plus One Robotics
One way to determine when to solicit interventions is to allow the human supervisor to decide when to provide the corrective interventions. However, these approaches-termen interactive IL algorithms [15, 16, 24j-require the human supervisor to continuously monitor the robot to determine when to intervene. This imposes significant burden on the supervisor and cannot
"human-gated"
AUTOLAB at the University of California, Berkeley Corresoondence to zyanhoqqe@berreeey.yyo
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Human
0
Novel?
a
π
No
Yes
Risky?
a
Figure 1: ThriftyDAgger: Given desired context switching rate og, ThriftyDAgger transfers control to human supervisor the current state 8, is (1) sufficiently novel or (2) sufficiently risky, indicating that the probability task success low under robot policy Tr Intuitively, one should not only distrust Ty in states significantly out of the distribution of previously encountered states, but should also cede control to a human supervisor in more familiar states where the robot predicts that it is unlikely to suecessfully complete the task.
effectively scale to settings in which a small number of humans supervise a large number of robots. To address this challenge, there has been recent interest in approaches that enable the robot to actively query humans for interventions, called "robot- sabbot-gtedd algorithms [19, 25, 26, 18]. Robot- -ated methods allow the robot to reduce burden on the human supervisor by only requesting interventions when necessary, switching between robot control and human control based on some intervention criteria. Hoque et al. [18] formalize the idea of supervisor burden as the expected total cost incurred by the human providing interventions, which consists of the expected cost due to context switching between autonomous and human control and the time spent actually providing interventions. However, it is difficult to design intervention criteria that limit this burden while ensuring that the robot gains sufficient information to imitate the supervisor's 's policy.
This paper makes several contributions. First, we develop intervention criteria based on a synthesis of two estimated properties of given state: novelty, which measures whether the state is significantly out of the distribution of previously encountered states, indicating that the robot policy should be trusted; and risk, which measures the likelihood of the robot successfully completing the task on its own. While state novelty has been considered prior work [26], the key insight in our intervention criteria lies in combining novelty with new risk metric to estimate the probability of task success. Second, we present a new pobot-gated interactive IL algorithm, ThriftyDAgger (Figure 1), which employs these measures jointly to solicit human interventions only when necessary. Third, while prior robot-gated algorithms [19, 18] require careful parameter tuning to modulate the timing and frequency of human intervention requests, ThriftyDAgger only requires the supervisor to specify desired context switching rate and sets thresholds accordingly. Fourth, experimental results demonstrate DerrfttyDggeet effectiveness for reducing supervisor burden while learning challenging tasks both in simulation and an image-based cable routing task on a physical robot. Finally, the results of human user study applying ThriftyDAgge to control fleet of three simulated robots suggest that ThriftyDAgge significantly improves performance on both the robots' task and an independent human task while imposing fewer context switches, fewer human intervention actions, and lower mental load and frustration than prior algorithms.
Related Work
Imitation Learning from Human Feedback: There has been significant prior work in offline imitation learning, in which the agent leverages an offline dataset of expert demonstrations either to directly match the distribution of trajectories in the offline dataset [4, 27, 1, 3, 28, 29, 30], for instance via Behavior Cloning [31, 321, or to learn reward function that can then be optimized via reinforcement learning [33, 27, 34]. However, while these approaches have shown significant success in a number of domains [7, 10, 32], learning from purely offline data leads to a trajectory distribution mismatch which yields suboptimal performance both in theory and practice [12, 13]. To address this problem, there have been number of approaches that utilize online human feedback while the agent acts in the environment, such providing suggested actions [12, 35, 36, 17] or preferences [37, 38, 39, 40, 41, 42]. However, many of these forms of human feedback may be unreliable if the robot visits states that significantly differ from those the human supervisor would

---

themselves visit; in such situations, it is challenging for the supervisor to determine what correct behavior should look like without directly interacting with the environment [16, 43].
Interactive Imitation Learning: A natural way to collect reliable online feedback for imitation learning is to periodically cede control human supervisor, who then provides corrective intervention to illustrate desired behavior. Human- -ated interactive IL algorithmss [15, 16, 24] such as HG-DAgger [15] require the human determine when to engage in interventions. However, these algorithms require a human to continuously monitor the robot to determine when to intervene, which imposes significant burden on the supervisor and is particularly impractical if a small number of humans must supervise large number of robots. Furthermore, it requires the human to determine when the robot needs help and when to cede control, which can be unintuitive and unreliable.
By contrast, robot-gated interactive II. algorithms, such as EnsembleDAgger [26], SafeDAgger [19), and LazyDDggger [18], allow the robot actively query for human interventions. In practice, these algorithms estimate various quantities correlated with task performance [19, 18, 44, 25] and uncertainty [26] and use them determine when solicit interventions. Prior work has proposed intervention criteria which use the novelty of states visited by the robot [26] or the predicted discrepancy between the actions proposed by the robot policy and those of the supervisor [19, 18). However, while state novelty provides a valuable signal for soliciting interventions, we argue that this alone insufficient, as state's novelty does not convey information about the level of precision with which actions must be executed in that state. In practice, many robotic tasks involve moving through critical "bottlenecks" [24], which, though not necessarily novel, still present challenges. Examples include moving an eating utensil close to a person's 's mouth or placing an object on shelf without disturbing nearby objects. Similarly, even if predicted accurately, action discrepancy is often a flawed risk measure, as high action discrepancy between the robot and the supervisor may be permissible when fine-graines control is not necessary (e.g. a robot gripper moving in free space) but impermissible when precision is critical (e.g. robot gripper actively trying to grasp object). In contrast, ThriftyDAgger presents an intervention criteria incorporating both state novelty and a novel risk metric and automatically tunes key parameters, allowing efficient use of human supervision.
3 Problem Statement
Given a robot, task for the robot to accomplish, and a human supervisor with a specified context switching budget, the goal is to train the robot to imitate supervisor performance within the budget. We model the robot environment as a discrete-time Markov Decision Process (MDP) M with continuous states E S, continuous actions a E A, and time horizon T' [45]. We consider the interactive imitation learning (IL) setting [15], where the robot does not have access to shaped reward function or to the MDP's ' transition dynamics but can temporarily cede control to a supervisor who uses policy Th : S A. We specifically focus on tasks where there is goal set 9 which determines success, but that can be challenging and fong-horizon, making direct application of RL highly sample inefficient.
We assume that the human and robot utilize the same action space (e.g. through teleoperation interface) and that task success can be specified by convergence to some goal set G C S within the time horizon (i.e., the task is successful if G is reached within I' timesteps). We further assume access to an indicator function ggg : S + {0, 1}, which indicates whether a state belongs to the goal set G.
The IL objective is to minimize surrogate loss function J(Ty) to encourage the robot policy Tr : S - A to match Ty.
T
J(TT) = Σ Farmar (S)T) Tn (8+)))
where (s), (s)) is an action discrepancy measure between ( and TA (s) (e.g. MSE loss), and dir is the marginal state distribution at timestep / induced by the robot policy n., in M.
In the interactive IL setting, in addition to optimizing Equation (1), a key design goal is to minimize the imposed burden on the human supervisor. To formalize this, we define switching policy म, which determines whether the system is under robot control गr. (which we call autonomous mode) or human supervisor control Ty (which we call supervisor mode). Following prior work [18], we define (IT)), the expected number of context switches in an episode under policy म, as follows: C(T)) = ((T)=I= Ee,-dy (m:(s;;)) where ms(s;; T) is an indicator for whether not a context switch occurs from autonomous to supervisor control. Similarly, we define (TT) the expected
3

---

number of supervisor actions in an intervention solicited by π. We then define the total burden (x))
imposed on the human supervisor as follows:
B(T)) = C(T) - (L+ ((T).
where L is the lalency of a context switch between control modes (summed over both switching
directions) in units of timesteps, where each action takes one timestep. The interactive IL objective to minimize the discrepancy from the supervisor policy while limiting supervisor burden within some bbb:
π arg min (J B(x') rT..
(3)
x'Ell
Because it is challenging to explicitly optimize policies to satisfy the supervisor burden constraint in Equation (3), we present novel intervention criteria that enable reduction of supervisor burden by limiting the total number of interventions to user-specified budget. Given sufficiently high latency L, limiting the interventions (TT) directly corresponds limiting supervisor burden ((T).
4 ThriftyDA gger
ThriftyDAgger determines when to switch between autonomous and human supervisor control modes by leveraging estimates of both the novelty and risk of states. Below, Sections 4.1 and 4.2 discuss the estimation of state novelty and risk of task failure, respectively, while Section 4.3 discusses ThriftyDAgger's integration of these measures to determine when to switch control modes. Section 4.4 then describes online procedure to set thresholds for switching between control modes. Finally, Section 4.5 describes the full control flow of ThriftyDAgger.
4.1 Novelty Estimation
When the robot policy visits states that lie significantly outside the distribution of those encountered in the supervisor trajectories, does not have any reference behavior to imitate. This motivates initiating interventions to illustrate desired recovery behaviors in these states. However, estimating the support of the state distribution visited by the human supervisor is challenging in the high-dimensional state spaces common in robotics. Following prior work [26], we train an ensemble of policies with bootstrapped samples of transitions from supervisor trajectories. We then measure the novelty of a given state by calculating the variance of the policy outputs at state across ensemble members. In practice, the action a E A outputted by each policy is a vector; thus, we measure state novelty by computing the variance of each component of the action vector a across the ensemble members and then averaging over the components. We denote this quantity by Novelty(s). Once in supervisor mode, noted in Hoque et al. [18]. we can obtain more precise correlate of novelty by computing the ground truth action discrepancy between the supervisor's 's actions and those of the robot policy.
4.2 Risk Estimation
Interventions may be required not only in novel states outside the distribution of supervisor trajectories, but also in familiar states that are prone to result in task failure. For example, a task might have a "bottleneck" region with low tolerance for error, which has low novelty but nevertheless requires more supervision to learn a reliable robot policy. To address this challenge, we propose a novel measure of state's "riskiness," capturing the likelihood that the robot cannot successfully converge to the goal set G. We first define a Q-function to quantify the discounted probability of successful convergence to G from given state and action under the robot policy:
Q (Sc, 0+) Eeg may
(4)
where (sg(s:) is equal to 1 if belongs to G. We estimate 27 (St, dz) via function approximator 2% parameterized by d, and define a state's riskiness in terms of this learned Q-function:
Risk (s,a) = = 11-O8 (s,g()
(5)
In practice, we train Q3 on transitions (st, ae, S1+1) from both autonomous mode and supervisor mode by minimizing the following MSE loss inspired by [46]:
(st; Ges s411: Φ) = 2 ( (0 ar) - (Ig(s:)+ (1 lgg(s))) ( (( (+(11))) (6)
1
Note that since Q% is only used to solicit interventions, must only be accurate enough to distinguish risky states from others, rather than be able to make the fine-grained distinctions between different states required for accurate policy learning in reinforeement learning.

---

4.3 Regulating Switches in Control Modes
We now describe how ThriftyDAgger leverages the novelty estimator from Section 4.1 and the risk estimator from Section 4.2 to regulate switches between autonomous and supervisor control. While in autonomous mode, the switching policy т initiates a switch to supervisor mode at timestep t either (1) state s; is sufficiently unfamiliar or (2) the robot policy has a low probability of task success from st Stated precisely, I initiates a switch to supervisor mode from autonomous mode at timestep t if the predicate Intervene St, 5h, Ba)) evaluates to TRUE, where Intervene oh, .x,) is TRUE if (1) Novelty(st) > 5n or (2) Risk (st. Fr (st)) > Sr. and FALSE otherwise. Note that the proposed switching policy only depends on Risk" for states which are not novel (as novel states already initiate switches to supervisor control regardless of risk), since the learned risk measure should only be trusted on states in the neighborhood of those on which it has been trained.
In supervisor mode, π switches autonomous mode the action discrepancy between the human and robot policy and the robot's task failure risk are both below threshold values (Section 4.4), indicating that the robot is in a familiar and safe region. Stated precisely, π switches to autonomous mode from supervisor mode if the predicate Cede(s; St. 8,, B,) evaluates to TRUE, where Cede (s. 6,, 8,) is TRUE if (1) || (st) - TA (s)) < 8, and (2) disk"""sss क. (st)) < Br. and FALSE otherwise. Here, the risk metric ensures that the robot has a high probability of autonomously completing the task, while the coarser 1 I-step action discrepancy metric verifies that we are in familiar region of the state space where the 2% values can be trusted. Motivated by prior work [18] and hysteresis control [47], we use asymmetric switching criteria with stricter thresholds in supervisor mode (3, < (3,)) to encourage lengthier interventions and reduce context switches experienced by the human supervisor.
4.4 Computing Risk and Novelty Thresholds from Data
One challenge of the control strategy presented in Section 4.3 lies in tuning the key parameters (ō,, 5,. 3h, (.)) governing when context switching occurs. As noted in prior work [26]. performance and supervisor burden can be sensitive to these thresholds. To address this difficulty, we assume that the user specifies their availability the form of desired intervention budget ar E [o. 1]. indicating the desired proportion of timesteps in which interventions will be requested. This desired context switching rate can be interpreted in the context of supervisor burden as defined in Equation (2): if the latency of a context switch dominates the time cost of the intervention itself, limiting the expected number of context switches to within some intervention budget directly limits supervisor burden.
Given Oh, we set 3h. to be the (1 - ous) -quantile of Risk (s, ,. (s)) for all states previously visited by Try and set dn. to be the (1 or --qanttile of Novelty(s) (s) for all states previously visited by T., We set o, to be the mean action discrepancy on the states visited by the supervisor after π. is trained and set B, to be the median of Risk (s, πy. (s)) for all states previously visited by ky. (Note that 8, can easily be set to different quantiles to adjust mean intervention length if desired.) We find that these settings strike a balance between informative interventions and imposed supervisor burden.
4.5 DrriftyDAgger Overview
We now summarize the ThriftyDAgger procedure, with full pseudocode available in the supplement. TAriftyDAgger first initializes म, via Behavior Cloning on offline transitions (D,, from the human supervisor, T.). Then, म. collects an initial offline dataset D, from the resulting Tr, initializes Q% by optimizing Equation (5) on D, DD,, and initializes parameters 3h, Sr, 6n. and 8, as in Section 4.4. We then collect data for N episodes, each with up to 1" timesteps. In each timestep of each episode, we determine whether robot policy ग. or human supervisor n should be in control using the procedure in Section 4.3. Transitions autonomous mode are aggregated into D, while transitions in supervisor mode are aggregated into Dr. After each episode, म. is updated via supervised learning on Dj, and 2% is then updated on D, U Dr, to reflect the probability of task success of the updated n..
Experiments
In the following experiments, we study whether DAgger can balance task performance and supervisor burden more effectively than prior IL algorithms in three contexts: (1) training a simulated robot to perform peg insertion task (Section 5.3); (2) supervising a fleet of three simulated robots to perform the peg insertion task in a human user study (Section 5.4); and (3) training a physical surgical robot to perform cable routing task (Section 5.5). the supplementary material, we also include results from additional simulation experiment a challenging block stacking task.
5

---

5.1 Evaluation Metries
We consider ThriftyDAgger's performance during training and execution. For the latter, we evaluate both the (1) autonomous success rate, or success rate when deployed after training without access to human supervisor, and (2) intervention-aide success rate, or success rate when deployed after training with human supervisor in the loop. These metrics are reported in the Peg Insertion study (Section 5.3) and the Physical Cable Routing study (Section 5.5). For all experiments, during both training and intervention-aided execution, we evaluate the number of interventions, human actions, and robot actions per episode. These metrics are computed over successful episodes only to prevent biasing the metrics by the maximum episode horizon length T': such bias occurs, for instance, when less successful policies appear to take more actions due to hitting the time boundary more often. Additional metrics including cumulative statistics across all episodes are reported in the supplement. In our user study (Section 5.4), we also report the following quantities: throughput (total number of task successes across the three robots), performance on an independent human task, the idle time of the robots in the fleet, and users' qualitative ratings of mental load and frustration. By comparing the amount of human supervision and success rates across different algorithms, we are interested in evaluating how effectively each algorithm balances supervision with policy performance.
5.2 Comparisons
We compare ThrietyDAgger to the following algorithms: Behavior Cloning, which does not use inter- ventions; HG-DAgger [15], which human-gated and always requires supervision; SafeDAgger [19], which is robot-gated and performs interventions based on estimated action discrepancy between the human supervisor and robot policy; and LazyDAgger [18], which builds on SafeDAgger by introducing asymmetrie switching criteria encourage lengthier interventions. We also implement two ablations: one that does not use a novelty measure to regulate context switches (ThriftyDAgee) (-Novelty)) and one that does not use risk to regulate context switches (ThriftyDAgger (-Risk)).
5.3 Peg Insertion in Simulation
We first evaluate Thrifty on long-horizon (100+ timesteps) peg insertion task (Figure 2) from the Robosuite simulation environment [48]. The goal is to grasp a ring in a random initial pose and thread it over cylinder at a fixed target location. This task has two bottlenecks which motivate learning from interventions: (1) correctly grasping the ring and (2) correctly placing it over the cylinder. A human teleoperates the robot through keyboard interface provide interventions. The states consist of the robot's joint angles and ring's pose, while the actions specify 3D translation, rotation, and opening intervention frequency os = 0.01 (Section 4.4). We collect 30 offline task demos (2,687 state-action pairs) from human supervisor to initialize the robot policy for all compared algorithms. Behavior Cloning is given additional state-action pairs roughly equivalent to the average amount of supervisor actions solicited by the interactive algorithms (Table 4 in the appendix). For Trgget and each interactive IL baseline, we perform 10,000 environment steps, during which each episode takes most 175 timesteps and system control switches between the human and robot. Hyperparameter settings for all algorithms are detailed in the supplement.
closing the gripper. For ThriftyDAgger and its ablations, we use target
Results (Table 1) suggest that DAgier simultaneously solicits fewer interventions and achieves a significantly higher autonomous success rate than prior robot-gated algorithms, although it does request more human actions due to its conservative exit criterion for interventions (Cede(sr, (s,, 6,, B.))). The number of human actions falls significantly at execution time (Table 1), when the robot policy has been trained on online data and is therefore less risky. We find that all interactive II algorithms substantially outperform Behavior Cloning, which does not have access to supervisor interventions. Notably, ThriftyDAgger achieves a higher autonomous success rate than even HG-DAgger, in which the supervisor is able to decide the timing and length of interventions. This indicates that ThriftyDAgger's intervention criteria enable it autonomously solicit interventions as informative as those chosen by a human supervisor with expert knowledge of the task. Furthermore, ThriftyDAgee achieves a 100% intervention-aide successfully identifies the required states at which to solicit interventions. We find that both ablations of ThriftyDAgger (Ours (-Novelty) and Ours (-Risk)) achieve significantly lower autonomous success rates, indicating that both the novelty and risk measures are critical to ThriftyDAgger's performance. We calculate ThriftyDAgges 's context switching rate to be 1.15% novelty switches and 0.79% risk switches, both approximately within the budget of ay 0.01.
success rate execution time, suggesting that ThriftyDAgge

---

Start
Goal
Novel
Risky
Safe
Simulation:
Peg Insertion
Physical: Cable Routing
Figure 2: Experimental Domains: We visualize the peg insertion simulation domain (top row) and the cable routing domain with the physical robot (bottom row). We visualize sample start and goal states, in addition to states which ThriftyDAgger categorizes as novel, risky, or neither. ThriftYDAgerr DAgger marks states as novel if they are far from states that the supervisor visited and risky if the robot is stuck in bottleneck, e.g. if the ring is wedged against the side of the cylinder (top) or the cable is near all four obstacles (bottom).
Table I: Peg Insertion in Simulation Results: We first report training performance (number of interventions (Ints), number of human actions (Acts (H)), and number of robot actions (Acts (R))) and report the success rate of the bully-trainee policy at exccution time when no interventions the fully-traines policies with interventions allowed and report the same intervention statisties and the success rate (Int-Aided Succ.). We find that ThriftyDAgger achieves the highest autonomous and intervention-aided success rates among all algorithms compared. Notably, ThriftyDAgger even achieves a higher autonomous success rate than HG- DAgger, in which the human decides when intervene during training.
allowed (Auto Suce.). We then evaluate
Algorihh
Training Interventions
Anto Succ.
Exccution Interventons
Ini-Aided Succ.
Inis
Acis (H)
Acts (R)
Inis
Acts (H)
Acts [R]
Cloming
15.9
21,100
SafeDAgger
3.89 1 1.44
19.8 9.9
88.8
19.4
24/100
4.00 1: 1.37
19.5 d2 5.3
77.5 11.7
17/20
Lazy DAgger
1.46 2 1.15
13.2
12.2:12
12.1
102 100.14.
18.2
28/100
1.73 1: 1.29
12.6
1.6:110 14.1
91.7
1 21.0
11/20
HG-DAgger
1.40 1: 0.88
20.3
2 15.6
97.
1: 17.5
57/100
1.15 0.73
17.1
11.6
103.6
14.0
20/20
(-Novelty)
35.8
49/100
0.33
11.00
12/20
Oun (-Risk)
0.99 1: 0.96
7.8
12.0
101.2
19.2
49/100
1.39 1: 0.95
9.5 12.0
109.1
22.9
18/20
Ouns: DAgger
0.88 :: 1.01
43.6
2 21.5
60.0 1: 32.8
73/100
1.35 1: 0.66
21.3
5 15.0
81.8
1 21.8
20/ 20
5.4 User Study: Controlling A Fleet of Three Robots in Simulation
We conduct user study with 10 participants (7 male and female, aged 18-37). Participants supervise a fleet of three simulated robots, each performing the peg insertion task from Section 5.3. We evaluate how different interactive IL algorithms affect the participants' (1) ability to provide effective robot interventions, (2) performance on a distractor task performed between robot interventions, and (3) levels of mental demand and frustration. For the distractor task, we use the game Concentration (also known as Memory or Matching Pairs), in which participants identify as many pairs of matching cards as possible among a set of face-down cards. This is intended to emulate tasks which require continual focus, such cooking meal or writing research paper, in which frequent context switches between performing the task and helping the robots is frustrating and degrades performance.
The participants teleoperate the robots using three robot-gated interactive IL algorithms: SafeDAgger, LazyDAgger, and DAriftyDAgger. The participant is instructed to make progress on the distractor task only when no robot requests an intervention. When an intervention is requested, the participant instructed pause the distractor task, provide an intervention from the requested state until the robot (or multiple robots queued after each other) no longer requires assistance, and then return the distractor task. The participants also teleoperate with HG-DAgger, where they no longer perform the distractor task and are instructed to continually monitor all three robots simultaneously and decide on the length and timing of interventions themselves. Each algorithm runs for 350 timesteps, where in each timestep, all robots in autonomous mode execute action and the human executes action on the currently supervised robot (if applicable). The supplement illustrates the user study interface and fully details the experiment protocol. All algorithms are initialized as in Section 5.3.
Results (Table 2) suggest that ThriftyDAgger achieves significantly higher throughput than all prior algorithms while requiring fewer interventions and fewer human actions, indicating that ThriftyDAg. ger requests interventions more judiciously than prior algorithms. Furthermore, Thrifty DAgger also enables a lower mean idle time for robots and higher performance on the distractor task. Notably, DnggetyDAgger solicits fewer interventions and total actions while achieving a higher throughput than HG-DAgger, in which the participant chooses when to intervene. We also report metrics of users'

---

Table 2: Three-Robot Fleet Control User Study Results: Results for experiments with 10 human subjects and 3 simulated robols on the peg insertion task. We report the total numbers interventions, human actions, and robot actions, as well as the throughput, or total task successes achieved across robots, for all algorithms. Additionally, for robot- gated algorithms, we report the Concentration score (number of pairs found) and the mean idle time of robots in the flect in timesteps. Results suggest that ThriftyDAgger outperforms all prior algorithms across all metrics, requesting fewer interventions and total human actions while achieving higher throughput, lowering the rohots' mean idle time, and enabling higher performance on the Concentration task.
Algorithm
Interventions
Humin Actions
Robot Actions
Concentration Pairs
Throughput
Mean Idle Time
HG-DAgger
2..6
198.0 321
x34,4
N/A 17.7 + 8.2
3.T
N/A 38.4 14.1
SafeDAgger
22.1
234.1 = 318
700.7 = 70.4
3.0
LazyDAgger
10.0
219.5 + 43,3
719.2 = 89.7
20.9 + 7.9
5.1
37.1 = 20.5
Ours: ThrityDAgger
2.9
179.4 34.9
793.2 = 86.6
33.0 8.5
9.2
25.8 + 19.3
Table 3: Physical Cable Routing Results: We first report intervention statisties during training (number of interventions (Ints), number of human actions (Acts (H)), and number of robot actions (Acts (RJ)) and report the success rate the fully-trained policy at execution time when no interventions are allowed (Auto Succ.). We then evaluate the fully-trained policies with interventions allowed and report the same intervention statistics and the success rate (Int-Aided Succ.). We find that ThriftyDAgger achieves the highest autonomous and
intervention-aided success rates among all algorithms compared. Notably, ThriftyDAgger achieves comparable
autonomous success rate to HGgDAgger. which the human decides when to intervene during training.
Algorilii
Training Interventions
Execution
Aid-Ai Suce.
Ints
NA Acts (HI)
Acts (R)
Ints
Acts (H)
Acts (R)
Behavior Cloning
NIA
NIA
0/15
N/A
N/A
NA
NA 15/
HG-DAgger
1.55
13.9
55.5 4
10/15
0.40
2.7
1.99
Ours: ThriftyDAgeer
1.42 +
1.14
15.2 + 12.4
45.5 + 18.3
12/15
0.40
+ 0.71
1.5 3.1
61.++665
15/15
mental workload and frustration using the NASA-TIL -LX scale [49] in the supplement. Results suggest that users experience lower degrees of frustration and mental load when interacting with ThriftyDAg- ger and LazyDAgger compared to HG-DAgger and SafeDAgger We hypothesize that participants struggle with HG-DAgger due the difficultly of monitoring multiple robots simultaneously, while SageDAgger's 's frequent context switches lead to user frustration during experiments.
5.5 Physical Experiment: Visuomotor Cable Routing
Finally, we evaluate ThriftyDAgger on long-horizon cable routing task with da Vinei surgical robot [50]. Here, the objective is to route a red cable into a Figure-8 pattern around 4 pegs via teleoperation with the robot's master controllers (see supplement). The algorithm only observes high-dimensional 64 x 64 x 3 RGB images of the workspace and generates continuous actions representing delta-positions in (x. y). As in Section 5.3, ThriftyDAgger uses a target intervention frequency of a = 0.01. We collect 25 offline task demonstrations (1,381 state-action pairs) from a human supervisor to initialize the robot policy for ThriftyDAgge and all comparisons. We perform 1,500 environment steps, where each episode has at most 100 timesteps and system control can switch between the human and robot. The supplement details the hyperparameter settings for all algorithms.
Results (Table 3) suggest that both ThriftyDAgger and HG-DAgger achieve a significantly higher au- tonomous success rate than Behavior Cloning, which never able to complete the task. Furthermore, ThriftyDAgger achieves a comparable autonomous success rate to HG-DAgger while requesting fewer interventions and a similar number of total human actions. This again suggests that ThriftyDAgger's intervention criteria enable it to solicit interventions equally as informative or more informative than those chosen by human supervisor. Finally, at execution time ThriftyDAgger achieves a 100% intervention-aide aided success rate with minimal supervision, again indicating that DAgger successfully identifies the timing and length of interventions to increase policy reliability.
Discussion and Future Work
We present Thrifty DAgger, a scalable robot-gated interactive imitation learning algorithm that leverages learned estimates of state novelty and risk of task failure to reduce burden on a human supervisor during training and execution. Experiments suggest that ThriftyDAgger effectively enables long- hong-horizon robotic manipulation tasks in simulation, on physical robot, and for a three-robot fleet while limiting burden on a human supervisor. In future work, we hope to apply ideas from ThriftyDAgeer to interactive reinforcement learning and larger scale fleets of physical robots. We also hope to study how ThriftyDAgger's performance varies with the target supervisor burden specified via on. In practice, ag could even be time-varying: for instance, 0% may be significantly lower at night, when human operators may have limited availability. Similarly, on may be set to a higher value during training than at deployment, when the robot policy is typically higher quality.
8

---

Acknowledgments
This research was performed at the AUTOLAB at UC Berkeley in affiliation with the Berkeley AI Research (BAIR) Lab and the CITRIS "People and Robots" (CPAR) Initiative. The authors were supported in part by the Scalable Collaborative Human-Robot Learning (SCHooL) Project, NSF National Robotics Initiative Award 1734633, and by donations from Google, Siemens, Amazon Robotics, Toyota Research Institute, Autodesk, Honda, Intel, and Hewlett-Packart and by equipment grants from PhotoNeo, NVidia, and Intuitive Surgical. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors. We thank our colleagues who provided helpful feedback, code, and suggestions, especially Anca Dragan, Vincent Lim, and Zaynah Javed.
References
[1] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469 -833, 2009.
[2] J. A. Bagnell. An invitation to imitation. Technical Report CMU-RI-TR-15-00 Carnegie Mellon University, Pittsburgh, PA, 3 2015.
[3] Osa, J. Pajarinen, G. Neumann, A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective on imitation learning, arXiv preprint arXiv: /81 1.06711, 2018.
[4] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1), 1991.
[5] Y. Pan, C.-A. Cheng, K. Saigol, K. Lee, X. Yan, E. Theodorou, and B. Boots. Agile autonomous driving using end-to- end-to-end deep imitation learning. In Proc. Robotics: Science and Systems (RSS), 2018.
[6] F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4693-4700. IERE, 2018
[7] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. Conf. on Robot Learning (CoRL), 2017.
[8] B. Fang, S. Jia, D. Guo, M. Xu, S. Wen, and F. Sun. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Applications, 3(4):362-369, 2019.
[9] A. Ganapathi, P. Sundaresan, Thanamjeyan, A. Balakrishna, D. Seita, J. Grannen, M. Hwang. R. Hoque, J. E. Gonzalez, N. Jamali, K. Yamane, S. Iba, and K. Goldberg. Learning Dense Visual Correspondences Simulation to Smooth and Fold Real Fabries. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2021.
[10] Sundaresan, P. Grannen, B. Thananjeyam, A. Balakrishna, M. Laskey, K. Stone, E. Gonzalez, and K. Goldberg. Learning interpretable and transferable rope manipulation policies using depth sensing and dense object descriptors. In International Conference on Robotics and Automation (ICRA). IEEE, 2020.
[11] O. Kroemer, Nickum, and G. Konidaris. A review of robot learning for manipulation: Challenges, representations, and algorithms. J. Mach. Learn. Res., 22:30-1, 2021.
[12] S. Ross, G. Gordon, and J. A. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011.
[13] M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg, DART: Noise injection for robust imitation learning. In Conf. on Robot Learning (CoRL), 2017.
[14] A. Irpan. Deep reinforcement learning doesn't 't work yet. https://wwx alexirpan com/ 2018/02/14/r1 http//11////hhrrd hard. htmll...l,,....m.mmmmmmmmmm.11.m 2018

---

[15] M. Kelly, C. Sidrane, K. Driggs-Campbell Campbell. and M. J. Kochenderfer. HG-DAgger: Interactive imitation learning with human experts. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2019.
[16] Spencer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang, P. Ramadge, and S. Srinivasa. Learning from interventions: Human-robot interaction as both explicit and implicit feedback. In Proc. Robotics: Science and Systems (RSS), 2020.
[17] Jauhri, C. Celemin, and J. Kober. Interactive imitation learning in state-space. In Conference on Robot Learning (CoRL). PMLR, 2020.
[18] R. Hoque, A. Balakrishna, C. Putterman, M. Luo, D. S. Brown, D. Seita, B. Thananjeyan, E. Novoseller, and K. Goldberg. LazyDAgger: Reducing context switching in interactive imitation learning. International Conference on Automation Sciences and Engineering (CASE), 2021.
[19] Zhang and K. Cho. Query-efficient imitation learning for end-to-ent autonomous driving. In Association for the Advancement of Artificial Intelligence (AAAI), 2017.
[20] W. Crandall, M. A. Goodrich, D. R. Olsen, and C. W. Nielsen. Validating human-robot interaction schemes in multitasking environments. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 35(4):438-44) 449, 2005.
[21] J. Y. Chen, M. J. Barnes, and M. Harper-Sciarini. Supervisory control of multiple robots: Human- Human- performance issues and user-interface design. IEEE Transactions Systems, Man, and Cybermetics, Part C (Applications and Reviews), 41(4):435-45 454, 2010.
[22] G. Swamy, S. Reddy, S. Levine, and A. D. Dragan. Scaled autonomy: Enabling human operators to control robot fleets. In 2020 IEEE Internationul Conference on Robotics and Automation (ICRA), pages 5942-5948. IEEE, 2020.
[23] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V. Vanhoucke, and S. Levine. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. Conference on Robot Learning (CoRL), 2018.
[24] A. Mandlekar, D. Xu, R. Martin-Martín, Y. Zhu, L. Fei-Fei, and S. Savarese. Human-in caaun-i--the-loog imitation learning using remote teleoperation, 2020.
[25] M. Laskey, S. Staszak, W. Hsieh, Mahler, F. Pokorny, A. Dragan, and K. Goldberg. SHIV: Reducing supervisor burden using support vectors for efficient learning from demonstrations in high dimensional state spaces. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2016.
[26] K. Menda, K. Approach to Safe Imitation Learning. In Proc. JEEE/RSJ Int. Conf. on Intelligent Robots and Sysiems (IROS), 2019.
and M. J. Kochenderfer. EnsembleDAgger. A Bayesian
[27] J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in Neural Informa- tion Processing Systems, 2016.
[28] S. Arora and P. Doshi. A survey of inverse reinforcement learning: Challenges, methods and 2018
progress. arXiv preprint
[29] A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 25(2), 2013.
[30] A. Paraschos, C. Daniel, J. R. Peters, and G. Neumann. Probabilistic movement primitives, In Advances in neural information processing systems, pages 2616-2624, 2013.
[31] F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), 7 2018.
[32] Codevilla, Santana, A. M., and A. Gaidon. Exploring the limitations of behavior cloning for autonomous driving, International Conference Computer Vision, 2019.
10

---

[33] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty- -irst international conference on Machine learning, page 2004.
[34] D. S. Brown, W. Goo, and Niekum. Beeter-than-demonntraao imitation learning via automaticaly-ranked demonstrations. In Conference on Robot Learning (CoRL), 2019.
[35] A. Balakrishna*, B. Thananjeyan" J. Lee, F. Li, A. Zahed, J. E. Gonzalez, and K. Goldberg. On-policy robot imitation learning from a converging supervisor. In Conference on Robot Learning (CoRL). PMLR, 2019.
[36] K. Judah, A. Fern, and T. Dietterich. Active imitation learning via state queries. In Proceedings of the ICML workshop on combining learning strategies to reduce label cost. Citescer, 2011.
[37] D. Sadigh, A. D. Dragan, S. S. Sastry, and S. A. Seshia. Active preference- based learning of reward functions. In Proceedings of Robotics: Science and Systems (RSS), 2017.
[38] F. Christiano, Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[39] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in Atari. In Advances in Neural Information Processing Systems, 2018.
[40] M. Palan, N. C. Landolfi, G. Shevehuk, and D. Sadigh. Learning reward functions by integrating human demonstrations and preferences. Proceedings of Robotics: Science and Systems (RSS), 2019.
[41] E. Biyık, M. Palan, N. C. Landolfi, D. P. Losey, and D. Sadigh. Asking easy questions: A user-friendly approach to active reward learning. arXiv preprint 1rXiv:1910.04365, 2019.
[42] D. Brown, R. Coleman, R. Srinivasan, and S. Nickum. Safe imitation learning via fast Bayesian reward inference from preferences. In International Conference on Machine Learning, pages 1 1175-1177. PMLR, 2020.
[43] S. Reddy, A. D. Dragan, and S. Levine. Shared autonomy via deep reinforcement learning. Proc. Robotics: Science and Systems (RSS), 2018
[44] S. Ross and A. Bagnell, Reinforcement and imitation learning via interactive no-regret learning, 2014.
[45] M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[46] B. Thananjeyan*, A. Balakrishna*, S. Nair, M. Luo, K. Srinivasan, M. Hwang, and J. Gonzalez. Recovery rl: Safe reinforcement learning with learned recovery zones. 2020.
[47] B. K. Bose. An adaptive hysteresis-band current control technique of a voltage-fed PWM inverter for machine drive system. IEEE Transactions on Industrial Electronics, 37(5), 1990.
[48] Y. Zhu, Wong, A. Mandlekar, and R. Martin-Marthn Robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. 2tt.:00099.122,,,... 2020.
[49] S. G. Hart. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, volume 50, pages 904-908. Sage publications Sage CA: Los Angeles, CA, 2006.
[50] P. Kazanzides, Z. Chen, A. Deguet, G. source research kit for the da Vinci surgical system. In 2014 IEEE international conference on robotics and automation (ICRA), pages 6434-6439. IEEE, 2014.
S. Fischer, R. H. Taylor, and S. P. DiMaio. An open-
[51] E. Todorov, T. Erez, and Y. Tassa. MuJoCo: Physies Engine for Model-Based Control. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2012.
11


Reviews:
[{"review_id": "mJwUnby5V_Z", "reviewer": "Reviewer_PRtc", "summary": "This paper addresses the problem of robot-gated interactive imitation learning, where a robot actively cedes control to a human operator in situations where it requires action supervision. Compare to prior work, the primary novelty is the inclusion of a \"risk\" term (in addition to a \"novelty\" term) to determine when the robot should cede control to the human operator, or take control back from the human operator. The method is shown to outperform baselines in terms of success rate and amount of human effort needed on a peg insertion task in simulation and a cable routing task on a physical robot. A user study is also conducted, where users attempt to focus on a pattern matching game, while also trying to provide interventions (as requested) by a fleet of 3 robot arms that are trying to learn the peg insertion task in simulation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe paper is well-written and tackles an important problem. The proposed method is shown to compare favorably to several baselines on a simulated peg insertion task and a real cable routing task, in terms of the amount of human effort needed to learn the task, and the final performance of the trained model. The included user study across 10 users that each try to assist a fleet of 3 robot arms is also very interesting as a proof-of-concept for the proposed method. The parameters of the switching policy (which are usually manually specified in prior work) are determined from a single hyperparameter that correlates to the budget for the expected amount of effort a human is willing to provide in training the robot -- this is a significant improvement over other methods. The attached video also provides a useful overview.\n\nWeaknesses\n\nThe two main weaknesses are the lack of novelty and breadth of the experiments. The main difference from LazyDagger appears to be the inclusion of a risk term, and automatic hyperparameter tuning. That being said, the risk term appears to empirically make a large difference, but there isn't as much intuition or justification provided in the paper for why it makes such a large difference. Furthermore, it's unclear why the proposed risk term works well - the risk term should vary naturally during the episode, by state. At the beginning of the episode, there are several timesteps left to reach the goal, and towards the end, there will be few. How can this term distinguish between states that are far away from the goal, and states that will never reach the goal? Providing some concrete qualitative examples of states along with the value of the risk term at those states might be useful to understand this. \n\nAlso, while the number of baselines is high, the number of tasks used for evaluation is lacking - only one task in simulation and one in real is used for evaluation. It would be useful to demonstrate the effectiveness of the method on at least one more simulated task, and preferably one that is more challenging than the peg insertion task (the fitting appears to be relatively high tolerance).  \n\nAdditional comments follow:\n- Reporting additional metrics could be beneficial to provide a more complete picture, with respect to the comparison to BC. Table 1 would ideally report the number of episodes / total number of samples used for training each algorithm (both human and robot samples). Including information on the number of successful and unsuccessful trajectories used for each method would be useful as well (perhaps in the supplementary), since the proposed method is able to leverage unsuccessful trajectories to train the risk metric (value function). It might also be useful to report other metrics like wall clock time - there's likely a tradeoff here (for example, perhaps 100 demonstrations could be collected in 1 hour of human time, allowing the robot to match the success rate of the proposed method, which might use 15 minutes of total human time, spread out over 3 hours. The proposed method would take more overall human + robot wall clock time, but less human time). \n- The paper concludes that the proposed method outperforms HG-Dagger, after presenting the results in Table 1 on the peg insertion task. However, as mentioned previously, there should be at least 1 more task in simulation, to make the claim more solid, as the performance gap appears to be much tighter on the real world task. Furthermore, while the proposed algorithm outperforms HG-Dagger in the user study, the setup there was already biased against HG-Dagger, since participants must actively intervent on 3 robots operating at once (which is definitely cognitively demanding). Including further results in the single robot case across multiple users would also substantiate the claim on outperforming HG-Dagger further.\n- From the supplementary material, it seems like every action in the peg insertion task is followed by 10 zero actions, in order \"to avoid bias due to variable teleoperation speeds\". This sounds like an unconventional design choice that could impact the difficulty of the task and could merit further discussion and motivation. This might also explain why the motion of the robot appears stilted in the included video (although that could also be an artifact of the teleoperation interface).\n- What happens if the algorithm cedes control to the human, but the robot is in a state in which it is difficult or impossible to recover? Does the human just choose to terminate the episode? How often does this occur in practice? \n- Line 328 - typo - \"at which solicit interventions\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the proposed method has some favorable characteristics and is shown to outperform baselines, the method is similar to prior work except for what appears to be one critical component (using the risk metric in determining when to switch between robot and human). Further discussion of this critical component would be beneficial. The breadth of the empirical evaluation is also limited.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Discuss the role of the risk term - why it is expected to help significantly, especially in the context of prior work that does not consider it (like LazyDagger and SafeDagger). ", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "aYCD0pCjwR", "reviewer": "Reviewer_pnHZ", "summary": "This paper presents ThriftyDAgger, an approach to reduce the burden on human operators during interactive imitation learning. To achieve this, the authors estimate both the novelty and the riskiness of states during execution---the latter being the most significant of their contributions---and only request human intervention when either prediction deems it necessary. The authors demonstrate good performance on a variety of simulated and real world task, achieving state-of-the art performance in both while simultaneously reducing the burden on human operators.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nOverall, this is a solid, well-written paper and (despite a somewhat narrow theoretical contribution) the experiments are impressive and reinforce the utility and broad applicability of the technique. Among the experiments, perhaps the most impressive are those included in Table 2: Three-Robot Fleet Control User Study Results. These results do an excellent job of showing how human operators can simultaneously train a small fleet of agents while also trying to perform a task of their own. These results show the improvements afforded by ThriftyDAgger and were immediately convincing.\n\nI have a number of comments and suggestions that, if addressed, would help improve the clarity of the work.\n\nFirst, while it is certainly advantageous that ThriftyDAgger achieves a lower average number of interventions (for this section, I focus on the results shown in Table 1), it seems that the LazyDAgger has a similar \"gating mechanism\" to reduce the amount of context switching---and therefore the number of interventions. What do the authors hypothesize is the mechanism that allows for a lower number of interventions for ThriftyDAgger? (Is it perhaps what the authors refer to as the \"automatic parameter tuning\"?) Relatedly, how should we think about direct comparison between the different approaches for this \"number of interventions\" metric, since I expect it depends quite heavily on the parameter selection? My intuition is that simultaneously achieving lower numbers of interventions and also higher success rates is the key takeaway, but some clarity on the subject from the authors would be helpful.\n\nSecond, I understand why the authors use the phrase \"Automatic Parameter Tuning\" to describe how the risk and novelty are evaluated, yet I am not sure that this is a particularly appropriate name for this process since the parameters for this process are (technically) not changing, and instead the process by which novelty and risk are defined depend on statistics of the data. Instead, it might be more transparent to present the approach in this way, as the risk and novelty online as data is collected. The title of Sec 4.4 could be something like \"Computing Risk and Novelty Thresholds from Data\": more verbose, but also a better reflection of what the system is doing. In the text, the authors can still claim that this method avoids the need for oft-sensitive parameter tuning, even if one might not regard it as a \"parameter tuning\" approach. I am open to debate on the subject.\n\nFinally, while I appreciate the definitions provided in Eqs (2) and (3), I am unsure that the definition in equation (3) comes up again, since this does not appear to be the optimization objective used to compute the expected policy. Could the authors clarify the relation between the process used to learn the policy and the objective defined in Eq. (3)? Similarly, while Eq. (2) is used to build intuition, the \"burden\" does appear to be directly calculated anywhere; this should be mentioned in Sec. 3.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a novel idea that is particularly well-executed. Though the central contribution of the paper is not overwhelmingly transformative, they have put in considerable effort to clearly communicate their contribution in relation to other work and empirically demonstrate its benefits.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "I enumerate my suggested changes and provide justification for them in the \"Strengths and Weaknesses\" section above. In summary, they are as follows:\n\n1. Inclusion of a clearer discussion of what is implied by the lower number of interventions for their technique and how those results should be interpreted to compare the different approaches.\n2. Clarifying the discussion of the \"Automatic Parameter Tuning\" (including potentially changing the name of the relevant section and idea).\n3. Some clearer language surrounding the theoretical motivation."}}, {"review_id": "MkCUuZJZ6tc", "reviewer": "Reviewer_UPFi", "summary": "This paper introduces another interactive learning method based on data aggregation, within the family of the DAgger algorithms. The method is able to control when there should be a switch between the autonomous control and the human teleoperation based on two different measures: uncertainty about the visited state, and probability of success which is computed similarly to a value function, assuming that there is a sparse reward only in a target state. The approach reduces the load of the teachers, since they don't have to keep full attention to the robot while it is learning, because the active learning scheme requests the new demonstrations according to the intervention criteria.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper is very easy to read, it is a pity not seeing the pseudocode in the main document, however, the text gives a clear explanation of its flow. The experiments cover well most of the questions that could come up about such a method. Below there are some comments with questions that can make the paper more complete.\n- All this family of DAgger methods that have been developed incrementally (EnsembleDAgger, SafeDAgger, HG-DAgger, LazyDAgger), work under the strong assumption that the user is perfect and never makes mistakes. What happens when the teacher makes wrong demonstrations often? how is the performance of the learning process affected?\n- Could it really be claimed that the novelty property is a contribution?\n- How does the method behave at the beginning, when most of the states do not have a high Q value due to the failures of the learning policy, but the budget of human requests is limited, does it go to the extreme cases?\n- How does the method deals with situations in which the human wants to intervene in order to change the current policy, but the robot is not switching to the teleoperation mode? It seems that a successful policy would not request feedback from the human, however, the method still should allow customizing the policies to the users' preferences.\nWhat happens in tasks wherein the objective is not easily defined by the convergence to a set of target states? For instance, a biped robot learning to keep balance and walk? \n- What kind of interface was used for the real robot experiment? what kind of considerations should be taken regarding the Human-Robot interface? These methods cannot be completely isolated from the human factors, therefore, these details are as important as the learning method implementation.\n- Authors should elaborate the idea in line 236. It is not completely clear why negative results are discarded.\n- The word \"Berkeley\" is really not needed to be mentioned, when describing the participants of the experiment. The review process is double-blind, however, this unnecessary information makes it clear what group the paper comes from. I really disliked this point.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The presentation of the method is clear, the experiments cover almost a complete evaluation of the proposed approach. There are many questions that should be approached, but I think the authors do not need to reconsider the method or the paper structure in order to address them.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See Strengths And Weaknesses", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for generating queries for supervision from humans in an imitation learning context. The primary idea is to consider novelty of a state and the associated “risk” (i.e., likelihood of hindered task progress) in order to minimize context switching for the human supervisor. The resulting approach (ThriftyDagger) is demonstrated on simulation and hardware experiments involving manipulation tasks. In addition, results of a human user study on using the approach for controlling a group of three simulated robots is presented.

Strengths:
+ The reviewers agree that the inclusion of the risk term in deciding when to make queries to the human supervisor is novel.
+ The reviewers generally agree that the experimental evaluation of the approach is very thorough and convincingly demonstrates improvements in comparison to various baselines approaches.
+ The paper is very clearly written and the technical exposition is crisp. 
+ The approach only requires choosing one hyperparameter (the context switching rate); other parameters are chosen based on this. 

Weaknesses:
- The primary technical novelty of the approach is in the inclusion of the risk term. The reviewers request more justification and intuition for why this term helps the empirical performance (see the comments from Reviewer pnHZ and PRtc for more specific points to discuss).
- Reviewer PRtc noted that demonstrating the approach on a more challenging simulation task could be beneficial (especially since the performance gap between the proposed approach and baselines is smaller on the hardware experiments).

Suggestions:
I urge the authors to consider the reviewers’ detailed feedback in order to further improve the paper. The primary suggestion for improvement is to add more discussion and intuition on the benefits of including the risk term (which is the primary technical novelty in the approach). The reviewers also requested a few additional details and explanations, and suggested changing the phrase “automatic parameter tuning” to something more representative (e.g., “computing risk and novelty thresholds from data”).

----- Post rebuttal -----

The authors have addressed the reviewers' primary concerns during the rebuttal phase. In particular, the authors have (i) added intuition on why the risk term helps,  (ii) added a baseline (picking and stacking), and (iii) provided further details and clarifications. The revised paper is stronger as a result, and the reviewers are in agreement that the paper constitutes a strong contribution and should be accepted. 

Decision: Accept (Oral)