Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
Authors: Haoyuan Cai
Zhenghao Peng
Bolei Zhou
Venue: ICML 2025 poster (2025)
Keywords: Imitation Learning
Human-in-the-loop Reinforcement Learning
Shared Autonomy
Primary area: reinforcement_learning

Abstract: Interactive Imitation Learning (IIL) allows agents to acquire desired behaviors through human interventions, but current methods impose high cognitive demands on human supervisors. We propose the Adaptive Intervention Mechanism (AIM), a novel robot-gated IIL algorithm that learns an adaptive criterion for requesting human demonstrations. AIM utilizes a proxy Q-function to mimic the human intervention rule and adjusts intervention requests based on the alignment between agent and human actions. By assigning high Q-values when the agent deviates from the expert and decreasing these values as the agent becomes proficient, the proxy Q-function enables the agent to assess the real-time alignment with the expert and request assistance when needed. Our expert-in-the-loop experiments reveal that AIM significantly reduces expert monitoring efforts in both continuous and discrete control tasks. Compared to the uncertainty-based baseline Thrifty-DAgger, our method achieves a 40% improvement in terms of human take-over cost and learning efficiency.
Furthermore, AIM effectively identifies safety-critical states for expert assistance, thereby collecting higher-quality expert demonstrations and reducing overall expert data and environment interactions needed. Code and demo video are available at https://github.com/metadriverse/AIM.

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
Haoyuan Cai
Zhenghao Peng'
Bolei Zhou'
Abstract
Robot-Gated Interactive Imitation Learning
Interactive Imitation Learning (IIL) allows agents acquire desired behaviors through human in- terventions, but current methods impose high cognitive demands on human supervisors. We propose the Adaptive Intervention Mechanism (AIM), a novel robot-gated III. algorithm that learns an adaptive criterion for requesting hu- man demonstrations. AIM utilizes a proxy Q- function to mimic the human intervention rule and adjusts intervention requests based the alignment between agent and human actions. By assigning high Q-values when the agent deviates from the expert and decreasing these values as the agent becomes proficient, the proxy Q-function enables the agent to assess the real-time align- ment with the expert and request assistance when needed. Our expert-in- expert-in-the-looy experiments veal that AIM significantly reduces expert moni- toring efforts in both continuous and discrete con- trol tasks. Compared to the uncertainty-basse baseline Thrifty-DAgget our method achieves a 40% improvement terms of human take-over and learning efficiency. Furthermore, AIM effectively identifies safety-critical states for ex- pert assistance, thereby collecting higher-quality expert demonstrations and reducing overall ex- pert data and environment interactions needed. Code and demo video are available at ht attps: //githull com/co/////////oo adwiaerse/eeeieeeeeee
O
can have a rest.
can handle it.
can help you.
Can you help me?
Figure I. In robot-gated IIL. the agent explores the environment and requests human assistance based on intervention criterion. The expert is not involved in learning until the agent requests help.
manipulation tasks (Finn et al., 2017: Fang et al., 2019; Ganapathi al., 2021). However, RL often suffers from poor sample efficiency during policy learning, and the re- ward function struggles to perfectly capture human pref- erences, causing learned policies to produce unintended behaviors. To address these limitations, researchers have turned to Imitation Learning (II.), which leverages offline expert demonstrations and aims to match human behavior on this offline dataset (Osa al., 2018). Nevertheless, IL is susceptible distribution shifts and nut-rf-distribution (OOD) states, where the agent encounters scenarios not cov- ered by the training data (Ross et al., 2011; 1; Ravichandar et al., 2020; Chernova & Thomaz, 2022; Zare et al., 2024).
1. Introduction
Interactive Imitation Learning (IIL) incorporates human participants to intervene the training process and pro- vide online demonstrations, which helps improve alignment and learning efficiency (Reddy al., 2018; Kelly al., 2019; Spencer et al., 2020; Peng al., 2024; Seraj et al., 2024). There are two different categories of human in- volvement: duman--gtee intervention (Kelly et al., 2019; Li et al., 2022b; Peng et al., 2024) and robot-gated involve- ment (Menda et al., 2019; Hoque et al., 2021a). Human- gated III. leverages human abilities to identify the agent's
Reinforcement learning (RL) has been successful wide variety of tasks, from playing Atari games (Mnih et al., 2013) to autonomous driving (Li et al., 2022a) and robotic
'Computer Science Department, University of Califor- nia, Los Angeles (UCLA), Correspondence to: Bolei Zhou <bolei@cs.ucl..duu>>
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
mistakes, predict future trajectories, and adjust the inter- vention based on the agent's performance. However, this requires humans to continuously monitor the whole train- ing process and immediately intervene in safety-critical states (Peng et al., 2024; 2025). Monitoring the learning process imposes a significant burden on the human supervi- sor. In contrast, robot-gated IIL allows the agent to au- tonomously request assistance based on an intervention criterion, reducing the cognitive load on human supervi- sors. Prior works on robot-gated IIL use uncertainty-basee or preference-based intervention criteria, which may fail to align with human intents in deciding whether to inter- vene (Hoque et al., 2021a; Menda et al., 2019; Hejna et al., 2023). Common uncertainty-bant -based methods (Menda et al., 2019; Kelly et al., 2019) request human help when the un- certainty estimate is larger than a fixed threshold. Without adjusting the threshold adaptively, the agent keeps request- ing expert help even when it has successfully imitated the human expert. Furthermore, these methods require tun- ing multiple hyperparameters for each task and training an ensemble of policy networks to compute action variance, which harms the learning efficiency (Hoque et al., 2021a).
action difference from the expert automatically re- duce the expert intervention rate the agent's s policy converges toward expert behavior.
2.
We evaluate our algorithm in MetaDrive and Mini- Grid environments, demonstrating that AIM requires fewer expert demonstrations and monitoring efforts to achieve near-optimal policies.
Experiments show that AIM requires fewer environ- ment interactions and expert data to handle the safety- critical states. The expert demonstrations requested by AIM contain corrective actions in safety-critical states so that they can assist a novice agent in imitating the expert's 's policy.
2. Related Work
Human-Gated Interactive Imitation Learning Human- gated IIL algorithms rely on human experts to proactively be involved in the training loop and provide corrective actions at dangerous or repetitive states. Human-Gated DAgger (HG-DAgger) (Kelly et al., 2019) and Intervention Weighted Regression (TWR) (Mandlekar et al., 2020) per- form imitation learning on human intervention data. These methods do not leverage data collected by agents or limit the human intervention frequency, harming the sample ef- ficiency. EGPO (Peng et al., 2021) and PVP (Peng et al., 2024; 2025) design proxy cost or value functions to sup- press the frequency of human involvement. A recent line of human-gated IIL methods, including CEILING (Celemin & Ruiz-del Solar, 2019) and RLIF (Luo et al., 2023) re- quires humans manually assign different weights or pref- erence signals in addition corrective actions. Still, they require humans to monitor the screen during the entire train- ing process and stay aware of the agent's potential mis- takes. Although these human evaluative feedbacks reduce the demonstrations needed, providing these signals can be time-consuming and thus incur more human involvement.
To solve these issues, we propose the Adaptive Intervention Mechanism (AIM), a novel IIL algorithm that learns robot- gated intervention criterion resembling human-ga -ated inter- vention mechanisms. First, AIM recovers the underlying intervention mechanism that aligns with human intentions, ensuring that the interventions are critical and effective. By training a proxy Q-function that approximates the human's 's decisions in intervention, AIM proactively requests human assistance when the agent's 's policy deviates from the expert. Second, as the agent's 's policy converges toward the expert's s, the proxy Q-values decrease, which automatically reduces the intervention rate over time. In this way, AIM adaptively shrinks its requests for help as proficiency improves, without any hand-tuned schedule (see Fig. 8). Compared with robot- gated IIL baselines, AIM effectively seeks human guidance in safety-criticn states. Third, AIM eliminates the need for computationally intensive uncertainty quantification by utilizing a proxy value function loss, enhancing training efficiency and scalability. We evaluate our method AIM in MetaDrive (Li et al., 2022a) and MiniGrid (Chevalier- Boisvert et al., 2018), showing that AIM achieves a higher learning efficiency compared to various baselines in both continuous and discrete action spaces.
Robot-Gated Interactive Imitation Learning Robot- gated III. methods rely on intervention mechanism, and the key is to lift the burdens on human experts while ob- taining sufficient information for the training of the expert policy. Existing robot- -ated methods include Ensemble- DAgger (Menda et al., 2019) and Thrifty-DAgger (Hoque et al., 2021a), which utilize the variance of actions as the uncertainty estimates and assume that the expert should in- tervene in the states with high uncertainty. However, such heuristic criteria may not align with human intervention strategies, and the uncertainty estimates of dangerous states may fluctuate during training. In addition, prior robot-gated algorithms require hyper-parameter tuning based on the tasks (Hoque et al., 2021b; Biré et al., 2024) or manually specify a desired intervention rate (Zhang Cho, 2016;
We summarize our contributions as follows:
1. We propose the Adaptive Intervention Mechanism (AIM), an Interactive Imitation Learning algorithm that proactively requests expert intervention during en- vironment interaction. AIM learss a proxy Q-function to adaptively capture states where the agent's 's actions diverge from the expert's. s. AIM leverages the real-tin eele
2

---

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
Hoque et al., 2021a), which further aggravates the expert's burden and fails to adjust the intervention criterion as the agent becomes proficient. Confidence-basee intervention employs task-specific and hard-coded confidence models to estimate the familiarity of current states, such as the nearest neighbor distance to the states in the training data (Cher- nova & Veloso, 2009; Saeidi et al., 2018). While intuitive, these confidence models struggle in high-dimensional ob- servation spaces and lack generalization across different tasks. In contrast, AIM trains proxy Q-function that di- rectly captures human intervention decisions and reduces the intervention frequency as the agent becomes proficient, eliminating the need to manually set hyperparameters. By relying on actual intervention data rather than heuristic con- fidence measures, AIM also generalizes to high- -iteenimensii observations without requiring task-specific engineering.
In robot-gated III, the agent has access the switch-to- human function [Tss, ar) when the agent interacts with the environment itself without access to ah- Once the human expert intervenes at state the agent uses an- other continue-with-humne function 1'(s, ar, as) to decide whether to stop requesting human help. The first switch- to-human function (s a,) : S x A - {0, 1] decides whether to ask the human expert for help when the agent takes action ar state The agent uses the second function (s, dr (n)) : AxAxA + (0, } to decide whether to continue requesting expert help. When (s, a, = O, the agent will inform the expert to stop providing demonstrations and will not request hu- man assistance until (s (s'))) = again at some future state s'.
4. Method
3. Preliminaries
In this section, we first provide a motivating example in Sec. 4.1 to show the drawbacks of previous uncertainty- based robot-gated Interactive Imitation Learning (IIL) meth- ods. Then, we introduce our approach of learning an adap- tive intervention mechanism in Sec. 4.2 that emulates how humans teach an evolving agent policy. We provide our algorithm pipeline in Sec. 4.3.
In this section, we introduce our settings of interac- tive imitation learning environments. The robot environ- ment is modeled by a Markov Decision Process (MDP) M = AS P, 7,,77 do) with a state space S, an action space A. state transition function P S x A - 8, reward function r, a discount factor 7 and an ini- tial state distribution dg. For any agent policy (a 8) : 5 A + 0, 1]. the expected cumulative return is ((x,)= = E,-r. EEi 7'r(s,, a:)]. where Pz, is the dis- tribution of trajectories T = (s0, a0, $1, 01 a.)...) induced by Tr. dg and P. The reinforcement learning objective is to learn agent policy (a | s) which maximizes JTZ When nr, is deterministic, we denote the function (s) that outputs robot action a,, at state s. In this paper, we consider the reward-free setting in which the agent has no access to the task reward function r(s,
4.1. Motivating Example
Human Human-gated IIL algorithms like HG-DAgger (Kelly et al., 2019) and PVP (Peng al., 2024) require humans to moni- tor the entire training loop and correct the agent's 's actions at safety-critical states or when the agent encounters difficul- ties discovering the optimal strategy, as is shown in Alg. 1. To alleviate the burden on human supervisors by eliminat- ing the need for continuous monitoring, we aim to design a robot-gated criterion I'( a,) to proactively request human assistance in a manner that imitates humans' intervention mechanism [e a,, 4x) in Alg. 1. Prior works of robot- gated IIL, including Ensemble- -Dgger (Menda et al., 2019) and Thrifty-DAgged (Hoque et al., 2021 assume that the agent makes frequent mistakes at novel states and is more proficient at handling frequently encountered states. There- fore, these methods employ heuristic uncertainty estimates to identify novel states and trigger human intervention if the uncertainty of current states exceeds a pre-defined threshold.
In imitation learning, the robot learns from human behay- iors by imitating the human policy Th. s). Traditional imitation learning algorithms learn from human expert tra- jectories The ~ P which may lead to poor performance due to out-of-distribution states (Ross et al., 2011).
Human- Human-gated interactive imitation learning incorporates an interaction mechanism in which the expert applics an inter- vention criterion 7cXp (s, dr; an) : S A x A -> {0, 1) to decide whether take control when the agent outputs action a, at state s. The behavior policy that generates action in the training process follows
In Fig. 2, we illustrate the drawbacks of uncertainty-basee methods by showing the mismatch between the uncertainty estimates and the human-gated intervention mechanism. We use the MetaDrive (Li et al., 2022a) and illustrate the cases when the human expert intervenes from two trajecto- ries. The y axis of the uncertainty estimate is Var(a,) E, where Var(a,) is the variance of agent actions and e is the switch-to-humal threshold in Ensemble-DAgee (Menda et al., 2019), i.c., human demonstration is requested when
Ta(als) ( (1 (s) ur an) )ou a.)
(1)
+ 1 (T
Here, the ó function in Eq. 1 denotes the Dirac delta dis- tribution, representing a deterministic action a,. In Eq. 1, (s, ay, an) works as a human gated intervention crite- rion.
3

---

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
Pratice Mep
Algorithm HHman--Gated Interactive Imitation Learning
for trajectory i =1, 2, ..2....d. do
for timestep = 2......dd do
Taceover: YES
Ialecver Ratw:46. -5.5%% Stape:155
Agent samples action or (st)
Human observes (8t, a,) and decides whether to issue an intervention (st ar: aA).
if Texp
Texp (St, ar, (p) = then Human samples a/ ~ TTy, (sa).
Kerarap
Observe 8e+1 P( St, (1)).
Uicesiity
Add (st, ah, S(+1) to the human buffer Bi-
Tabenver::00 85%
else
Doldee CIANN
Totul Stupu:348. 8481
Agent takes control.
Observe 84+1 ~ P(: St, ar).
Add (8t, Or, 5c+1) 1) to the novice buffer B,..
Figure 2. Comparison between the uncertainty estimate and our Adaptive Intervention Mechanism (AIM) interactive imitation learning under MetaDrive safety environments. We plot the human intervention behavior, the corresponding uncertainty estimate, and AIM's Q-value when supervising novice agent (upper) and pro- ficient agent (lower). step B, the front car makes sudden right turn and the human believes intervention is needed. At step D, the expert believes the agent is proficient enough to navigate this curve and avoid collision. This highlights the limitations of fixed uncertainty thresholds and the need for an adaptive intervention mechanism adjusting to the agent's improving performance.
end if Train ग. on B₁, via imitation learning. end for end for Output: Tr.
following the same continue-with-human function
f(a,, an) I[lla, - -azll² 2 > c).
Var(a,) > c. Similarly, we plot (s, - in Fig. 2 to visualize when the AIM agent requests human help, where (&,s, a,) AIM's proxy Q function defined in Sec. 4.2 and is a threshold defined in Eq. 6. The curves are smoothed using a 20-step running average.
then the agent can recover the expert policy with a per- formance guarantee when e is small enough (Peng al., 2021; 2024). Following these works, set [' a,, ax) I (ar,aa)) i.e., the agent stops requesting human for help if |a, axll <e
In steps B and D, the uncertainty estimates fail to align with human intervention decisions: the safety violation at step B is not predicted by the uncertainty estimate, while at step D the uncertainty estimate requests human assistance even when the agent behaves correctly. The uncertainty estimates with a pre-defined threshold limit robot-gated IIL's udaptability to the evolving agent policy. This fact highlights the necessity of adaptive behavior in the design of robot- gated intervention mechanisms. Fig. 8 in the Appendix further contrasts AIM with the uncertainty-baser baseline across different training stages. AIM quickly and precisely identifies states where the agent is already competent, while the uncertainty-basse based method keeps querying the expert at the same locations, wasting intervention effort.
4.2. Training the Adaptive Intervention Mechanism
We propose the Adaptive Intervention Mechanism algorithm (AIM), which can recover the humans' underlying interven- tion criterion and automatically adjust the intervention rate by capturing the evolving nature of the agent policy.
The key idea of AIM is to use a proxy Q-function Q/ )y) to approximate the human-gated intervention mechanism: the higher Q (s, a,) is, the more likely human would inter- vene. Given set of human demonstrations B3 = {(s, an)} and the current agent policy πy, we label the Q value of Q (s, o,)) ) as - for any (s, (;) sampled from Br, i.e., when the agent takes exactly the action a, at state s, the human participant unlikely to intervene as the agent aligns with the human policy. addition, we sample a, from the cur- rent agent policy πy and label the Q value of Q8(s,, a,) as +1 if ||a,- an 1| > 6. The expert will likely provide corrective demonstrations if the agent behaves differently, which conforms to Eq. in Sec. 4.1. This learning objec- tive is achieved by fitting the proxy Q-function Q with the following AIM loss:
Another problem in robot-gated IIL is how to stop hu- man intervention. Previous works utilize the action differ- ence ||a, 12 as the criterion to determine intervention- stopping function ((rr.a)) = I[]la, 12 > e). where e is a pre-defined parameter (Menda et al., 2019; Hoque et al., 20)]a;b). In addition, some works show that if a hy- pothetical human expert intervenes and leaves intervention

---

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
Algorithm 2 Adaptive Intervention Mechanism (AIM)
JAIM (0) =
1006 (s, an) + +1(²]
Input:
Hyperparameter 8.
E [(ar, an) 106 (z) - -1²² (s.as) )B
Run human-gate -attaann-tteddaa IIL (Alg. 1) for trajectories. Initialize the proxy Q-function Q with J(0) in Eq. 5. Initialize the switch-to-human threshold 3 by Eq. 6.
+ (maa))
(3)
Initialize the switch-to-agent agent threshold by Eq. 8. for trajectory = 1, 2, do for timestep = 1.2...... do
The proposed AIM loss function offers significant ad- vantages over the uncertainty-based intervention criterion in Ensemble-DAgger (Menda et al., 2019) and Thrifty- DAgger (Hoque et al., 2021a) by ensuring adaptability to the evolving agent policy. Uncertainty estimates based on the variance of actions make the agent reluctant to seek humans for help at frequently encountered states, even when the agent behaves differently with humans. In con- trast, our proxy Q-function (5, as) remains responsive to the changing agent policy. Specifically, the Q-value of (s,s, a,) approaches +1 continuously as long as a, devi- ates from dn at any state 8 in B₁. This design motivates the agent to request human demonstrations it keeps making mistakes, mitigating the optimality gaps.
Agent samples action or TT if Q (s, a,) > 8 then
Human takes action On ~ TT (st). repeat
Observe 8441 ~ P(. St, an).
Add (st, an 814-) to the human buffer B,.
Train TT. on B₁ with the loss function Eq. t 1+1. 1.
Agent samples action a, (st). Human takes action an ~ TT (tt).. until ||,,- - an Se.
end
Observe 56-1 ~ P(- 8t; a,,)
Furthermore, the intervention rate of AIM also adapts to the agent's 's performance. As ग., becomes increasingly aligned with the human policy Th, in Eq. 3, a growing number of states in B, will satisfy [ar, ox) = O. This leads the average Q-value Q ar) to decrease towards - 1, shrink- ing the intervention rate smoothly, making the agent less likely to request expert help in states that it has already mastered. The gradually decreasing intervention rate aligns with the human intervention mechanism in Fig. 2, where the expert intervenes less frequently as the agent demonstrates proficiency.
Add (s,,ar, ar 8(+1) 1) to the novice buffer B,. Train Qg with the loss function (0) in Eq. 5.
Update the wwitch-to-huma threshold /3 by Eq. 6.
end for end for Output: Policy ग. and proxy Q-function Q/-
act with the agent in the first n trajectories and provide cor- rective demonstrations through active interventions follow- ing Alg. 1. The human demonstrations B;, = {(s, an s'}} and the agent's self-exploration data B,- = {(s, ar, s')} collected in this process are used to train an initial proxy Q-value Q/ for robot-gated intervention. Then, we need to set a switch-to-human threshold 33 such that the agent requests humans' help if (s, parameter ā E (0, 1) and define 33 as the (1 5)-th quantile of (s, a,) where is sampled from the novice buffer B, and 0r ~ T (s). We denote
In addition to the AIM loss, we incorporate a Temporal Difference (TD) loss to propagate the proxy Q-value to the agent-collected data during self-exploration (Peng et al., 2024). This propagation generalizes our proxy Q-function Q the states where humans have not been intervened yet. We define the TD loss function
B. We set hyper-
E (T)(0) (",a,")) (o(s..) max a'
(a(s',a')
27
(4)
B quantile
(Qb(s, 5),
(6)
where Q3 represents a target network with update delay. Integrating the TD loss allows AIM to anticipate potential agent mistakes in the future and request human assistance before they occur, while prior incertainty-based approaches only estimate the action variance at the current state. The final loss of AIM is as follows:
I" a,), = (0) > 3).
where (s a,)) is the switch-to-human function. Following prior works robot-gated IIL (Hoque et al., 2021a:b), when the expert is currently intervening at state the agent stops requesting human intervention from the next step if l|a,
J(Ø) = AIM (0) + (A).
(5)
1 < 6, where
E (s.an J- a m,, (s)
[llar- - an 12]
(8)
4.3. Algorithm
We outline the workflow of the Adaptive Intervention Mech- anism (AIM) in Alg. 2. First, we let the human expert inter-
We set the continue- continue-with--iee with-umman function as 1' ar, an) T[laa, - a, 1| > e) following Eq. 2.
=
5

---

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
Then, we start the robot-gated interactive imitation learning process. The agent requests human assistance if (s, ar) 1, and resumes autonomous control if (s, dr, (b) O. After each human demonstration, AIM updates the switch- to-human threshold /3 following Eq. 6, the agent policy Tyy and the proxy Q-function Q/ following Eq. 5.
Take Over - Explorr
AIM (Ours)
Thrifty-DAgger
Note that AIM is compatible with discrete action spaces, as we can set the switch-to-human function as f(a,, an) = Iaa # anl- Compared with human-gated IIL. methods, AIM only requires humans to proactively provide human- gated corrective demonstrations in the imitial one or two trajectories (n I 2), significantly reducing the cognitive load of human in later training. In addition, in AIM and all the baselines of interactive imitation learning, we train the agent policy nr by imitation learning on B3 as in HG- DAgger (Kelly et al., 2019) to compare their intervention criteria fairly, and the loss function of nr follows
Figure 3. Comparison of intervention strategies between AIM and Thrifty-DAgger in MetaDDive after 2K training steps. Arrows indicate steering actions in a top-down view. AIM only requests expert help near traffic cones and roadblocks, whereas Thrifty- DAgger frequently requests takeovers even on straight roads. See Appendix A for details.
that require human- interventions, the neural expert follows Eq. 2 to provide corrective demonstrations the action-difference function [(a,, an) 1. During the evalu- ation, we report the average of all the metrics obtained by the learned agent alone in 50 rollouts without interaction with the neural experts.
((TT) =
(8.0%)~B, [s) 111
E
(9)
5. Experiments
In this section, conduct experiments to answer the fol- lowing questions: (1) Does our algorithm require fewer expert demonstrations and efforts to learn a near-optimal policy than other interactive imitation learning methods? (2) Does the learned intervention criterion help the agent receive sufficient human guidance at safety-critical states, thereby capturing all necessary information for effectively imitating the expert? To investigate these questions, con- duct experiments various reinforcement learning tasks with different state and action spaces.
5.2. Experimental Setting
Evaluation Metries We define the expert-involved steps as the total environment steps that require the expert to mon- itor, a measure of the expert's 's effort in guiding the agent during training. We limit the total expert-involved steps to transi-
2000 and report the total number of tions (expert data usage) and the overall intervention rate, which is the ratio of expert data usage to the total data usage. In MiniGrid, report the success rate the agents' evalu- ation metric. The success rate is the ratio of episodes where the agent reaches the goal to the total evaluation rollouts. In MetaDrive, we also report the episodic return and route completion rate during evaluation. The route completion rate the ratio of the agent's 's successfully traveled distance to the length of the complete route.
5.1. Tasks
We consider the MetaDrive driving experiments (Li et al., 2022a) with continuous action spaces and MiniGrid Four Room task Shhevalier-Boisverr et al., 2018) () with discrete action spaces. In MetaDrive, the agent needs to navigate towards the destination in heavy-traffic scenes without crash- ing into obstacles and other vehieles. The agent uses the sensory state vector s 259 as its observation and outputs control signal a = (ao, a1) E [[1, 1|² representing the steering angle and the acceleration, respectively. We evalu- ate the agent's learned policy in a held-out test environment separate from the training environments. In MiniGrid, the agent observes its local neighborhood and learns a naviga- tion policy to open each door on its way to the goal.
Experimental Details The neural expert is trained using PPO-Lagrangian (Ray et al., 2019) with 20M enviromment steps. We train each interactive imitation learning bascline five times using distinct random seeds. Then, we roll out 50 trajectories generated by each model in the held-out evalua- tion environment and average each evaluation metric as the model's performance. We report the average performance of the best checkpoints from the five random seeds as the result of each baseline. We also provide the standard devi- ation of each metric among the five runs of each bascline. In AIM, we set 5 = 0.05. For fairness, we introduce the same warm-up stage to AIM and each baseline by letting the expert monitor the initial n trajectories.
Experiments involving real human participants time- consuming, and their interaction results vary significantly in different trials. Following the prior works on interactive imitation learning (Hejna et al., 2023; Peng et al., 2021), we incorporate well-trained neural policies in the training loop to approximate human policies. In the initial a trajectories
6

---

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
Table 1. Comparison of methads with training/testing statistics in the MetaDrive environment with 2000 expert-involved steps. The overall intervention rate is given besides the expert data usage. We report the average performance and standard deviation of the best checkpoints from five random seeds.
Training
Testing
Method
Robot-Gated
Expert Data Usage
Total Data Usage
Success Rate
Episodic Return
Route Completion
Neural Expert BC
0.84 0os
336.5 + ma
0,93 +:000
2K 0.9K (0.45)
2K
0.33 D004
243.0 + 467
0.62 + 0.03
HG-DAgger
2K
0,61 h07
310.8 167
0.78 +0007
PVP
0.4K (0.19)
2K
0.62
270.4
0.77
Ensemble- Ensemble-DAgger
2K (0.55)
3.6K
0,60
267.4 9.90
0.54 0.10
Thrifty-DAgger
2K (0.21)
9.5K
0.58 0.08
250.0
0.73 +
AIM (Ours)
1.9K (0.24)
7.7K
0.82 it 0.06
328.4 i: 20.4
0.91 0013
Thilty Oigger Ensembie Oagger PvP APE OOus
Tivilty Degger
Ensembre Orgger
(Ouns)
Ensemble Dagger
Thrity Orgger
PXP
(0uns)
300
250
150
320
Eaper-tnuolant Steps
500
14
2.3k
2.5k
150
15k
2.5k
500
18
1.kk
2k
2.5
(a) Evaluation Success Rate
(b) Evaluation Crash Rate
(c) Evaluation Mcan Reward
Figure 4. Training curves of Interactive Imitation Learning (III.) approaches in MetaDrive. We report the evaluation metrics of the corresponding policy in a held-out test environment, where evaluation is conducted after expert-in inattiin-the--oog training without expert involvement. We apply running average with window length of 40 smoothen the curves. AIM imitates the expert's behavior with fewer eewerexpert-inee expert-involved steps than all the baselines, achieving a higher success rate and episodic return, as well as a lower safety cost.
Baselines We test these robot-gated interactive imitation learning baselines: Behavioral Cloning (BC), Ensemble- DAgger (Menda et al., 2019), and Thrifty-DAgeer (Hoque et al., 2021a). We also evaluate the human-gated interactive imitation learning methods: Human-Gatee DAgger (HG- DAgger) (Kelly et al., 2019) and Proxy Value Propagation (PVP) (Peng et al., 2024). Fig. 4 plots each method's s evalua- tion metrics against the number of expert- involved steps. For the human-ge gated PVP baseline, the expert-invole involled steps correspond to the total data usage because the expert moni- tors the entire training process. For AIM and the robot-gated baselines. the expert-involved steps correspond to the expert data usage. We compare the performance of each method under the same 2K expert-involved-see step budget in Table and Table 2.
petitors while markedly reducing expert-involved steps (see Fig. 4), thereby easing the expert's monitoring effort.
We also compare the performance of AIM with other IIL baselines in the MiniGrid Four Room task (Chevalier- Boisvert al., 2018) with discrete action spaces. From Table 2, we can conclude that AIM also outperforms these baselines and alleviates the burdens on human supervisors in environments with discrete action spaces.
5.4. A Case Study in a Toy MetaDrive Environment
Fig. 3 visualizes the intervention strategies of AIM and Thrifty-DAges after 2K training steps in a simplified MetaDrive setting. The toy scenario consists of a straight road flanked by a set of traffic cones and a single roadblock.
From Fig. 3, AIM requests expert help exclusively near the traffic cones and the roadblock while remaining autonomous in the unobstructed areas. Such spatially focused queries reflect efficient intervention strategy: the agent solicits demonstrations precisely where data are scarce and errors are most likely. In contrast, Thrifty-DAgger issues queries along the entire obstacle corridor, including many states the agent has already encountered frequently, thereby involving unnecessary expert effort.
5.3. Baseline Comparison
In Table 1, we compare the performance of our proposed AIM algorithm against several baseline methods in the MetaDrive simulator (Li et al., 2022a). Our algorithm AIM achieves a higher success rate and episodic return com- pared with all the robot-gated IIL baselines under the same amount of expert demonstrations. Across both robot-gated and human-gated IIL baselines, AIM outperforms all com-
7

---

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
Table Comparison of methods with training/testing statistics in the Minigrid Four Room environment with no more than 2000 expert-
involved steps. The overall intervention rate is given besides the expert data usage.
Training
Testing
Method
Robot-Gated
Expert Data Usage
Total Data Usage
Success Rate
Neural Expert
0.78 0:0
BC
2K
2K
0.01 1:00
HG-DAgger
0.2K (0.12)
2K
0.20 004
PVP
0.2K (0.12)
2K
0.34 0.10
Disemble-DAgeee Thrifty-DAgger
2K (0.36)
5.6K
0.38 4: 0.008
2K (0.27)
7.4K
0.42 0.12
AIM (Ours)
0.4K (0.09)
4.4K
0.63 0.05
The Bggge
PVP AI (Dung)
Thithy-iigger
Fasentie-BAgge
PSP AMM (Oum)
Invemble-lgge Tithyyoggee PVP AMM (Ound)
07
08
35
-34
04
45
03
46
Usage
2k: 4k
1k
24
5k
10k
(a) Deviation Ratio in Safety- Critical States
(b) Expert Intervention Rate in Training
(c) Average Qb(s, a-) when intervenes
Figure 5. Training statistics of III. approaches in MetaDrive (lower is hetter). (a) We define the deviation ratio as the fraction of safety-critical states in which the agent's action deviates from the expert'. AIM drives faster decline than Engemb and Thrifty-DAgger matching the ouman-gated PVP. (b) AIM and PVP require comparable number of expert demonstrations, and AIM's s intervention frequency stays below MesembleeDAgger and Thrifty-DAgger during the first 4K steps. (c) AIM aligns agent actions with human behavior more rapidly than Ensemble-DAgger and Thrifty-DAgger.
Fig. 8 extends this comparison across multiple training stages. It shows that AIM progressively reduces its query frequency as the agent masters each corner case, whereas Thrifty-DAgger continues to request assistance repeatedly in the same safety-critical regions.
and Thrifty-DAgge (Hoque al., 2021a), AIM sustains markedly lower deviation ratio throughout training, showing that it aligns with the expert far more quickly, especially in truly hazardous situations where a sudden turn or an immediate brake is required. This trend reveals that fixed- threshold uncertainty heuristics under-query the expert when safety is at risk, allowing errors persist and wasting en- vironment samples. By contrast, the proxy Q-function in AIM triggers assistance precisely when needed, producing sharper decline in the deviation ratio. Furthermore, after 5K environment steps, the AIM and PVP curves nearly co- incide, indicating that AIM attains the expert-level behavior of the human-gated method without requiring continuous human monitoring. In Fig. 5(b) we present the overall ex- pert intervention rate, complemeninng the deviation analysis above. Surprisingly, AIM attains its superior alignment with the expert while requesting fewer interventions than both Ensemble- and Marifty-DAgge
5.5. Effectiveness of Our Intervention Strategy
In this section, we address the second question: Does the learned intervention criterion help the agent receive suffi- cient human guidance at safety-critical states, thereby cap- turing all necessary information for effectively imitating the expert?
To better investigate this problem, we define safety-critical state in MetaDrive as one where the PPO expert's ac- steer
tion as = (ak , 4% d with its 2-norm ||ax||2 > 0.5, ie., a sudden brake or a sharp turn. Recall that we are working with a PPO-Lagrangian expert therefore we can query its actions at all states, even though the intervention might not be active. Within these safety- -riiticlli states, we label the state where the agent action deviates from the ex- pert, i.e. f(ar, ax) 1, as deviation state. In Fig. 5(a), we plot the ratio between the number of deviation states and the safety-critical states. Compared with other robot- gated III. baselines Ensemble-DAgger (Menda et al., 2019)
In addition, we compare how quickly each baseline aligns the agent's 's actions with expert intent in Fig. 5(c). To this end, we train an AIM proxy Q-function Q6 using Eq. 5 for each baseline to measure how much state-action pair (s, a,))) deviates from expert behavior. We recall that high Q-value implies that the agent's 's action violates expert in- tent. Fig. 5(c) indicates that Ensemble-DAgger and Thrifty-
8

---

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
Performance of on Agent-Requested Expert Data
Ensemble-DAge -Agger - ThriftyDAgge
PVP AIM (Ours)
Table 3. Ablation studies of AIM with testing statistics in the
0.6
MetaDrive environment with 2000 expert-involved steps.
Success Rate
Testing
Method
0.5
Success Rate
Episodic Return
Route Completion
AIM reward
0.61
275.1
0.82
0.4
AIM no TD loss
0.57
246.7
0.75
AIM (Ours)
0.82
328.4
0.91
0.3
0.2
Expert Data Usage
0.1
interventions. By directing queries to states with the greatest agent-expert divergence, AIM ensures that each requested demonstration delivers high informational value, thereby facilitating more efficient policy learning without the need for continuous expert monitoring.
600
800
1k
1.2k
1.4k
1.6k
1.8k
Figure 6. We plot the success rates of the agent trained by Behav- ioral Cloning using the human buffer collected at different training stages. This offline-trained agent's performance reflects the quality of the expert data collected by each method. The near overlap between the AIM and human-gate gated PVP curves shows that the demonstrations AIM requests are at least as informative as those gathered under continuous human supervision.
5.6. Ablation Studies of AIM
In Table 3, we perform the ablation studies of AIM in the MetaDrive environment. "AIM reward" labels the reward function r(8, an) as +1 and r(s, ar) as -1 at any interven- tion instead of labeling the Q function. "AIM- w.o. TD loss" disables TD learning by setting (T) = O, so the proxy Q-function is trained solely with the supervised labels from the human buffer Bn. Table shows that dropping the TD loss or replacing the Q-labeling with reward-labeling hurts the performance of AIM.
DAgger require more environment data before their agents' actions conform to expert behavior. The three subfigures in Fig. also demonstrate that AIM and human- -atan PVP share similar takeover mechanisms, but we note that PVP achieves it at the cost of continuous expert oversight.
In the next experiment, we investigate if AIM requests fewer human demonstrations to capture all the information needed for successful imitation. We will show that AIM-requested expert data contains rich expert guidance information in critical states, allowing us to train a new agent policy from scratch by imitation learning on this offline dataset without any environmental interaction. The experiment works as follows. After training AIM, we extract the human replay buffer BR- We denote 13% as the first T state-action pairs in B,, where T < 2000. Then, we discard all the expert demonstrations collected in the initial warm-up stage from each BY so that BYJ only contains the expert demonstrations requested by the agent's intervention criterion. We utilize the dataset BY to train an agent policy .ं. from scratch using imitation learning with the loss function Eq. 9. We then plot the success rate of T. w.r.t. T in Fig. 6. The results show that the human replay buffer B3, obtained through AIM contains sufficient information needed to train a new offline IL policy. Moreover, the offline performance curve of AIM nearly coincides with that of the human-gated gated PVP baseline, demonstrating that the demonstrations AIM requests are at least as informative as those collected under continuous human supervision. This explains why AIM enables the agent to imitate human behavior and recovers the expert policy more rapidly and efficiently in Table I and Table 2.
6. Conclusion
In this work, we propose AIM, a novel robot-gated inter- active imitation learning algorithm that learns an adaptive criterion on requesting human help. By training a proxy Q-function approximate the human-gated intervention strategy, AIM proactively requests human assistance when the agent's policy deviates from the expert. This design enables the robot-gated -ated criterion to naturally evolve with the agent's ' policy, steadily reducing human interventions as the agent becomes proficient. A key factor in AIM's success is its ability to emulate the human intervention mechanism by targeting safety-critical states. By effectively capturing the most informative expert guidance information, AIM ef- fectively reduces the expert's 's cognitive efforts to teach the agent in these states. Compared to existing Interactive Imita- tion Learning methods, AIM achieves near-optimal success rates with fewer interventions from the human supervisors, outperforming both human-gated and robot-gated baselines across MetaDrive and MiniGrid tasks.
Limitations. We assume that the expert knows the optimal control strategy and behaves correctly. Additionally, this paper does not include real-human experiments or user stud- ies, and human demonstrations may be imperfect or faulty. Extending AIM to support human interactions with multiple agents remains unexplored.
These results indicate that AIM reduces the human demon- strations needed and enhances the effectiveness of human

---

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
Impact Statement
Hoque, R., Balakrishna, A., Novoseller, E., Wilcox, A.,
Brown, D. S., and Goldberg, K. Thriftydagger: Budget- aware novelty and risk gating for interactive imitation learning, 2021a.
Our AIM fosters a safer and more efficient interactive im- itation learning framework by requesting expert guidance only when needed, minimizing human cognitive load, and enhancing evaluation performance. While AIM has the po- tential to accelerate the deployment of intelligent systems, developers should also avoid introducing expert biases and overdependence on automation.
Hoque, R., Balakrishna, A., Putterman, C., Luo, M., Brown, S., Seita, D., Thananjeyan, B., Novoseller, E.,, and Goldberg, K. Lazydagger: Reducing context switching in interactive imitation learning. In 2021 IEEE 17th interna- tional conference automation science and engineering (case), pp- 502-509. IEEE, 2021b.
Acknowledgement
The project was supported by NSF grants CCF-2344955 and IIS I1S-2397769. ZP is supported by the Amazon Fellowship via UCLA Science Hub.
Kelly, M., Sidrane, C.,
K., and Kochen- derfer, M. J. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8077-8083. IEEE, 2019.
References
Biré, E., Kobanda, A., Denoyer, L., and Portelas, R. Ef- ficient active imitation learning with random network distillation, arXiv preprint 2:xx:2411.00894 2024.
Li, Q., Peng, Z., Feng, L.,, Zhang, Q. Xue, Z., and Zhou, B. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 2022a.
Celemin, C. and Ruiz-del Solar, J. An interactive framework for learning continuous actions policies based on correc- tive feedback. Journal of Intelligent Robotic Systems, 95(1):77-97, 2019.
Li, Q. Peng, Z., and Zhou, B. Efficient learning of safe driving policy via human-ai copilot optimization. In International Conference on Learning Representations, 2022b.
Chernova, S. and Thomaz, A. L.. Robot learning from human teachers. Springer Nature, 2022.
Luo, J., Dong, P., Zhai, Y., Ma, Y., and Levine, Rlif: Interactive imitation learning as reinforcement learning. arXiv preprint UrXiv:2311.129990 2023.
Chernova, S. and Veloso, M. Interactive policy learning through confidence- based autonomy. Journal of Artificial Intelligence Research, 34:1-25, 2009.
Mandlekar, A., Xu, D.. Martin-Martin, R., Zhu, Y., Fei- Fei, L., and Savarese, S. Human-in-the-loog imitation learning using remote teleoperation. ArXiv preprint, abs/2012.06733, 2020. URL htttt://aaxiiv abs/2012.
Chevalier Chevalier-Boisvert, M., Willems, L., and Pal, S. Minimalis- tic gridworld environment for openai gym. https:// // github. com/ maxi /ttp///mmmm//mmmmmmmmm//////////
2018.
Fang. B., Jia, S., Guo, D., Xu, M., Wen, S., and Sun, F. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Appli- cations, 3:362-369, 2019.
Menda, K., Crigge-Camphel,, K., and Kochenderfer, M. J. Ensembledagger: A bayesian approach to safe imitation learning. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5041-5048. IEEE, 2019.
Finn, C., Yu, T., Zhang, T., Abbeel, P., and Levine, One- shot visual imitation learning via meta-learning. In Con- ference robot learning, pp. 357-368. PMLR. 2017.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.550 2013.
Ganapathi, A.. Sundaresan, P. Thananjeyan, B., Balakr- ishna, A., Seita, D., Grannen, J., Hwang, M., Hoque, R., Gonzalez, J. E., Jamali, N., et al. Learning dense visual correspondences in simulation to smooth and fold real fab- rics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 11515 1115-11522. 11522. IEEE, 2021.
Osa, T. Pajarinen, J., Neumann, G., Bagnell, J. A., Abbeel, P., and Peters, J. An algorithmic perspective imitation learning. ArXiv preprint, abs/1811.06711, 2018. URL https: / /aaaivivvvv//////////aaa/////aaa www.ii..og/abbee.. htt.///aaab1111.1111111
Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Nickum, S., Knox, W. B., and Sadigh, D. Contrastive prefence learn- ing: Learning from human feedback without arXiv preprint arXiv:2310. 13639, 2023.
Peng, Z., Li, Q., Liu, C., and Zhou, B. Safe driving via expert guided policy optimization. In 5th Anmual Confer- ence on Robot Learning, 2021.
10

---

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
Peng, Z., Liu, Z.,, and Zhou, B. Data-efficient learning from human interventions for mobile robots. arXiv preprint arXiv:2503.088 250.00990,,, 2025.
Peng, M., Mo, W., Duan, C., Li, Q. and Zhou, Learn- ing from active human involvement through proxy value propagation. Advances in neural information processing systems, 36, 2024.
Ravichandar, H., Polydoros, A. S., Chernova, S., and Billard, A. Recent advances robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3:297-330, 2020.
Ray, A., Achiam, J., and Amodei, D. Benchmarking safe ex- ploration in deep reinforcement learning, arXiv preprint arXiv:1910.01 708, 7(1):2, 2019.
Reddy, S., Dragan, A. D., and Levine, S. Shared autonomy via deep reinforcement learning. Robotics: Science and Systems, 2018.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imita- tion learning and structured prediction to no-regret online learning. Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 627-635. JMLR Workshop and Conference Proceedings, 2011.
Sacidi, H., Opfermann, J. D., Kam, M., Raghunathan, S., Leonard, S., and Krieger, confidence based shared control strategy for the smart tissue autonomous robot (star). In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 1268-1275, 2018. doi: 10.1109/IROS.201 201000..201.... 2018.8594290.
Seraj. E., Lee, K. M., Zaidi, Z., Xiao, Q., Li, Z., Nasci- mento, A., van Waveren, S., Tambwekar, P., Paleja, R., Das, D., et al. Interactive and explainable robot learning: A comprehensive review. Foundations and Trends@ in Robotics, 12(--3):55-349, 2024.
Spencer, Choudhury, S., Barnes, M., Schmittle, M., Chi- ang. M., Ramadge, P., and Srinivasa, S. Learning from interventions. In Robotics: Science and Systems (RSS), 2020.
Zare, M., Kebria, M., Khosravi, A., and Nahavandi, S. A survey of imitation learning: Algorithms, recent develop- ments, and challenges. IEEE Transactions on Cybernet- ics, 2024.
Zhang, J. and Cho, K. Query-efficient imitation learn- ing for end-to-end autonomous driving. arXiv preprint arXiv: 166050065500 2016.

---

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
A. Comparison of Intervention Mechanisms in MetaDrive
To visualize the differences in the intervention mechanisms between AIM and the uncertainty- method Thrifty-DAgger (Hoque al., 2021a), we design toy MetaDrive environment in Fig. 7. The toy environment consists of two curved segments (one the start and one the end) and straight road between them. The straight road contains traffic cones and single roadblock, as illustrated in Fig. 7(a) and Fig. 7(b), respectively. Fig. 7(c) provides a top-down view of the straight road. There are no other traffic participants in this toy environment. The expert is trained using PPO (Ray et al., 2019) for 20 million environment steps and can safely navigate to the destination without colliding with traffic cones or the roadblock.
Traffic Cones
(h) Roadblock
(c) Top-Down View
Figure 7. Illustration of the toy MetaDrive environment. The key objective is to navigate to the destination without crashing into the traffic concs or the roadblock.
Take-Over
AIM 1K Steps
Thrifty-DAger- 1K Steps
Self Explore
AIM 2K Steps
Thrifty-DAgger 2K Steps
AIM 3K Steps
Thrifty-Dgger 3K Steps
AIM 4K Steps
Thrifty 4K Steps
Figure Comparison of intervention strategies between AIM Thrifty-DAgger in MetaDrive at different training stages. Arrows indicate the agent's 's steering actions in top-down view, with green arrows showing autonomous actions and yellow arrows showing expert takcovers. AIM reduces expert queries as the agent becomes more proficient, while Thrifty-DAgge continues to request expert assistance frequently on the straight road.
In Fig. 8, we compare the intervention mechanisms of AIM and Thrifty-DAgger at different training steps. After 2K steps, AIM's takeover mechanism confirms that the agent has learned to navigate curves and stay on the road, so it only requests expert help near the traffic cones and the roadblock. After 3K steps, AIM no longer requests assistance at the roadblock, and
12

---

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
by 4K steps, AIM agent requires no expert intervention and confidently navigate the goal. In contrast, Thrifty-DAgger needs request expert help almost continuously along the straight road during the first 2K steps. Even after 4K steps, Thrifty-DAgger still requests takeovers near the traffic cones and the roadblock. This demonstrates that the uncertainty-based takeover strategy in -Agger fails to adapt to the agent's improving performance, thereby demanding more expert demonstrations and cognitive effort.
B. Environment Details
Figure 9. MetaDrive Safety Benchmark
Figure 10, MiniGrid
MetaDrive Safety Benchmark. In Table 1, we compare AIM against several interactive imitation-leamingg baselines the MetaDrive safety benchmark (Li et al., 2022a). In this environment, the agent controls vehicle using low-level throttle, brake, and steering inputs to reach the goal while avoiding collisions, as shown in Fig. 9. MetaDDive's safety benchmark evaluates the agent's performance by procedurally synthesizing diverse driving scenarios: the training split consists of 50 unique maps, and the test split contains another 50 distinct maps. Each episode randomly selects a road layout and randomizes the spawn positions of both traffic and the ego vehicle, ensuring that agents encounter novel situations during both training and testing. the MetaDrive safety benchmark, the sensory state vector is used as the agent's observation. The agent's 's observation includes: (1) its current state variables such the steering angle, heading, speed, and distance to lane boundaries; (2) the navigation cues that indicate the direction toward the checkpoints and destination; and (3) the surrounding information represented by a -i0-eimmnsi Lidar-like point cloud with 50m detection range, capturing nearby vehicles and obstacles. We implement both the agent's policy and AIM's proxy Q-function using a two-layer MLP architecture.
MiniGrid Multi-Room Environment. In Table 2, we evaluate AIM the MiniGrid task shown in Fig. 10. The MiniGrid multi-room task requires extensive exploration: the agent navigate through sequence of doors, opening each one in Seervaaier-Boiiveer et al., 2018). The agent's 's starting position, the goal
turn, before finally reaching the green goal square location, the positions of all doors, and the room geometries are randomized in each episode. The state and action spaces are discrete. The agent's observation is represented by the semantic map of the agent's local neighhorhood, rendered as a 7x7 grid; the agent cannot see beyond walls or other obstacles. The action space has four valid actions: turn left, turn right, move forward, and open the door.
C. Hyperparameters
In MetaDrive, we implement the control policy and the proxy Q-network using two-layer MLP, where each hidden layer has 256 units with ReLU activations. For MiniGrid tasks, all models employ three-layer convolutional network with filter sizes of 16, 16, and 32. Each convolution uses a 2x2 kernel, and a max max- max-pooling layer is inserted between the first and second convolutional layers. The ReLU activation is applied after each layer. When training AIM and the robot-gated basclines Ensemble-DD DAgger and Thrifty-DAgger we use the same switch-to-agent threshold following Eq.
13

---

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
Table AIM (MetaDrive)
Table 5. AIM (MiniGrid)
Hyper-parameter
Value
Value
Discounted Factor y
0.99
Discounted Factor y
0.99
Learning Rate
le-4
Learning Rate
le-4
Gradient Steps per Iteration
1
Gradient Steps per Iteration
32
Train Batch Size
1024
Train Batch Size
200
Switch-to- Quantile 8
0.05
Switch-to-Human Quantile
0.05
Table 7. PVP (MiniGrid)
Table 6. PVP (MetaDrive)
Hyper-parameter
Value
Hyper-parameter
Value
Discounted Factor y for Target Network Update
0.99
Discounted Factor for Target Network Update
0.99
0.005
0.005
Learning Rate
cc-4
Learning Rate
le-4
Gradient Steps Iteration
32
Gradient Steps per Iteration
Target Network Update Interval
Train Batch Size
1024
Train Batch Size
Free Level
0.95
Free Level
0.95
Table Satemblee-Aggger (MetaDrive)
Table 9. Ensemble-DAgger (MiniGrid)
Hyper-parameter
Value
lyper-parameter
Value
Number of Instances
5
Number of Instances
Learning Rate
le-4
Learning Rate
lc-4
Gradient Steps per Iteration
Gradient Steps per Iteration
32
Train Batch Size
1024
Train Batch Size
200
BC Warmup Steps
200
BC Warmup Steps
100
Switch-to-Human Threshold
le-3
Switch-to-Huuaa Threshold
Se-3
Table 10. Thrifty-DAgger (MetaDrive)
Table 11. Thrifty-DAgger (MiniGrid)
Hyper-parameter
Value
Hyper-parameter
Value
Number of Instances
S
Number of Instances
Discounted Factor 7
0.99
Discounted Factor y Learning Rate Gradient Steps per Iteration
Learning Rate
le-4
le-4
Gradient Steps per Iteration
1
32
Train Batch Size
1024
Train Batch Size
200
Warmup Steps
200
BC Warmup Steps
100
Switch-to-Human Quantile of Novelty 6j
0.05
Switch-to- Quantile Novelty
Update Frequency of 81
25
Update Frequency of 6
25
Switch-to-Human Quantile of Risk 82
0.01
Switch-to-Human Quantile of Risk 62
0.01
Update Frequency of 82
25
Update Frequency of òz
25
14


Reviews:
[{"review_id": "BaJiGMbaZ0", "reviewer": "Reviewer_HDAh", "summary": "The authors proposed Adaptive Intervention Mechanism (AIM), a new robot-gated shared autonomy mechanism that better align agent with human expert thorugh a proxy Q-function. This algorithm requires less human monitoring comparing to human-gated interactive imitation learning methods, while more intelligently and efficiently request human expert intervention comparing to other robot-gated imitation learning methods. The authors tested the algorithms on MetaDrive and MiniGrid Four Room Test and achieved SOTA performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to carefully read through and understand our paper, and provide constructive feedback. We summarize and respond to each question as follows:\n\n__Other Strengths And Weaknesses:__\n\n>1.Why L2 distance instead of other distance metrics? Is $a_h$ unique and deterministic in task like MetaDrive?\n\nWe use L2 distance because it is simple to implement and effectively identifies states where the agent's actions deviate significantly from expert behavior. Fig 3 in https://limewire.com/d/teXVP#Okg7PsYIne shows that using the L1 distance metric does not affect the performance. In MetaDrive, the expert action $a_h$ is stochastic, as the expert policy is a stochastic policy trained with Lagrangian PPO.\n\n>2.Elaborate more on the choice of fixing the threshold epsilon in Eq. 9.\n\nThe threshold $\\epsilon$ controls __the length of expert demonstrations after the agent requests help__. We choose $\\epsilon$ based on observations from the human replay buffer in warm-up steps (Eq. 8, Line 240). A small $\\epsilon$ leads to overuse of expert help, while a large $\\epsilon$ results in insufficient corrections, slowing down training.\n\n>Does an adaptive switch-to-agent threshold work and perform better?\n\nAn adaptive switch-to-agent threshold does __not__ significantly improve the number of expert-involved steps. In our current setup, the agent requires only 5–10 steps of expert help before returning to self-exploration, which is nearly the minimum needed for safety.\n\n>3.Give several examples under safety-critical states, where other baselines robot-gated IL baselines fail to request for human help but AIM did.\n\nIn Fig. 1 of https://limewire.com/d/teXVP#Okg7PsYIne , when the car approaches the road boundaries, AIM successfully requests human intervention, while other uncertainty-based methods output a low uncertainty estimate and fail to signal the need for help.\n\n>4.Why did authors not compare AIM with Ensemble-Dagger and Thrifty-Dagger for mini-grid?\n\nIn Page 6 Line 310-314, we mentioned that the two methods rely on the action variance for uncertainty estimation, which doesn’t work well with __discrete action spaces__ like MiniGrid. 'Action variance' refers to the variance of the output actions across the ensemble of policy networks. \n\nTo apply the two baselines to MiniGrid, we need to replace their variance-based uncertainty estimations by the __entropy-based__ estimation, which is the entropy of the action distributions derived from the soft Q-value. Table 1 of https://limewire.com/d/teXVP#Okg7PsYIne shows that AIM reduces the expert-involved steps needed compared with the two baselines.\n\n>5.In page 7 line 374, the authors are claiming AIM’s mechanism saves training time and the total environment data usage according to figure 4, but results show that AIM is not the one with least total data usage.\n\nThanks for pointing out the misleading claim. In Figure 4, we show that AIM requires fewer environment samples than other robot-gated baselines (Thrifty-DAgger and Ensemble-DAgger) to approach expert actions in safety-critical states. In Table 1 and Table 2, we highlight that AIM reduces expert-involved steps and cognitive effort, though it may require more training time. Thus, we revise line 374 to:\n\n\"Compared with __other robot-gated IIL baselines__, AIM requires fewer environment data usage to __imitate expert actions in safety-critical states__.\"\n\n>6.How does the intervention rate change during the training/testing stage?\n\nWe visualize the overall intervention rate in the training stage in Fig. 4 of https://limewire.com/d/teXVP#Okg7PsYIne . Our method AIM’s intervention rate matches that of human-gated PVP and is lower than other robot-gated baselines.\nAccording to Line 281-284, in the test stage, we evaluate the agent’s performance without expert involvement, so there’s no intervention rate during testing.\n\n>7.Does introduce warm up stage to Ensemble/Thrifty-DAgger lead to better performance?\n\nIn our experiment (Table 1 and Table 2), we already introduced a warm up stage to all the baselines including Ensemble-DAgger, Thrifty-DAgger, and PVP for fairness. (i.e., demonstrating initial two trajectories as we do for AIM).\n\n__Other Comments Or Suggestions:__\n\n>1.What’s $I^{exp}$, $\\delta$ in the formula (1)?\n\n$I^{exp}$ is the human-gated intervention criterion, where the expert decides whether to take over control at state s using human control signal $a_h$ when the agent outputs action $a_r$. The $\\delta$ function in Eq. 1 is the __Dirac delta distribution__.\n\n>2.What’s the unit of the y axis for the plots in figure 2? How is uncertainty estimated?\n\nIn Figure 2, the y-axis represents the uncertainty estimation: $Var(a_n) - \\varepsilon$, where $Var(a_n)$ is the variance of agent actions and $\\varepsilon$ is the switch-to-human threshold in Ensemble-DAgger. Human help is requested when $Var(a_n) > \\varepsilon$. We plot $\\max(0, Var(a_n) - \\varepsilon)$ to visualize the timesteps when human help is requested.", "claims_and_evidence": "See bullet points.", "methods_and_evaluation_criteria": "See bullet points.", "theoretical_claims": "See bullet points.", "experimental_designs_or_analyses": "See bullet points.", "relation_to_broader_scientific_literature": "Clear literature review.", "essential_references_not_discussed": "See bullet points.", "other_strengths_and_weaknesses": "1. Why L2 distance instead of other distance metrics, for example distributional divergence for action-difference function f? It’s not very clear that in human demonstration data, in same state, a_{h} is unique and deterministic? Especially in task like MetaDrive?\n2. Could the authors elaborate more on the choice of fixing the threshold epsilon in (9), I would be interested in knowing if an adaptive switch-to-agent threshold would work and perform even better, or if not better, what could be the reason.\n3. Like the clear illustration in figure 2, could the authors give several example under safety-critical states, where other baselines robot-gated IL baselines fail to request for human help but AIM did? \n4. The experiments for the continuous action space environment and discrete action space environment seems not well-aligned. Why authors did not compare AIM with Ensemble-Dagger and Thrifty-Dagger for mini-grid? \n5. In both table 1 and table 2, results show that AIM is not the one with least total data usage, which is fine, since this not the superior aspect of AIM the authors are claiming. However, in page 7 line 374, the authors are claiming AIM’s mechanism saves training time and the total environment data usage according to figure 4, I think it’s not a fair comparison. (AIM with low rate of requesting human intervention could lead to longer training time in some cases.) If comparing vertically in figure 4, using same amount of data, AIM has the least deviation in the critical states, then you should also limit the experiments in table 1 and 2 to same total data then compare the success rate.\n6. Since the authors claim the AIM’s great performance origins from it’s adaptive mechanism, I think if would be helpful to add a direct visualization of how the intervention rate changes during the training/testing stage, whether the agent request fewer human intervention as the agent getting more proficient. This should also be compared with other methods.\n7. Since AIM has a human-gated warm up stage, would there be a chance that introducing this warm up stage to Ensemble-DAgger and Thrifty-DAgger would also leads to better performance? Then authors argument would be weakened.", "other_comments_or_suggestions": "1. What’s I^{exp}? What’s delta? Though some explanation appears in very later part of the paper, the author should explain each term in the formula (1) in the following paragraph, in order to have a clear background knowledge for the readers.\n2. What’s the unit of the y axis for the plots in figure 2? Probability of human taking over?  And how is uncertainty estimated? It seems not quite reasonable that many steps come with 0 uncertainty. Would be great to add clear legends to help better explanation.", "questions_for_authors": "See bullet points.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O3WzI8Om3Q", "reviewer": "Reviewer_xbPG", "summary": "This paper develops an approach to imitation learning called Adaptive Intervention Mechanism (AIM) that learns whether to ask an expert for an action label based upon whether AIM thinks the imitation learner already knows the correct action. An objective function is developed (Equation 3) that governs this AIM scheme by learning the weights of a Q-function, weighted by the level of disagreement between the expert and learner. Results are generated on some simple benchmarks and show that the proposed approach generally outperforms some baselines.\n\n### Update after rebuttal\n\nThe reviewer appreciates the authors’ response and responded below. Overall, the response from the authors suggests a plan to improve the paper that would indeed improve the contribution to ICML. However, it would have been helpful to actually see these changes and have more details to ensure that the paper's claims match its contribtuions. I was initially a weak accept, and the rebuttal has solidified that rating for me.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reading our paper in detail and providing valuable suggestions. We summarize and respond to each question as follows:\n\n__Claims And Evidence:__\n\n>However, the term \"sufficient\" in the third claim (Line 98) should be softened unless providing a proof.\n\nWe revise Line 98: “The expert demonstrations requested by AIM contain corrective actions in safety-critical states, so that they can assist a novice agent to imitate the expert’s policy.”\n\n__Relation To Broader Scientific Literature:__\n\n>The awareness of human-centered literature and decades of work on this topic seems to be lacking. The paper does not discuss Confidence-based Autonomy (CBA).\n\nThanks for sharing relevant works! We will add these references in the revised version.\n\n__Other Strengths And Weaknesses:__\n\n>The limitations section is quite short and does not fully address the many weaknesses of not doing a real user study.\n\nWe will add this to the limitation section: \"This paper does not include real-human experiments or user studies, and human demonstrations may be imperfect or faulty.\"\n\n>Figures 4-5 show unconvincing results regarding the superiority of AIM vs. PVP.\n\nFigures 4 and 5 show that AIM and PVP perform similarly in imitating the expert during safety-critical states and collect high-quality expert demonstrations. In short, __AIM’s intervention rule behaves similarly with the human-gated PVP’s rule, even though AIM is a robot-gated interactive IL method.__\n\nIn Figure 4 and 5, we do __not aim to show that AIM’s intervention rule is better than PVP__. According to Line 358-362 in Page 7, since PVP is a human-gated IL algorithm, the key advantage of AIM over PVP is that it requires fewer human cognitive efforts and expert-involved steps.\n\n__Questions for Authors:__\n\n>Under what conditions of f in Equation 2 is the agent able to recover the expert policy?\n\nWe clarify in Line 164 (Page 3) that the expert’s intervention method follows Eq. 2:\n$I^{exp}(s, a_r, a_h) = f(a_r, a_h) = \\mathbb{I}[\\|a_r - a_h\\|^2 > \\epsilon]$ ($a_r$ is the robot action, $a_h$ is the human action). With sufficient expert demonstrations, we can bound the difference of the value functions of the expert policy and the student’s final learned policy by $\\epsilon$ and the horizon $H$. \n\n>Must the f in Equation 2 conform to the equation embedded in the text from Menda et al., 2019 and Hoque et al., 2021a;b?\n\nThe function f does not need to conform to Menda et al. (2019) and Hoque et al. (2021). The general form of the intervention is based on the probability of the expert taking the robot’s action, which reduces to Eq. 2 if the expert follows a Gaussian policy. The proof is in Theorem 3.3 of “Guarded Policy Optimization with Imperfect Online Demonstrations”  (Z Xue et al., 2023).\n\n>It is confusing to say that $Q_{\\theta}^I(s, a_r)$ is equal to -1 or +1 by assignment (Lines 209-218). That implies that \\gamma = 0 and Equation 4 is meaningless -- there is no TD-Error.\n\nIn Line 209-218, the proxy value assignment is a learning objective but not a hard constraint to the proxy value function. The AIM loss helps select human preferable actions at those states $s$ with human interventions, and TD loss propagates human preference to the states $s$ __without human interventions__.\n\n>Should it not be the reward function definition for -1 and +1 -- not the Q-function?\n\nWe include an ablation study in Fig. 3 of https://limewire.com/d/teXVP#Okg7PsYIne . The figure shows that dropping the TD loss or replacing the Q-labeling by reward-labeling will damage the performance of AIM.\n\nReplacing the Q-labeling by reward-labeling fails in our setting because negative rewards cannot be matched with transitions from unsafe agent actions. In human-involved transitions $(s, a_h, s’)$, we can assign +1 since $s’ \\sim P(s, a_h)$ is generated by the human action. However, for dangerous agent actions $a_r$, querying the environment to obtain the resulting state $s’’ \\sim P(s, a_r)$ is not feasible.\n\n>It is unclear how this paper is presenting an approach to \"shared autonomy.\" See this work by Reddy et al. (2018).\n\nThanks for pointing out that our definition of \"shared autonomy\" differs from Reddy et al. (2018), and we revise the terminology to \"__interactive imitation learning__.\" While Reddy et al. apply human-AI shared control during both training and testing, we only use it in the training phase and evaluate the agent's performance without human involvement in the test phase.\n\n>Why are none of the baselines able to match the Neural Expert?\n\nIn Table 1, we require all the baselines to use no more than 2K expert-involved steps. These baselines can match the neural expert with 3.5K expert-involved steps. See Fig 2 of https://limewire.com/d/teXVP#Okg7PsYIne .\n\n>Why is the neural expert not at near 100% performance?\n\nThe neural expert is trained using Lagrangian PPO with 20M environment steps. MetaDrive safety environments can present challenging scenarios where a well-trained expert may fail.", "claims_and_evidence": "The claims that AIM is a novel algorithm, that the algorithm was evaluated and showed positive results is generally accurate. However, the term \"sufficient\" in the third claim (Line 98) should be softened unless providing a proof.", "methods_and_evaluation_criteria": "Yes, generally speaking, the approach and how it is evaluated make sense. However, a user study would have been helpful. There are issues described below.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The experimental design has a key weakness that it lacks a user study. Some of the results appear mixed (e.g., Figures 4-5). However, the analysis is reasonable for an ICML paper.", "relation_to_broader_scientific_literature": "Except for the issue noted below regarding Confidence-based Autonomy, the paper generally covers the recent literature in interactive machine learning for an ML audience. However, the awareness of human-centered literature and decades of work on this topic seems to be lacking. I recommend more thoroughly reading the paper's own references, such as \n\nArgall, B. D., Chernova, S., Veloso, M., and Browning, B. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469–483, 2009.\n\nMore recent papers/books might also help:\n\nEsmaeil Seraj, Kin Man Lee, Zulfiqar Zaidi, Qingyu Xiao, Zhaoxin Li, Arthur Nascimento, Sanne van Waveren, Pradyumna Tambwekar, Rohan Paleja, Devleena Das and Matthew Gombolay (2024), \"Interactive and Explainable Robot Learning: A Comprehensive Review\", Foundations and Trends® in Robotics: Vol. 12: No. 2-3, pp 75-349. http://dx.doi.org/10.1561/2300000081\n\nRavichandar, H., Polydoros, A.S., Chernova, S. and Billard, A., 2020. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3(1), pp.297-330.\n\nZare, M., Kebria, P.M., Khosravi, A. and Nahavandi, S., 2024. A survey of imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions on Cybernetics.", "essential_references_not_discussed": "The paper does not discuss Confidence-based Autonomy (CBA).\n\nChernova, S. and Veloso, M., 2009. Interactive policy learning through confidence-based autonomy. Journal of Artificial Intelligence Research, 34, pp.1-25.\n\nThis paper sets up essentially the same problem and an analogous solution approach. There is an imitation learner that decides when to ask humans to take over control (add labels) vs. to autonomous execute (which still allows humans to observe and manually override). While Chernova's work was based upon a measure of model uncertainty (confidence), it is still directly related to how this paper uses a Q-function (albeit one I am confused about -- see my question regarding whether a TD-error actually means anything or exists). At a minimum, this prior work must be discussed, and the authors should do a more thorough literature review to find related papers that they might have missed by focusing only on recent trends in gated DAgger-like approaches. Ideally, this paper would benchmark against CBA.", "other_strengths_and_weaknesses": "Imitation Learning is, by definition [Ross et al., 2011], interactive and online. Humans give labels in real-time based upon online policy rollouts. The notion of \"interactive imitation learning\" seems to be confusing established concepts. The paper does provide references (such as Kelly et al., 2019) to back up this terminology, but I think it is unhelpful. If this paper refers to \"imitation learning\" in the sense of any learning from demonstration algorithm (whether it be Behavior Cloning or Inverse Reinforcement Learning (IRL)), then that needs to be clearer. Also, IRL is non-interactive but online. There is also offline IRL, etc. I think a more wholistic view of the literature and reaching further back into established concepts would improve the paper. I'd also refer the authors to this book [Chernova & Thomaz, 2022], which is a helpful treatise to define these concepts.\n\nRoss, S., Gordon, G. and Bagnell, D., 2011, June. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics (pp. 627-635). JMLR Workshop and Conference Proceedings.\n\nChernova, S. and Thomaz, A.L., 2022. Robot learning from human teachers. Springer Nature.\n\nAs per Lines 275-278, the paper says, \"Following the prior works on interactive imitation learning (Hejna et al., 2023; Peng et al., 2021), we incorporate well-trained neural policies in the training loop to approximate human policies.\"\n\nHowever, the limitations section is quite short and does not fully address the many weaknesses of not doing a real user study. Users are not \"perfect,\" so claims should be softened.\n\nFigures 4-5 show unconvincing results regarding the superiority of AIM vs. PVP.\n\nIt would have been helpful to include Adversarial Inverse Reinforcement Learning or a more competitive baseline. AIM requires online interaction with the user and the environment. AIRL may do quite well if given access to the environment but only limited data.", "other_comments_or_suggestions": "There is a missing article in \"mimic human intervention rule\"", "questions_for_authors": "Under what conditions of f in Equation 2 is the agent able to recover the expert policy? Must it conform to the equation embedded in the text from Menda et al., 2019 and Hoque et al., 2021a;b? This claim seems to strong without offering further proof, and the description should be clearer if the authors are relying on prior work for the proof.\n\nIt is confusing to say that $Q^I_{\\theta}(s, a_r)$ is equal to -1 or +1 by assignment (Lines 209-218). Should it not be the reward function definition for -1 and +1 -- not the Q-function? If it truly is the Q-function, then that implies that \\gamma = 0 and Equation 4 is meaningless -- there is no TD-Error. Can the authors kindly clarify?\n\nIt is unclear how this paper is presenting an approach to \"shared autonomy.\" See this work by Reddy et al. (2018).\n\nReddy, S., Dragan, A.D. and Levine, S., 2018. Shared autonomy via deep reinforcement learning. arXiv preprint arXiv:1802.01744.\n\nWhy are none of the baselines able to match the Neural Expert? \n\nWhy is the neural expert not at near 100% performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jvyfAmw57y", "reviewer": "Reviewer_JBzX", "summary": "The paper proposes an adaptive intervention strategy aiming to use the shared automony to improve the robot execution process. Previous robot -gated designs rely on the entropy to judge whether to let the human intervent. Using this strategy, the robot would frequently ask humans for the help, whihch is constly. Authors propose an a adaptie interactive intervention straetey, where Q function is learned, whch is further used to judge when to interact. Experiments demonstrated the effectiveness enabled by the method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your effort to thoroughly review our paper and for your feedback. In response to your feedback, we have included qualitative evaluations and ablation studies that have strengthened the study.\n\n__Experimental Designs Or Analyses:__\n\n>1.Qualitative evaluations are missing.\n\nWe include qualitative evaluations in Fig 1 of https://limewire.com/d/teXVP#Okg7PsYIne to show that when the car approaches the road boundaries, AIM successfully requests human intervention, while other uncertainty-based methods output a low uncertainty estimate and fail to signal the need for help.\n\n__Other Strengths And Weaknesses:__\n\n>1.Line 053: why \"fixed intervention criterion\" would cause \"fixed request rate\"? The logic between such two sentences is not clear.\n\nIn Line 053, __fixed intervention criterion__ refers to requesting human help when the __uncertainty estimate exceeds a constant $\\epsilon$__, as in EnsembleDAgger, which uses the variance of actions in the policy ensemble as the uncertainty estimate. \n\nAccording to Fig 4 of https://limewire.com/d/teXVP#Okg7PsYIne, __EnsembleDAgger requests expert help at a fixed intervention rate__ after 1K steps, implying that some states’ uncertainty estimates always exceed $\\epsilon$. In contrast, ThriftyDAgger adapts $\\epsilon$ and avoids a fixed request rate.\n\nTherefore, we __revise Line 053__ as follows:\n\n\"The uncertainty-based methods including (Menda et al., 2019; Kelly et al., 2019) request human help when __the uncertainty estimate is larger than a fixed threshold__. __Without adjusting the threshold adaptively__, the agent keeps requesting expert help even when it has successfully imitated the human expert.\"\n\n>2.Line 071: What do you mean by \"dynamically adjusts to...agent policy\"? That's quite confusing.\n\nWe __revise Line 071__ for clarity:\n\n\"Our learned intervention criterion adaptively __shrinks the intervention rate as the agent becomes performing__. Our learned intervention criterion can request human help at an appropriate rate based on the agent's performance during training.\"\n\nWe further explain Line 071 as follows. According to Line 184-188, the __intervention frequency of AIM gradually drops down__ as the agent’s performance improves during training. The reason is that AIM’s Q value depends on the proportion of states that the agent’s action aligns with the expert’s action. During training, the average of AIM’s Q value decreases towards -1, leading to the shrink of the intervention rate. The shrink of the intervention rate is also shown in Fig 4 of https://limewire.com/d/teXVP#Okg7PsYIne . We can observe that our method’s intervention rate matches that of human-gated PVP even though it’s a __robot-gated__ method, implying that our intervention mechanism adapts to the agent’s performance and resembles the human-gated intervention rule. In addition, our intervention rate is lower than other robot-gated baselines. \n\n>3.Line 295: \"we report the success rate\" or \"we employ the success rate as the evaluation metric\" would be better.\n\nThanks for your suggestion! We revise Line 295 as follows:\n\"In MiniGrid, we employ the success rate as the agent’s evaluation metric.\"\n\n__Questions For Authors:__\n\n>1. Could the authors demonstrate the value of their method in more complex tasks such as those involving physical environments, e.g. Mujoco tasks like Ant in Gymnasium.\n\nWe appreciate the reviewer’s interest in evaluating our method on physical tasks such as those in MuJoCo. However, tuning and training a near-optimal expert and all baseline methods demand time and resources. Moreover, human demonstrations for MuJoCo tasks like Ant are infeasible, as a human cannot reasonably demonstrate the complex multi-legged locomotion task. Our MetaDrive environment with a high-dimensional observation space generates diverse driving scenarios such as fixed or movable traffic vehicles, traffic cones, and warning triangles, which already offers a compelling and challenging benchmark. We plan to include experiments on physical tasks in future work.\n\n>2. Why the proposed Q function could help the agent receive sufficient human guidance at \"safety-critical states\"? Why the Q-function can emphasize the \"safety-critical states\"?\n\nThe key is that the proposed Q-function can classify states where __the agent’s actions align with human actions__ and those where __there is a significant action discrepancy__.\n\nBy explicitly using an action difference function $f$ to label these states, the Q-function learns to emphasize those states where human guidance is most needed. Additionally, incorporating a TD loss propagates these signals to nearby states. \n\nWe also include an ablation study in Fig 3 of https://limewire.com/d/teXVP#Okg7PsYIne , showing that dropping the TD loss or replacing the Q-labeling by reward-labeling will damage the performance of AIM. This implies the effectiveness of our AIM Q-function design.", "claims_and_evidence": "The claims made  are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The method and evaluation criteria make sense for the problem.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experiemtnal designs are reasonalbe but a little bit simple.\n\nThe experiments mainly focus on evaluating the performance evolution of the agent as the number of human-involved steps increases. By comparing with previous robot-gated and human-gated imitation learning methods, authors wish to demonstrate that their method can achieve higher performance in a shorter time with fewer human-invovled steps. They also demonstrate that their method can help the agent receive sufficient human guidance at safety-critical issues. \n\nThe effectiveness of the proposed method can be validated by quantitative results shown in the table and plots. The main limitation lies in the simplicity of their evaluated tasks. Besides, qualitative evaluations are missing.", "relation_to_broader_scientific_literature": "Related to the robotics and adaptive policies.", "essential_references_not_discussed": "References are properly discussed.", "other_strengths_and_weaknesses": "Strengths:\n- The motivating example in the method section is a good point and can help readers understand the problem setting and motivations. \n\nWeaknesses:\n- Line 053: \"so the agent keeps requesting help at a fixed rate even when it has...\" why \"fixed intervention criterion\" would cause \"fixed request rate\"? The logic between such two sentences is not clear.\n- Line 071: \"Second, our learned intervention criterion dynamically adjusts to the frequently changing agent policy during training.\" The meaning of this sentence is also unclear. What do you mean by  \"dynamically adjusts to...agent policy\"? That's quite confusing. \n- Line 295: \"evaluation performance\"? That's strange. \"we report the success rate\" or \"we employ the success rate as the evaluation metric\" would be better", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Effecitveness in more complex tasks: Could the authors demonstrate the value of their method in more complex tasks such as those involving physical environments, e.g. Mojoco tasks like Ant in Gymnasium. \n- Method design: why the proposed Q function could help the agent receive sufficient human guidance at \"safety-critical states\"? Why the Q-function can emphasize the \"safety-critical states\"?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Adaptive Intervention Mechanism (AIM), an imitation learning framework designed to improve the efficiency of human-robot collaboration by adaptively determining when to request expert input. Unlike previous robot-gated shared autonomy methods that rely on high-entropy heuristics, AIM leverages a learned Q-function to predict the necessity of expert guidance based on disagreement between the agent and the expert. This approach enables more selective and efficient intervention, reducing the need for constant human supervision while maintaining high task performance. The method was validated on benchmark environments such as MetaDrive and MiniGrid Four Room, where it achieved state-of-the-art results and demonstrated improved alignment with expert behavior compared to existing human-gated and robot-gated imitation learning strategies.
Decision: Accept (poster)