Faithful Dynamic Imitation Learning from Human Intervention with Dynamic Regret Minimization
Authors: Bo Ling
Zhengyu Gan
Wanyuan Wang
Guanyu Gao
Weiwei Wu
Yan Lyu
Venue: NeurIPS 2025 poster ()
Keywords: Human-in-the-loop Imitation Learning
Dynamic Regret
Online Learning
Off-policy Imitation Learning
Primary area: reinforcement_learning
TL;DR: We propose  a human-in-the-loop learning method that achieves faithful imitation via distribution alignment and adapts to evolving behavior using dynamic regret minimization.

Abstract: Human-in-the-loop (HIL) imitation learning enables agents to learn complex behaviors safely through real-time human intervention. However, existing methods struggle to efficiently leverage agent-generated data due to dynamically evolving trajectory distributions and imperfections caused by human intervention delays, often failing to faithfully imitate the human expert policy. In this work, we propose Faithful Dynamic Imitation Learning (FaithDaIL) to address these challenges. We formulate HIL imitation learning as an online non-convex problem and employ dynamic regret minimization to adapt to the shifting data distribution and track high-quality policy trajectories. 
To ensure faithful imitation of the human expert despite training on mixed agent and human data, we introduce an unbiased imitation objective and achieve it by weighting the behavior distribution relative to the human expert's as a proxy reward.
Extensive experiments on MetaDrive and CARLA driving benchmarks demonstrate that FaithDaIL achieves state-of-the-art performance in safety and task success with significantly reduced human intervention data compared to prior HIL baselines.

Faithful Dynamic Imitation Learning from Human Intervention with Dynamic Regret Minimization
Bo Ling
Zhengyu Gan Southeast University zhengyugan@seu. edu. en
Wanyuan Wang Southeast University uywang@seu. edu. en
Southeast University bo po_lingdseu. edu. en
Guanyu Gao
Weiwei Wu
Nanjing University of Science and Technology jusao@njust. edu. on
Southeast University weiweiwu@seu, edu cn
Yan Lyu'
Southeast University lvyanlydseu edu. cn
Abstract
Human-in-the-loon (HIL) imitation learning enables agents learn complex be- haviors safely through real-time human intervention. However, existing methods struggle efficiently leverage agent-generated data due dynamically evolving trajectory distributions and imperfections caused by human intervention delays, often falling faithfully imitate the human expert policy. In this work, we propose Faithful Dynamic Imitation Learning (FaithDalL) to address these challenges. We formulate learning from human intervention as an online non-convex problem and employ dynamic regret minimization to adapt to the shifting data distribution and track high-quality policy trajectories. To ensure faithful imitation of human expert despite training mixed agent and human data, we introduce an unbiased imitation objective and achieve by weighting the behavior distribution relative to the human expert's 's as proxy reward. Extensive experiments on MetaDrive and CARLA driving benchmarks demonstrate that FaithDalL achieves state-of- the-art performance in safety and task success with significantly reduced human intervention data compared to prior HIL baselines. The corresponding source code is available htttps::// /tttts:://githut com/Willle wtw.lliiaiinaaaaaaiaaaaaaeeeaaee island/FaithDail
Introduction
Human-in-the-loop (HIL) imitation learning is promising approach to address key limitations of traditional reinforcement learning, such misalignment with human intent and poor sample efficiency. By incorporating human feedback and interventions into the learning process, HIL enables agents better align with human preferences, even when reward functions are difficult design or
prone to unintended biases [ 112 3. While early HIL methods relied on passive human feedback these methods are unsuitable for safety-critical domains like autonomous driving, where unacceptable risks can arise during data collection [ 9..
such as preference on behavior trajectories 4 5
In contrast, active human intervention (see Figure where humans provide real-time corrections and demonstrations during agent execution, directly enhances training-time safety. While some early methods like Human-Gated DAgger (HG-DAgger) g focus solely on numandprovided data, this data
"Coresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

State
D
D
D
D
Intervention n
Environment
Human
Agent
Action
Safe
Safe
Deviation Human Intervention
Safe
Initial
Novice
State
Novice Palicy
Novice Palicy
Palicy
Human Policy
Policy
a) Agent-Environment Interaction
b) Agentšehavior Trajectories
Figure 1: Learning from Intervention. a) Eeeent-Enviromment Interaction: human expert supervises the learning agent interacting with the environment, and intervenes when necessary. b) Agent Behavior Trajectories: includes safe but not optimal trajectory from initial suboptimal novice policy, improved novice policy trajectory, deviation (red) due human reaction latency, human intervention and subsequent recovery. Our FaithDalL leverages novice policy trajectories DV together with human intervention DH to improve data efficiency, while explicitly considering the evolving novice data distribution shifts and intervention latency to achieve faithful imitation human experts.
often costly and limited. To improve data efficiency, recent methods incorporate agent-generated trajectories (often called novice trajectories) supplementary training data, combining them with human interventions in an off- aff-policy imitation learning framework
However, leveraging ggent-generated trajectories in HIL introduces two challenges. First, as the agent's policy improves throughout training, the distribution of its generated trajectories shifts significantly (see green trajectories in Figure (b). Using these trajectories indiscriminately many originating from early, suboptimal policies impedes effective policy updates if all historical data treated equally. Our first key insight is that learning from human intervention problem is fundamentally an online learning problem with a dynamically changing data distribution. Instead of viewing it as static problem with expanding dataset like DAgger [12], we formulate it as a dynamic regret minimization problem to explicitly adapt to the evolving behavior distributions over time.
Second, even recent agent- generated data may be imperfect due to human reaction delays during interventions. The agent might execute incorrect actions just before an intervention (see red deviation trajectory in Figure [p)). Imitating these "deviation trajectories" risks teaching the agent suboptimal or unsafe behaviors. Our second key insight that while leveraging agent-gent ggent-generated data crucial for data efficiency, the primary imitation target should be the human expert policy. Therefore, we need an unbiased objective that ensures faithful imitation of the human expert, even when training on mixture of expert and novice data.
In this paper, we propose Faithful Dynamic Imitation Learning, FaithDalL, novel HILL approach that enhances data efficiency by integrating evolving novice trajectories through dynamic regret minimization. It further achieves faithful imitation of human experts by deriving a proxy reward from weighting the mixed behavior distribution to the expert's distribution. To the best of our knowledge, this the first work that formulate learning from human intervention as an online non-convex learning problem to minimize dynamic regret of imitation loss. In summary, our contributions are:
We propose Faithful Dynamie Imitation Learning framework, FaithDalL, that formally formulates learning from human intervention as an online non-convex learning problem and adopts dynamic regret the performance metric explicitly adapt to the changing data distribution induced by the evolving novice policy.
We propose an unbiased objective for faithful human expert imitation from mixed data (novice trajectories and human interventions). and achieve it by weighting the behavior distribution relative to the human expert's 's as proxy reward.
We conducted extensive experiments on MetaDrive and CARLA driving benchmarks. Re- sults show FaithDalL sigmificantly outperforms leading HIL baselines in safety and task success, with notably less human intervention data.

---

2 Related Work
Human-in-the-loop Imitation Learning. Many studies explore the incorporation of humans into
the training loop in imitation learning. Passive approaches, such as DAgger [2] and its variants
[3 14. [1 16 [7], address the compounding error problem [18] inherent in behavior cloning by periodically querying an expert for additional demonstrations on agent-visited states. Instead of providing demonstrations upon requests, active approaches allow experts to take control and guide the agent to safer states. Human-Gated DAgger (HG-DAgger) [8] involves human intervention during training and learns policy by leveraging the collected human data. However, using only human data is costly and limited. To improve data efficiency, recent methods such as Expert Intervention Learning (EIL) [1)], Intervention Weighted Regression (IWR) [T]. HACO (M. and PVP [9] incorporate agent-generated trajectories as supplementary data, combining them with human interventions in an off-policy imitation learning framework. EIL [19] and IWR [10] treat agent-generann data in supervised manner, while HACO [ and PVP [9] leverage off-policy RL. techniques to imitate a combination of agent-generated data and human data.
However, these methods face two major drawbacks: 1) they effectively minimize static regret on aggregate data, neglecting the evolving data distribution shifts, and 2) they imitate mixed behavior trajectories instead of focusing solely on the expert, leading to learning suboptimal actions caused by intervention latency (only partially addressed by EIL's fixed window, which cannot adapt dynamic environment
Our FaithDalL addresses these by formulating human-in in-the-loop learning as dynamic regret minimization to align with evolving distributions and proposing an unbiased off-policy imitation objective based on DICE (0) for faithful expert imitation.
Off-policy Imitation Learning.
comman-in-the-loyy (HIL) imitation learning, characterized by sparse and disruptive interventions, makes on-policy data collection inefficient; off-policy methods are therefore essential (1]. While behavior cloning (BC) [22) is inherently off-policy, it suffers from
compounding errors and ignores environment dynamics 18 For adversarial methods, DAC (21) extends GAIL [23 to off- off-policy setting using a replay buffer for better sample efficiency. The DICE
family 22 24 25 26 addresses off-policy imitation via stationary distribution matching. However, these methods primarily focus on expert demonstrations. Based DICE, some methods incorporate
offline supplementary imperfect demonstrations to enhance data efficiency [27] 28,, 29 30 31] but rely on striet coverage assumption 27 29 32
In our FaithDalL, by defining the behavior data as combination of agent-generated and human expert data, the coverage requirement (i.e., the support of the expert distribution is covered by the behavior distribution) is naturally satisfied, enabling faithful imitation learning from human interventions.
Online Non-convex Learning. Online learning offers principled framework for sequential decision making 33 In the convex setting, Online Gradient Descent (OGD) and its variants achieve optimal
sub-linear static regret, and many extensions bound dynamic regret [34 35 36 37 38, 39). With non-convex losses cossenncommon common in deep networks- -eerret minimization is Na-hard 40). Follow-the-
Perturbed-Leader (FTPL) algorithms 41 42] achieve sublinear static regret for online non-convex learning (with an oracle), but are not well-suited for dynamic distributions. To address this FTPL-D+ extends FTPL for dynamic environments using ensemble and aega-algg [40]. In this work, we adopt FTPL-D+ to effectively manage the dynamic shifts in the novice agent's 's behavior policy.
3 Problem Formulation
We formulate learning from active human intervention with evolving policy trajectorics as an online learning problem. Here, we first model agent-environment interaction as a Markov Decision Process, and then define the online learning problem as dynamic regret minimization.
The interaction between the environment and policy can be formulated as Markov Decision Process (MDP), denoted by M, and M = (S, p, r, y,d)), do), where S is the state space, AA is the action space, p is the transition probability function, r S x A + R is the reward function, 7 E (0, 1) is the discount factor, and do the distribution given state in S. Conventional Reinforcement Learning (RL. aims to find policy that maximizes the expected return: E, 1 7'r(st, at)), where E, denotes the expectation under the distribution induced by a, ~ x(-(sc), S141 ~ p(-(st,,,) a.). The distribution of policy can be defined by its
initial state 80- A policy π (als) takes action a in A
3

---

discounted state-action visitation distribution (s, = (1 - (1-1))((|s) Ei= P'((s = s|T). The > unique stationary policy that induces visitation d(ss, a) is given by {[|ss) d(s, a)/ Σ. a).
With human expert involvement, denote the human expert policy as T (a|s) and the learning agent's 's novice policy as x (a]s). We also use a Boolean indicator [(s, a² ): S x A - {0, to
indicate when the expert intervenes (s, a²) 1). Thus, the actual action interacts with environment is (8, a )a" - (1 [(s, ax)) ))a with corresponding behavior policy N xB:
π 3 (a = T(8, a )m (a|s) + (1 - T(s, a ))x² (a|s).
Since the novice policy x updates during training, its distribution d changes over time. Con- sequently, the actual distribution d8h of behavior policy T also changes over time. This evolving dynamics motivate to formulate the learning objective dynamic regrel minimization from an online learning perspective. Specifically, at each training round i, the novice policy म. interacts with the environment to collect data D.N. while human expert intervenes with policy π expanding the entire human expert dataset D. The combined dataset D.d D. U D can be viewed as being sampled from the distribution de induced by the behavior policy ni The learning objective is to minimize the dynamic regret:
M RD = Σ
M (( N D. D - Σ
( D.P. D.
(I)
where l denotes the imitation loss under the empirical mixture distribution estimated by sampling from D at round i (see details in Sec 4.1), and ग. arg ming (I(T, D. ". is the best policy in hindsight round i. Unlike static regret that assumes static policy learned from all past data, this dynamic regret objective allows us to continuously compare the current policy with the best policy under the latest data, and thus capturing the dynamic of evolving trajectory distribution.
4 Method
FaithDalL tackles learning from human intervention through two core components: an unbiased objective to ensure faithful imitation of the human expert, and an online non-convex learning algorithm that minimizes dynamic regret of the faithful imitation loss to adapt to evolving data distributions.
4.1 Faithful Imitation Objective with Behavior Trajectory
Previous methods that learn from human intervention typically perform off-policy imitation learning on the behavior distribution d8, i.e., they imitate the actual trajectories from both expert and novice policy. However, some novice trajectories may be poor demonstrations, human reaction delays during interventions can lead the agent to execute incorrect actions just before the human takes over. We therefore focus on faithfully imitating only the human expert, while still leveraging novice data for data efficiency.
We first define our imitation objective as minimizing Kullback-Leible divergence from agent policy to human policy distribution dd (s, a), i.e.
DKL (d (8, a) a (s,a)) Beeayaar log dt a) d (s, a)
Given that expert data is often sparse, prior works have incorporated suboptimal data to improve data
efficiency 22 28
30 3T. However, these approaches typically rely on coverage assumption that the suboptimal data visitation covers that of the expert (27]
29
29 32 which is not always
guaranteed 332 To deal with this, we introduce behavior distribution d
and reformulate the
objective as
log (s d (s,
+ +log d (3( as) dH (8, a)
(x ())a) (s,a) = Elaaaaer
(3)
dB (8,
= =a
log d (s)
+DKL(d (s. (s,
(4)

---

Since d is a mixture of novice policy distribution d and expert policy distribution d" we have Therefore, the coverage requirement satisfied at all times. To optimize this objective, we impose the Bellman-flow constraint Eas d(s, a) =
d > 0 wherever dl
7))
Ea d(s', (')p(s)s s', ( on states [43] and apply Lagrangian duality and convex conjugate [25] This reformulates the intractable imitation objective into a tractable optimization over a value function, using a proxy reward derived from the ratio d a)fa" (s, a). Specifically, minimize:
V = arg min (1 - Bodd V(s) + R (s,a)aan [S" (TFV(s, V())))
(5)
V
where V* denotes the optimal value, f denotes the convex conjugate of the KL divergence, and (s, a) F(s, a) y. -p) (s')) the Bellman operator defined using proxy reward
F(s, a) given by:
d (s, d (8,
1 c* (s,a)
F(s,a) log
log
c* (s,
where et is the optimal discriminator derived by:
max (s,a) [log c(s, a)] + (x.a) [log(1 - c(s, a))].
Although the optimization for Eq. (5) doesn't contain the policy, actually learns an implicit optimal policy through the visitation distribution ratio between the optimized and behavior policy. Therefore, we can extract the policy using weighted behavior cloning 44), where the optimal weight w (s, can be calculated by the ratio between the optimal policy distribution d' (s, a) and d (s, a), i.e.
w* a) = d' a)
= max (0, (f') (TAV" (*))))
d (s,
Then the optimal policy x¹ can be found with the optimal weight w (8, a). i.e.,
ा arg EaaxxE (w" (s, a) log x (a]s))
In summary, by first solving for the value function V via Eq. ()) and performing weighted behavior cloning using the derived weights w ws, a), we can learn the optimal policy π².
Empirical Faithful Imitation Objective. In practice, we do not have access the full distributions dl and d8,, so we sample from the datasets D.d and DB collected at each training round i. The policy be estimated by minimizing the empirical weighted behavior cloning loss, i.e.,
(( N D., ) - -
[wi (8, a) log n. (a|s))
(8)
where weight (f a) is derived from the optimized value function V by Eq. (6) in round i. This loss directly measures how well the novice policy π. imitates the human expert with the importance weighting. Theoretical derivations and implementation details can be found in Appendix A.
4.2 Learning from Evolving Behavior Trajectory with Dynamic Regret Minimization
Existing numan-in-the-loow imitation learning methods either assume convexity and target static regret using algorithms like Follow-the-Leader (FTL) (e.g.. DAgger [12), EIL 19). or employ deep neural networks leading to non-convex optimization landscapes (e.g., HACO II, PVP ( IWR (10). While the latter operate online, they often update policies by learning from the aggregate of all historical data, which is quite similar to the idea of optimizing for static regret using the Follow- carturbed-Leaded (FTPL) algorithm [42). That is, they optimize policy on the accumulated historical data at every round i. However, static regret is insufficient as overlooks the evolving quality of DB and can bias the policy towards outdated trajectories.
To address this, we employ FTPL-D+ 40), an ensemble learning framework designed for non-come convexx online learning to optimize for dynamic regret (Eq. (I. FTPL-D+ extends the classical Follow-the Perturbed-Leeat Leader (FTPL.) algorithm to non-stationary environments where the loss landscape evolves over time. It maintains set of K FTPL learners, each associated with a different rolling window size, to capture different temporal dynamics. This mechanism implicitly smooths out the non-stationarity of the data distribution and allows the overall strategy adapt varying timescales of change. Such
5

---

multi--nterrvl ensemble is particularly well-suited for minimizing dynamic regret, as it balances short-term adaptivity (via thort-horizon learners) and long-term stability (via long-horizon learners). Theoretical results of FTPI.-D+ [40] show that this structure achieves near-optimal dynamic regret bounds in general non-convex settings.
Building on this ensemble structure, each individual learner follows a standard restarting strategy: the time horizon M is partitioned into intervals of length т, and FTPL is restarted at the beginning of each interval. Given a policy parameterization 0x. the update at round i is:
Σ e(T, D,: DD o, 0z 1=K
1+1
arg min
where /4, = T 1)/r] + the beginning of the interval, Oz is the parameter of novice policy, and o, E R2 a random perturbation vector, whose components are typically sampled i.i.d. from an exponential distribution Exp(1) with parameter 7) > 0 at each round i.
properly chosen allows the policy to track the optimal behavior distribution. Since the optimal
K is unknown, FTPL-D+ maintains an ensemble of K base learners (novice policies { i } 1==1) each
associated with different interval parameter тæ: 2 for k 1, ,,..., K [log2 M J + A
meta-algorithm, Hedge [5]. adaptively assigns weights a;, to each learner. At each round i, after observing new data D and D.H, the weights are updated:
OLe p PE(T PE(T.....
0,+1,
E Ci. PF( P..
(10)
where p > is a learning rate controlling the sensitivity of the weight update. A novice policy is then sampled according to the distribution {0 (0 to interact with the environment in the next round. This enables dynamic adaptation to shifting behavior distributions.
Our Faithful Dynamic Imitation Learning (FaithDalL) algorithm is presented in Algorithm | The algorithm firstly initializes the ensemble of K novice policies, their adaptive weights 01, and interval parameter т = 2 (Lines 1-2). In each round i, novice policy T is sampled based on the current weights {a } to interact with the environment (Line 4). During the interaction, a human expert provides real-time interventions when necessary. We collect new novice trajectory data D.W
Algorithm Faithful Dynamic Imitation Learning from Human Intervention (FaithDalL)
1: Set K llog M| 1, D Ø
2: Initialize K novice policies ( (Ti,ARE=1 weights 0n < K interval parameter ти = 2
3: for i = 1 to M do
D Online learning rounds
4: Sample policy index k
((agorical((as.1]]
5: Execute T. to interact with the environment and collect novice policy data D.
6: Human expert provides real-time interventions, collect intervention data D. m.w
Update D. = D.L U D.,.new
7:
8:
Construct behavior data D. = DD, D
9:
for & to K do
Evaluate loss of each policy current data Compute imitation loss {( D. D by Eq. (6) and ( end for
10:
11:
12:
Update ensemble weights based Eq. 10
13:
for = 1 to K do
Update novice policies over each interval
14:
Let Prz Th [(i 1)/Tk] +1
15:
Sample perturbation oit ~ Exp(n)
16:
Update T. = arg ming (E) (( ( Dff) + oT.02)
17:
end for
18:
end for
return Policy N 1, where k* arg max 0'+11,

---

Table 1: Comparison of different approaches in MetaDrive-Keyboard and CARLA-Wheel
MetaDriiv-Keyboarr
CARLA-Wheel
Training
Testing
Training
Testing
Method
Total
Episodic Safety Cost
Human
Total
Episodic Return
Success Rate
Human Data
Total Data
Route Comp.
Success Rate
Safety Cost
Data
Data
PPO
IM
26.4K
327.33
3.31
0.76
1M
0.24
0.0
TD3
IM
1.90K
317.45
1.44
0.58
1M
0.11
0.0
Human
374.73
0.39
0.98
0.99
1.0
BC
30K
129.60
17.40
0.12
5K
0.42
0.20
HG-DAgger
7.5K
30K
143
297.60
7.05
0.59
6.8K
24K
0.64
0,47
IWR
6.1K
30K
112
327.32
9.16
0.75
5.7K
24K
0.69
0.60
HACO
9.9K
30K
76
239.41
4.29
0.26
4.8K
24K
0.52
0.40
PVP
7.0K
30K
54
343.86
2.51
0.85
6.6K
24K
0.92
0.73
FaithDaIL
4.8K
30K
55
354.35
1.47
0.91
4.2K
24K
0.95
0.91
Table 2: Ablation Study in MetaDrive-Keyboard and CARLA-Wheel,
MetaDrive-Keyboard
CARLA -hheel
Method
Episodic Return
Episodic Safety Cost
Success Rate
Route Comp.
Success Rate
FaithDalL w/o DRM
346.06 #5442
2.29 #0.29
0.87 =0.0
0.91 20.0
0.81 #0001
FaithDall. w/o FOP
350.26 13:57
1.78 2051
0.89
2005
0.86 20.4
0.73
2007
FaithDall. (Ours)
354.35 1343
1.47 20.28
0.91
2004
0.95 20.02
0.91 20.03
and accumulate intervention data D... into D. (Lines 5-7). The behavior data Djd is constructed by merging the novice data and human intervention data, i.e., D.P =DN UD" (Line 8). We then compute the imitation loss of each novice policy by Eq. (5) (6). and (8) (Lines 9-11). The ensemble
weights for the next round is updated by Eq. (TO) (Line 12). Each novice policy n is updated via FTPL using data from its corresponding interval, defined by 1',, = Lx 1)/TRJ + The update solves regularized imitation loss minimization over the data from rounds j = Mrr, to i, with exponential perturbation oi (Lines 13-17). Finally, after M rounds of training, FaithDalL returns the novice policy with the highest ensemble weight (Line 19).
Experiments
5.1 Experimental Setting
challenging driving simulators: MetaDrive Safety Benchmark [46] and CARLA Town01 [4]] MetaDrive a lightweight simulator in which agents must navigate vehicles safely in dense traffic. We created training and testing sets of 100 distinet scenarios to assess generalization. CARLA, a widely used autonomous driving platform, provides realistic urban settings in Town01 with low-level continuous control (acceleration, braking, steering). To evaluate robustness to different observation modalities, we use sensory state vectors in MetaDrive and bird's-ce s-eye view images CARLA. Further details are in Appendix C.
Environments. We evaluate our approach
Evaluation metrics. In MetaDrive, we use episodic return (cumulative reward per episode), safety cost (simulator-defined safety score), and success rate (reaching destination). Since CARLA does not provide reward or safety scores, we use route completion (distance traveled / total route length) and success rate. We also report human data and total data usage for training imitation algorithms.
Experiment Procedure. Three college students (aged 20-25 with valid driver's licenses) participated in the human-in-the-loon experiments. Prior to training, they practiced until proficient (achieving 2955 success over episodes). Fach participant then supervised a learning agent for approximately one hour per simulator, intervening via keyboard in MetaDrive (Appendix C, Fig. C.I) or Logitech

---

G923 racing wheel in CARLA (Appendix C, Fig. C.2) upon the agent risking safety/traffic violations or significantly deviating from human norms. Training scenarios were randomly selected from 100 diverse environments in MetaDrive and 25 varied routes with different start/end points, lighting, and weather conditions in CARLA Town01.
Implementation Details. Our implementation builds upon open-source repositories of ODICE [25 PVP [9] and FTPL-D+ [40]. Implementations of PPO and TD3 utilize Kable-Baselines? [48], while other HIL baselines use official implementations where available 9 [.]. During testing, agents operate autonomously without human intervention. Experiments were run on a machine with an Nvidia GeForce RTX 3070 Ti Laptop GPU and an Intel Core i7-12700FF CPU, supporting real-time simulation and training. syyeerparaaeter and other details are in Appendix D.
5.2 Baseline Comparison
Baselines. We compare our method with standard RL baselines of PPO 49 and TD3 [0)] We also collected 30K high-quality human expert demonstrations with 98% success rate to train Behavior Cloning (BC) policy 22. For human-in-ther -ite-inn-tee-ooppp baselines. we compare with Human-Gated DAgger
HIG-DAgger) g Intervention Weighted Regression (IWR) [0], Human-AI Copilot Optimization (HACO) 11, and Proxy Value Propagation (PVP) (9.
Performance Comparison. Table T summarizes the performance in MetalDrive and CARLA. RI. methods (PPO, TD3), while capable of learning, require vast amounts of data (IM steps) and incur significantly higher total safety costs during training compared HIL methods. This highlights the benefit of human guidance in reducing unsafe exploration and improving sample efficiency.
Among HIL methods, our FaithDalL consistently outperforms baseline methods. In MetaDrive (Table I Left), it achieves the highest episodic return and success rate, and the lowest episodic safety cost, despite using the least human intervention data (4.8K steps, 16% of total). HG-DAgger performs poorly due to heavy reliance on limited human data. IWR improves with policy data but suffers from errors in early suboptimal trajectories. HACO shows lower testing safety cost but underperforms in task success, potentially due to inaccurate value estimation. PVP performs well among baselines but imitates the mixed behavior policy, making it susceptible to novice mistakes.
Similar trends are observed in the more visually complex CARLA environment (Table I Right), where FaithDalL achieves the highest route completion and success rates with the fewest human interventions. These results underscore FaithDalL's strength: by performing faithful imitation to expert and dynamically tracking high-quality data segments, it unbiasedly learns the human policy while avoiding error propagation from self eelf-generate trajectories.
5.3
Ablation Study
We conducted ablation studies to assess effectiveness of FaithDalL's key components: Dynamic Regret Minimization (DRM) and the Faithful Off-policy imitation learning module (FOP).
FaithDalL w/o DRM: Removes the FTPL-D+ ensemble framework, reducing the algorithm to training on all historical data with the FOP objective (akin to static regret minimization).
FaithDalL w/o FOP: Replaces the DICE-based faithful imitation objective (Eqs.. 7 8 with standard behavior cloning on the mixed novice/expert data, but with the DRM component.
Table 2 summmrrzees performance comparison. We observe that removing either DRM or FOP leads to performance degradation, especially in episodic return and safety cost in MetaDrive, and route completion and success rate in CARLA. Without DRM, training on all historical data (akin to static regret) impedes adaptation the improving policy and complicates density ratio estimation for FOP. Without FOP, directly imitating mixed data exposes the agent to suboptimal novice actions due to intervention latency, preventing faithful expert imitation. These results confirm benefits of combining dynamic regret minimization and faithful imitation objective.
5.4
Case Studies
To qualitatively evaluate performance, we conducted MetaDrive case study analyzing agent testing trajectories under six distinct road conditions. Figure iillustrates bird's-eye view snapshots of agents'
8

---

HG- HG-DAgger
IWR
HACO
PVP
FaithDalL (Ours)
(a)
(b)
(c)
(d)
(e)
(f)
Figure 2: Qualitative comparison agent trajectories generated by different Human-in Houman-in-the-aae -oop (HIL) methods in MetaDrive scenarios. Each row depicts specific road condition: (a) straight road with static obstacles; (b) straight road with dynamic vehicles; (c) simple curve; (d) roundabout; (e) roundabout with dynamic vehicles; and (f) intersection with dynamic vehicles. The agent is indicated by the green rectangle, its recent path is traced by the green line, and other dynamic vehicles are represented by colored boxes. We can see that both HG-DAgger and IWR fail in scenarios with static and dynamic in (a.b,e.f) and both exhibit erratic trajectories curving roads (c,d). HACO performs showing unstable behaviors such as wide swings (c,d,e) and lane departures in (a,b). PVP showed risky maneuvers avoid obstacles, resulting in collisions in (f) and deviations in (a,b,).. In contrast, FaithDalL produces the smoothest and safest trajectories, closely matching expert driving.
goes off-road in (b,e) and collides (a.f), while IWR collides worse,
behaviors across these scenarios: (a) straight road with static obstacles; (b) straight road with dynamic vehicles; (c) simple eurve; (d) roundabout; (e) roundabout with dynamic vehicles; and (f) intersection with dynamic vehicles. Videos are provided in the supplemental materials.
Performance on Straight Roads with Obstacles (Static and Dynamic). On straight roads with static obstacles (Figure 24) or dynamic vehicles (Figure 2p), most baselines exhibited critical failures. HG-DAgger overreacted, causing collisions off-road deviations. IWR collided with both static and dynamic elements. HACO showed unstable behavior like wide swings and lane departures. PVP showed risky maneuvers avoid obstacles. In contrast, FaithDalL consistently demonstrated smooth, safe navigation, avoiding all obstacles and adeptly handling dynamic traffic.
Performance on Curves and Roundabouts (Without and With Dynamic Vehicles). On curves and roundabouts without dynamic vehicles (Figure 2 d), most baselines completed the trip but FaithDalL provided the smoothest, most stable trajectories, closely resembling expert driving, With

---

dynamic vehicles (Figure 2)), all baselines failed (veering off-road or colliding). FaithDalL was the only method to successfully and safely navigate around dynamic vehicles while staying on the lane.
Performance at Intersections with Dynamic Vehicles. Navigating intersections with oncoming traffic (Figure 2D proved be the most challenging scenario for the baseline methods. HG-DAgger, IWR, and HACO all resulted in collisions with incoming vehicles due to failure to react appropriately, PVP also failed avoid a collision, being struck by another car. Our method was the only one to exhibit safe, intelligent behavior; it slowed down, yielded, and proceeded safely through the intersection, avoiding any collision or potential danger.
Overall, these case studies highlight the superior robustness and safety of our proposed method across variety of challenging road conditions, particularly in complex dynamic environments where baseline HII. approaches frequently failed.
6 Conclusion and Future Work
In this paper, we proposed FaithDalL, a Faithful Dynamic Imitation Learning framework to learn from human intervention. FaithDalL incorporates policy-generated trajectories to improve data efficiency, while addressing challenges of dynamic data distribution shifts and trajectory deviations caused by intervention latency. FaithDalL the first work that formulates the learning as online non-convex optimization with dynamic regret minimization, thereby is able to track high-quality policy trajectories. We also proposed an unbiased objective for faithful expert imitation from mixed agent/human data via weighted DICE-based approach. Extensive experiments MetaDrive and CARLA demonstrated FalthDall. achieves state-of-the-ee safety and task success with significantly reduced human intervention.
Future work will address remaining challenges, including evaluation on realistic platforms (e-g.. robotic arms) with more participants, improving robustness diverse and suboptimal human in- terventions, and theoretical analysis on imitation capability our method with dynamic regret minimization. We also aim to generalize FaithDalL to broader HIL learning problems (e.g.. multi- modal intervention).
Acknowledgment
This work was supported in part by the Natural Science Foundation of China under Grant 62232004, 62572120, 62472093, 62572246; in part by the Key Research and Development Projects Jiangsu Province (No. NN..BE201100-222 1001-2): in part by the Natural Science Foundation of Jiangsu Province under Grant (No. BK20230024; in part by the Fundamental Research Funds for the Central Universities; and in part by the Big Data Computing Center of Southeast University.
References
[1] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in Al safety. arXiv preprint arXiv: 1606.665655 2016.
[2] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent aligmment via reward modeling: research direction. arXiv preprint 1rX.v:811..07771. 2018.
[3] Tom Everitt, Victoria Krakovna, Laurent Orseau, Mareus Hutter, and Shane Legg. Reinforce- ment learning with corrupted reward channel. arXiv preprint
2017.
[4] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[5] Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedhack-efffcient interactive rein- forcement learning via relabeling experience and unsupervised pre-training. arXiv preprint 20X1v
2021. 2021.
10

---

[6] Zizhao Wang, Xuesu Xiao, Garrett Warnell, and Peter Stone. Apple: Adaptive planner parameter learning from evaluative feedback. IEEE Roboties and Automation Letters, 6(4):7744-7749, 2021.
[7] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[8] Michael Kelly, Chelsea Sidrane, Katherine
and Mykel J Kochenderfer. Hg- dagger: Interactive imitation learning with human experts. 2019 International Conference on Robotics and Automation (ICRA), pages 8077-8083. IEEE, 2019.
[9] Zhenghao Mark Peng, Wenjie Mo, Chenda Duan, Quanyi Li, and Bolei Zhou. Learning from active human involvement through proxy value propagation. Advances in neural information processing systems, 36, 2024.
[10] Ajay Mandlekar, Danfei Xu, Roberto Martin-Martin, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Human-in the-loop -oop-----tt imitation learning using remote teleoperation. arXiv preprint arXiv:2012.06733, 2020.
[11] Quanyi Li, Zhenghao Peng, and Bolei Zhou. Efficient learning of safe driving policy via human-ai copilot optimization. arXiv preprint arXiv:2202. 10341, 2022.
[12] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- rional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[13] Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel S Brown, and Ken Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning, arXiv preprint arXiv:2109.08277: 2021.
[14] Jiakai Zhang and Kyunghyun Cho. efficeett imitation learning for end-to-end autonomous driving. arXiv preprint arXiv:1605.06450, 2016.
[15] Ryan Hoque, Ashwin Balakrishna, Carl Putterman, Michael Luo, Daniel S Brown, Daniel Seita, Brijen Thananjeyan. Ellen Novoseller, and Ken Goldberg. Lazydagger: Reducing context switching interactive imitation learning. In 2021 IEEE 17th international conference on automation science and engineering (case), pages 502-509. IEEF, 2021.
[16] Kunal Menda, Katherine bayesian approach safe imitation lcarning. In 2019 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 5041-5048. IEEE, 2019.
and Mykel J Kochenderfer. Ensembledagger. A
[17] Emilien Biré, Anthony Kobanda, Ludovic Denoyer, and Rémy Portelas. Efficient active imitation learning with random network distillation. In The Thirteenth International Conference on Learning Representations, 2025.
[18] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661-668. JMLR Workshop and Conference Proceedings, 2010.
[19] Jonathan Spencer, Sanjiban Choudhury, Matthew Barnes, Matthew Schmittle, Mung Chiang, Peter Ramadge, and Siddhartha Srinivasa. Learning from interventions: Human-robot interac- tion as both explicit and implicit feedback. In 16th robotics: science and systems, RSS 2020.
MIT Press Journals. 2020.
[20] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances neural information processing systems, 32, 2019.

---

[21] Ilya Kostrikov, Kumar Krishna Agrawal. Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actorr artticc Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint 1uii 2018.
[22] Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
[23] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[24] Ilya Kostrikov, Ofrr Nachum, and Jonathan Tompson. Imitation learning via off- off-policy distribu- tion matching, arXiv preprint 1912.00911 2019.
(25) Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. Odice: Revealing the mys- tery of distribution correction estimation via orthogonal-gradien update. arXiv preprint 200.00440.. 2024.
[26] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song. and Stefano Ermon. Iq-learm Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028 4039, 2021.
[27] Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou Off-policy imitation learning from observations. Advances in neural information processing systems, 13:12402-122413, 2020.
[28] Hana Hoshino, Kei Ota, Asako Kanezaki, and Rio Yokota. Opirl: Sample efficient off-policy inverse reinforcement learning via distribution matching. In 2022 International Conference on Robotics and Automation (ICRA), pages
IEEE, 2022.
[29] Yecheng Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Versatile offline imitation from observations and examples via regularized state-occupancy matching. In International Conference on Muchine Learning, pages 14639-14663. PMLR, 2022.
[30] Geon-Hye Beon-HHyeon Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang. Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2021.
[31] Geon-Hyeong Kim, Jongmin Lee, Youngsoo Jang, Hongseok Yang, and Bue-Eung Kim. Lobs- dice: Offline learning from observation via stationary distribution correction estimation. Ad- vances in Neural Information Processing Systems, 25:8252-8264 2022.
[32] Harshit Sikehi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. arXiv preprint arXiv:2302.085500 2023.
[33] Shai Chalev-Shwarte et al. Online learning and online convex optimization. Foundations and Trends@ in Machine Learning, 4(2):107-194, 2012.
[34] Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference machine learning (icml-03), pages 928-936. 2003.
[35] Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends@ in Optimization, 237-4): 2016.
[36] Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Opera- lions research, 6325):1227 2015.
[37] Dheeraj Baby and Yu-Xiang Wang. Online forecasting of setaal-variation-buuneee sequences. Advances in Neural Information Processing Systems, 32, 2019.
[38] Peng Zhao and Lijun Zhang. Improved analysis for dynamic regret of strongly convex and smooth functions. In Learning for Dynamics and Control, pages 48-59, PMLR, 2021.
[39] Yuanyu Wan, Lijun Zhang, and Mingli Song. Improved dynamic regret for online foank-wolfe. In The Thirty Sixth Annual Conference on Learning Theory, pages 3304-3327. PMLR, 2023.
12

---

[40] Zhipan Xu and Lijun Zhang. Online non-convex learning in dynamic environments, Advances in Neural Information Processing Systems, 37:51930-51966 2024.
[41] Naman Agarwal, Alon Gonen, and Elad Hazan. Learning non-convex games with an optimization oracle. In Conference on Learning Theory, pages 18-29. PMLR, 2019.
[42] Arun Sai Suggala and Praneeth Netrapalli. Online non-convex learning: Following the perturbed leader is optimal. In Algorithmic Learning Theory, pages 845-861. PMLR, 2020.
[43] Alan S Manne. Linear programming and sequential decisions. Management Science, 6(3):259-
267, 1960.
[44] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. advantage-weighter regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
[45] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
[46] Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 45(3):3461-3475 2022.
[47] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. Conference on robot learning, pages 1-16. PMLR, 2017.
[48] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of machine learning research, 22(268):1-8, 2021.
[49] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
[50] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction elearly state our key contributions, which are reiterated with bullet points in the introduction. These claims fully aligned with the theoretical framework and empirical results presented in the paper, and accurately reflect the scope and limitations of our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See limitation and future work paragraph in Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification: While we provide detailed derivations and optimization formulations for our learning objective (see Appendix A), these are used define the training loss rather than to establish formal theoretical results. We do not state any theorems or rely on explicit assumptions that require formal proof. As such, this item does not apply our submission. Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed description of our algorithm in Section Additionally. we include our implementation in the supplementary material and provide a link in the abstract for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
15

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include our implementation the supplementary material and provide link in the abstract for reproducibility. All code is anonymized.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/guidess http:///uuuiss/ooouuuuiniissssss Polliicyo)yy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.c//ppl http:liiiiguidee//deeeeeeeeeeeee
setommmsssioniiii ssionpolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See Section 5.1 and Appendix C for experimental details. Full details are available the code.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
The performance our method and the baselines is reported in Table while ablation study results are presented in Table
Justification: the results Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 5.1 and Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/ http:/://///rii./eeee httparrppps.ccpp...c/ccctttcttut
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethics and conform it
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
17

---

Justification: Our work proposes new algorithm for imitation learning from human interven- tion. primarily intended for theoretical and empirical evaluation simulated environments. It does not involve deployment, user data, or application to sensitive domains such as health- care or surveillance. Therefore, we believe the societal impact of this work currently limited. We encourage future users of this method to carefully consider its implications when applied in real-world settings.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work focuses on an imitation learning algorithm and does not release any models datasets that carry high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Section 5.1
18

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release the full implementation of algorithm in the supplemen- tary material, and the corresponding source code is available with anonymized URL of httpr//mmouu.opo...ccccc/////app Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Our study involved human participants who provided real-time interventions during training episodes. We describe the participant protocol and experimental setup in Secciinn 5.1 of the main paper and include additional details such as the instructions provided participants, screenshots of the interface, and compensation information in the supplementary material. All participants were compensated at or above the local minimum wage, and their involvement entirely voluntary.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
19

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [Yes]
The study was approved by the university institutional review board (IRB), which is stated in the Appendix F. Participants in the study received monetary compensation for their involvement. They were free to pause or withdraw from the experiment any time if they experienced any discomfort. All tasks were conducted in virtual simulation environment, ensuring no risk of physical harm. Each experimental session was limited to a maximum duration of 1.5 hours, followed by mandatory rest period of at least three hours. Throughout the training and data processing stages, no personally identifiable information collected exposed in the dataset or trained models.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs were not used in the development of the core methods, experiments, results of this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM for what should or should not be described.
20


Reviews:
[{"review_id": "L37vm5zO23", "reviewer": "Reviewer_xuad", "summary": "The paper studies the problem of imitation learning with online expert annotation, in which the learner can access the environment and generate states, meanwhile the expert can provide action for each state. The paper formulates it as a dynamic regret minimization problem, and proposes a learning approach based on the follow-the-perturbed-leader (FTPL) paradigm. Experimental results under autonomous driving benchmarks are provided to verify the effectiveness of the proposed approach.", "questions": "Please feel free to respond to the points in the weaknesses part.", "limitations": "The discussions of technical limitations seem to be lacking.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- Strengths:\n\n1. The paper suggests studying dynamical regret minimization in imitation learning, which is less-studied to my knowledge. This may inspire online learning theory researchers to consider imitation learning as a practical applicational area for dynamic regret studies. \n\n- Weaknesses:\n\n1. The learning problem may not be properly formulated. The dynamic regret actually assumes that the optimal policy changes over time, while in the paper, the optimal policy remains the same as the oracle policy. Even though in Eq. 1, $\\pi^*$ seems to be related to time index $i$, while the paper does not assume that the oracle's policy can change. Instead, the paper claims that the reason to consider dynamical regret is that the learner's policy can change. While this situation is common for all online learning problems, even when the static regret is considered. Actually, the motivation for the classical DAGGER algorithm is to consider the effect of changing learner policy. \n\n2. In the abstract and introduction part, it is said the expert reaction delay can be an issue. While the paper seems to lack further discussions on the problem.\n\n3. The notion of faithfulness needs further clarification. It seems that the paper utilizes this as a measurement on how good the learner can imitation the expert. But the relationship of faithfulness to the dynamic regret problem still lacks clear explanations. \n\n4. The paper lacks theoretical analysis on regret guarantees, which is important for online learning researches.", "was_revised": true, "final_justification": "My major concerns have been addressed in the rebuttal. Thus my score is raised accordingly.", "rebuttal": "### W1: Justification for Dynamic Regret\n  We clarify that our problem is not \"imitation learning with online expert annotation\". We study human-in-the-loop imitation learning with active human intervention, in which, human experts supervise the agent's execution, and intervene to make corrections once errors occur. The key difference of our human intervention from \"online expert annotation\" is that we learn from a  hybrid of expert trajectories as well as the changing agent-generated trajectories (which we call novice data), while \"online expert annotation\" assumes stationary sample demonstrations. *This changing distribution of imitation data,  is the key reason why we adopt dynamic regret.*  In contrast, DAgger uses static regret because its data is consistently sampled from the fixed human expert policy distribution.\n\n  In many classical dynamic‑regret formulations, the data distribution and loss function or environment dynamics  (summarized as loss landscape) change over time. As a result, the optimal policy that minimizes instantaneous loss often varies from one round to the next. However, *we argue dynamic regret is designed for problems with non-stationary loss landscape. Whether the optimal policy changes over time is not the determining factor for using dynamic regret.* [1] studied dynamic regret when optimal policy remains unchanged but  loss or data distribution changes. In this setting, the changing loss landscape introduces complexity beyond what is captured by static regret.\n\n  Therefore, although our faithful imitation objective (Section 4.1) assumes that the theoretical optimal policy is fixed, we can still formulate our problem as a dynamic regret problem because of the complexity induced by the changing distribution of imitation data.\n\n[1] On Online Optimization: Dynamic Regret Analysis of Strongly Convex and Smooth Problems (AAAI 2021)\n\n\n\n\n\n\n\n\n### W2: Link to expert reaction delay\nWe thank the reviewer for this comment. We will clarify the connection in the revision.\n\nExpert reaction delay creates suboptimal \"deviation trajectories\" where the agent acts incorrectly just before a human intervenes. Learning directly from this data leads to unfaithful imitation. To address this, our faithful imitation objective optimizes the KL divergence between the agent's policy and the expert's policy. Our method assigns lower proxy rewards to suboptimal trajectories, which in turn gives them lower weights during the weighted behavior cloning update. This process mitigates the negative impact of intervention delays, ensuring the agent learns to imitate expert policy faithfully.\n\n\n\n\n\n\n\n\n### W3: faithfulness clarification\nThe notion \"faithfulness\" is mainly about imitating human experts only, while still leveraging trajectory data from both human experts and the learning policy for data efficiency. This is to alleviate the negative impact of the suboptimial of learning policy trajectory data, and achieved by a faithful imitation objective that optimizes the difference between the behavior policy and the expert policy (see Section 4.1) with a discrimilator.  \n\nWe clarify that our target is to imitate a hybrid dataset consisting of human expert trajectories and the learning policy trajectories for data efficiency. This hybrid dataset brings two key callenges: 1) changing distribution and 2) deviation trajectories caused by expert reaction delay. The changing distribution is addressed by formulating the whole imitation learning problem as dynamic regret, and the effect of deviation trajectories addressed by our proposed faithful imitation objective.  This faithful imitation objective is used as a loss in dynamic regret. \n\nFurthermore, we argue that dynamic regret promotes more faithful imitation compared to static regret. Our approach allows the model to track the most recent and highest-quality policy trajectories, which progressively converge toward the expert's policy. This avoids the bias from outdated, suboptimal data that would dilute a static regret objective, thereby leading to a more faithful imitation.\n\n\n\n\n\n\n### W4: Theoretical Analysis\n\nThanks for the comments, we will provide the following theoretical analysis on regret guarantees. Our theoretical analysis shows that our algorithm is no-regret and can eventually converge to the optimal policy. \n\n(We encountered a rendering issue with long equations in the OpenReview rebuttal system. To address this, we have broken them down into smaller terms, which required introducing new notations (e.g., $k = i-1$) and may harm the paper's readability.)\n\nWe aim to prove that our FaithDaIL is no-regret and can eventually converge to the optimal policy, i.e., \n\n$\n\\frac{1}{M} \\mathbb{E}[\\mathcal{R}_D] \\to 0\n$\n\n\nWe first give some assumption:\n\n+ **Bounded Parameter Space**: The policy parameter space is effectively bounded due to regularization and optimizer constraints in practice.\n+ **Lipschitz Loss**: The imitation loss $\\ell_i(\\pi)$ is Lipschitz in $\\theta_\\pi$, assuming (1) $\\pi(a|s)$ is Lipschitz, (2) importance weights $\\omega_i^\\star(s,a)$ are bounded, and (3) $\\pi(a|s)$ is bounded away from zero.\n\nFollowing the analysis from FTPL-D+, the **expected dynamic regret** of FaithDaIL satisfies:\n\n$\n\\mathbb{E}[\\mathcal{R}_D] \\le \\mathcal{O}\\left( M^{\\frac{2}{3}} (V_M + 1)^{\\frac{1}{3}} \\right),\n$\n\nwhere:\n\n- $M$ is the number of rounds,\n- $V_M =  \\sum_{i=2}^{M} \\max_{\\pi \\in \\mathcal{K}} |l(\\pi, \\mathcal{D}_i^B, \\mathcal{D}_i^H) - l(\\pi,\\mathcal{D}_k^B, \\mathcal{D}_k^H)|$, $k=i-1$ is the total variation of the loss sequence.\n\nTo establish that the dynamic regret is sublinear, **it suffices to show that $V_M$ grows sublinearly with $M$**. We formalize the implicit intervention rate $\\beta_i$ and state the core assumption underlying its use:\n\n+ **Definition (Implicit Intervention Rate $\\beta_i$)** :\n\n$\n\\beta_i = \\mathbb{E}_{\\pi^N_i } \\left[ \\frac{|\\mathcal{D}_i^{H,\\text{new}}|}{|\\mathcal{D}_i^{H,\\text{new}}| + |\\mathcal{D}_i^N|} \\right]\n$\n\n+ **Assumption (Intervention Rate Decay)** : This reflects the intuition that as the agent improves, fewer human interventions are needed over time.\n\n$\n\\lim_{i \\to \\infty} \\beta_i = 0\n$\n\nTo relate $\\beta_i$ to the trajectory distribution shift, we invoke the following result (adapted from DAgger, Lemma 4.1):\n\n+ **Lemma (Distribution Divergence Bound)**: With $T$ as the MDP horizon,\n\n$\n\\|d_i^B - d_i^N\\|_1 \\le 2T\\beta_i\n$\n\nWe now bound the single-step loss variation $\\Delta_i = \\max_{\\pi} | \\ell_i(\\pi) - \\ell_{i-1}(\\pi) |$.\n\n$\n|\\ell_i(\\pi) - \\ell_{i-1}(\\pi)|\n\\leq |A - B| + |B - C|\n$\n\nwhere $k=i-1$ :\n\n$\nA = \\mathbb{E}_{(s,a) \\sim d_i^B}[w_i^\\star(s,a) \\log \\pi(a|s)]\n$\n\n$\nB = \\mathbb{E}_{(s,a) \\sim d_k^B}[w_i^\\star(s,a) \\log \\pi(a|s)]\n$\n\n$\nC = \\mathbb{E}_{(s,a) \\sim d_k^B}[w_k^\\star(s,a) \\log \\pi(a|s)]\n$\n\n\n\n1. **Term 1 (due to distribution shift):**：\n   \n$\n   \\left| A-B \\right| = \\left| \\int_{(s,a)\\in\\mathcal{S}\\times\\mathcal{A}}[d_{i}^B(s,a) - d_{k}^B(s,a)] w_i^\\star(s,a) \\log \\pi(a|s) ds  da  \\right|\n$\n\n$\n\\leq N_1 \\cdot N_2 \\cdot \\|d_i^B - d_k^B\\|_1 \n$\n\nwhere: \n\n$\nN1 = \\|w_i^\\star \\|_{\\infty}\n$\n\n$\nN2 = \\|\\log \\pi\\|_{\\infty}\n$\n\n2. **Term 2 (due to weight shift):**：\n \n$\n   \\left| B-C \\right| = \\left| \\int_{(s,a)\\in\\mathcal{S}\\times\\mathcal{A}}[w_{i}^\\star(s,a) - w_{k}^\\star(s,a)] d_k^B(s,a) \\log \\pi(a|s) ds  da  \\right|\n$\n\n$\n\\leq N_2 \\cdot N_3 \\cdot \\|w_{i}^\\star - w_{k}^\\star\\|_1 \n$\n\nwhere: \n\n$\nN2 = \\|\\log \\pi\\|_{\\infty}\n$\n\n$\nN3 = \\|d_k^B \\|_{\\infty}\n$\n\n**Step 1: Bounding distribution shift:** Using triangle inequality and Lemma (Distribution Divergence Bound):\n\n$\n\\|d_i^B - d_k^B\\|_1 \\le 2T(\\beta_i + \\beta_k) + \\|d_i^N - d_k^N\\|_1\n$\n\nAssuming the novice policy changes smoothly: \n\n$\\sum_{i=2}^M \\|d_i^N - d_{i-1}^N\\|_1 = o(M)$\n\n and $\\beta_i \\to 0$, we have:\n\n$\n\\sum_{i=2}^M \\|d_i^B - d_{i-1}^B\\|_1 = o(M)\n$\n\n**Step 2: Bounding weight shift:** Assuming that the variation in importance weights is primarily induced by the shift in behavior distributions, and that $V_i^\\star$ evolves smoothly under $d_i^B$, we have:\n\n$\n\\|w_i^\\star - w_{i-1}^\\star\\|_1\n$\n\n$\n\\le L_w \\cdot \\|d_i^B - d_{i-1}^B\\|_1 + o(1)\n$\n\nThus:\n\n$\n\\sum_{i=2}^M \\|w_i^\\star - w_{i-1}^\\star\\|_1 = o(M)\n$\n\nThen under the earlier Lipschitz assumptions, we assume relevant terms of N1, N2, N3 are uniformly bounded.\n\n**Conclusion: Sublinear functional variation.**\n\n$\nV_M = \\sum_{i=2}^M | \\ell_i(\\pi) - \\ell_{i-1}(\\pi) | = o(M)\n$\n\n**Final Result: No-regret Guarantee**\n\nFrom FTPL-D+ analysis:\n\n$\n\\mathbb{E}[\\mathcal{R}_D] = \\mathcal{O}(M^{2/3}(V_M + 1)^{1/3}) = o(M), \\quad\n\\frac{1}{M} \\mathbb{E}[\\mathcal{R}_D] \\to 0\n$\n\nHence, **FaithDaIL is a no-regret algorithm** in non-convex dynamic environments, with the learned policy converging to the round-wise optimal."}, {"review_id": "vynLBqbs0f", "reviewer": "Reviewer_ZERT", "summary": "This work proposes Faithful Dynamic Imitation Learning (FaithDaIL), a framework for Human-in-the-loop imitation learning that enhances data efficiency by integrating evolving novice trajectories through dynamic regret minimization. The authors start their paper by motivating the need for their approach and discussing the drawbacks and limitations of prior work. The authors then formulate learning from active human intervention with evolving policy trajectories as a dynamic regret minimization problem. Afterward, the authors propose FaithDaIL and describe several components, including the loss function and algorithm. Finally, FaithDaIL is evaluated in two challenging driving simulators. The authors find that FaithDaIL achieves SOTA safety and task success with much-reduced human intervention.", "questions": "+ Please respond to the weaknesses noted above.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ This paper deploys their algorithm in very challenging domains. As the algorithm is able to perform well in these domains, as well as require less human intervention, this lends support to the significance of FaithDaIL.\n+ The contributions in this paper are significant and theoretically grounded.\n+ The experiments in this paper utilize real human data collected from participants.\n\nWeaknesses:\n- Figure 2 is difficult to interpret and could use some enhancing and further captioning. The support text is helpful.\n- Could you provide further information regarding the experiment procedure and HIL experiments? Was data for HG-DAgger collected in the same manner?", "was_revised": true, "final_justification": "My final score for this paper is a 5: Accept. The proposed work is interesting and validated well. With the additional clarifications and findings provided during the author-reviewer discussion period, I think this paper would be a strong contribution to NeurIPS.", "rebuttal": "### W1: Figure Caption\nThanks for the comments. We will add the following descripion in the caption for interpretability.\n\nWe can see that both HG-DAgger and IWR fail in senarios with both static and dynamic obstacles (deviates off-road in (b) and (e), collides in (a) and (f)), and have erratic trajectories in curving roads (c) and (d). HACO performs even worse, shows unstable behavior like wide swings in (c), (d), and (e)  and lane departures in (a) and (b). PVP showed risky maneuvers to avoid obstacles, resulting in collides in (f) and deviates in (a), (b), and (e). In contrast, FaithDaIL produces the smoothest and safest trajectories, closely aligning with expert driving across all scenarios.\n\n\n\n\n\n### W2: Details on HIL Experiments\n We provided the HIL experiment details in Appendix C. To improve clarity, we reorganize and revise the experiment details here. The described Human-in-the-Loop (HIL) procedure was used to collect data for HG-DAgger and all baseline methods in the same manner.\n\nHuman Subject Recuitment: Three college students (aged 20–25), each holding a valid driver’s license and with at least two years of driving experience, voluntarily participated in the human-in-the-loop experiments. Participants were recruited through campus-wide advertisements and selected based on their availability, driving experience, and willingness to engage in interactive training tasks. We ensured full transparency by clearly explaining the purpose of the study, the nature of their involvement, and how the collected data would be used. Each participant signed a written informed consent form, indicating their full understanding and voluntary agreement. The study protocol was reviewed and approved by the university Institutional Review Board (IRB), ensuring compliance with ethical standards for research involving human subjects.\n\nHIL Experiments Procedure: Before commencing the experiments, all participants received detailed instructions and underwent a brief training session to familiarize themselves with the simulator interfaces and intervention mechanisms. They practiced for approximately 30 minutes until achieving proficiency, which was defined as attaining $\\ge 95\\%$ task success over 50 consecutive episodes. In the main human-in-the-loop (HIL) sessions,  each participant supervised a learning agent, intervening via a keyboard in MetaDrive or a Logitech G923 racing wheel in CARLA. The objective is to navigate the vehicle safely to the destination. Participants are encouraged to intervene whenever they sense potential danger or behavior misaligned with human norms. To mitigate potential order bias, the presentation sequence of algorithms (for experiments in MetaDrive), simulators, and task scenarios was randomized for each participant. Each human-in-the-loop (HIL) data collection session last approximately one hour (70 minutes for tasks in MetaDrive and 60 minutes for tasks in CARLA). Consequently, each participant dedicated a total of 5 hours to the MetaDrive experiments (which involved interacting with 5 different algorithms). For the CARLA simulator, each participant engaged with our FaithDaIL method for one 60-minute session; performance data for other baseline methods on CARLA were adopted from the findings reported in PVP."}, {"review_id": "TK82d42TyY", "reviewer": "Reviewer_MP88", "summary": "This paper introduces faithful dynamic imitation learning for human in the loop imitation learning. It propose a unbiased imitation learning objectives that  faithfully imitates human behavior and  contains a novel problem formulation by framing HIL as an online learning problem to tackle the non-stationary of the data distribution as agent’s policy improves.  Through experiments on the MetaDrive and CARLA simulators, the paper demonstrates that FaithDaIL achieves state-of-the-art performance in terms of safety and task success, while requiring significantly less human intervention data compared to existing HIL baselines.", "questions": "1. The non-stationarity you address is a common issue that also persists in general reinforcement learning (e.g., when the policy updates change the state visitation distribution). Do you believe the core ideas of your method, particularly the dynamic regret framework, could be generalized to improve sample efficiency or stability in standard RL settings beyond HIL?\n2. Can you explain the rationale behind selecting the FTPL-D+ algorithm for minimizing dynamic regret? Were alternative online learning methods evaluated during this decision process? Additionally, what practical trade-offs—such as the computational overhead introduced by the ensemble—does this choice entail compared to more straightforward approaches?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. As not an expert in HIL, I think the faithful imitation objective is very interesting and the way of converting it to the tracble form using odice through convex conjugate is clever.  For me the basic idea is similar to GAIL and reminds me of IQ-learn. \n2. By observing this instability, the paper employs FTPL-D+ to frame the task as an online learning problem with a dynamic regret objective. This is a much more principled approach than simply aggregating all historical data, as it allows the model not be held back by suboptimal data from early in training.\n3. the claims are supported by comprehensive empirical validation. The method outperforms a suite of baselines on two challenging driving simulators. The ablation studies clearly demonstrate the importance of both core contributions, and the qualitative case studies provide convincing visual evidence of the agent's superior safety and stability.\n\nWeakness:\n\n1.  The FTPL-D+ framework requires maintaining an ensemble of K policies, value functions, and discriminators. This approach seems computationally expensive and may pose scalability challenges, especially as the model complexity or the length of the training horizon (M) increases.\n2. The faithful imitation objective relies on a discriminator to estimate the density ratio, which serves as a proxy reward. I am not sure if this would introduce an adversarial training dynamic similar to that in GANs. The stability of such training could be a practical issue.\n3. The paper would benefit from a more detailed and intuitive explanation of the FTPL-D+ algorithm. While the reference is provided, a brief, self-contained overview of why this ensemble-based, multi-interval approach is well-suited to minimizing dynamic regret would make the paper more accessible to a broader audience.", "was_revised": true, "final_justification": "The author's response has resolved my issue. I will keep my score and recommend accept.", "rebuttal": "### W1: Computational Overhead\n We agree that an ensemble of K policies increases computational overhead. However, based on empirical observations from our implementation, we propose two practical strategies to manage this cost.\n\n First, in practise, we observe that the agent's policy distribution progressively converges toward the expert policy distribution, and the best-performing policies within the ensemble are usually those from recent rounds with shorter time intervals. This suggests that the full ensemble, which is required by the theoretical formulation, may not be necessary in practice. The computational cost can be significantly reduced by maintaining only a smaller subset of learners focused on more recent data, without a major impact on performance.\n\n Second, to address scalability as the training horizon (M) grows, the process can be partitioned into sequential stages in practice. Each stage runs for a shorter horizon and \"warm-starts\" using the best policy from the previous one. This approach prevents the ensemble size from growing indefinitely across a very long training process. We plan to evaluate this staging strategy in future work.\n\n\n\n\n\n\n### W2: Adversarial Training Instability\n We thank the reviewer for this important question. We agree that training stability with a discriminator is a key concern, and we designed FaithDaIL's framework to be fundamentally different from adversarial approaches like GAIL or GANs.\n\n Our method avoids a direct min-max game by decoupling the optimization of the policy and the discriminator. At each round, the discriminator is first trained to help compute a proxy reward and importance weights. The policy is then updated via weighted behavior cloning to imitate behaviors with higher weights, rather than trying to fool the discriminator. This sequential and non-adversarial training paradigm, inherited from the DICE family of algorithms, is inherently more stable [1]. \n  Furthermore, the FTPL-D+ ensemble provides an additional layer of robustness by adaptively managing multiple learners, which naturally mitigates the impact of any single, potentially volatile, discriminator estimation. We will revise Section 4.1 to better clarify this decoupled update process.\n \n[1] DemoDICE: Offline Imitation Learning with Supplementary Imperfect Demonstrations (2022 ICLR)\n\n\n\n\n### W3: Explanation of FTPL-D+\n   We thank the reviewer for the helpful suggestion. We will add the following explanation of the FTPL-D+ algorithm in Section 4.2.\n\n   FTPL-D+ is an ensemble variant of Follow-the-Perturbed-Leader (FTPL), designed to handle non-stationary characteristics where the loss landscape changes over time. It maintains a set of $K$ FTPL learners, each associated with a different rolling window size (e.g., covering the past $h$ rounds), to capture different temporal dynamics. This mechanism implicitly smooths out the non-stationarity of the data distribution and allows the overall strategy to adapt to varying timescales of change. Such a multi-interval ensemble is particularly well-suited for minimizing dynamic regret, as it balances short-term reactivity (via short-horizon learners) and long-term stability (via long-horizon learners). Theoretical results of FTPL-D+ [2] show that this structure achieves near-optimal dynamic regret bounds in general non-convex settings.\n \n[2] Online non-convex learning in dynamic environments (2024 NIPS)\n\n\n\n\n### Q1：Generalization to RL\n\n   We thank the reviewer for this insightful question. We think that the core ideas of our FaithDaIL -- optimize dynamic regret to address non-stationarity distribution --- could benefit in a range of non-stationary RL environments, including：\n  - Non-stationary dynamics, due to environmental changes, multi-agent interactions, or sim-to-real domain shifts: One possible approach is to handle non-stationary environmental changes by formulating an objective to evaluate the accuracy of the dynamic transition model within the dynamic regret framework, which helps the learner adapt to shifts in the environment's underlying dynamics.\n  - Reward function drift, as seen in preference-based or feedback-driven learning: we can solve the non-stationary reward function shift by defining the expected cumulated reward as the objective of dynamic regret and solving it using a method like FTPL-D+\n  - Evolving tasks in continual reinforcement learning: we can define a sequence of task-specific loss functions, where each loss may reflect metrics like task completion performance, and incorporate them into the dynamic regret framework accordingly.\n\n\n### Q2: Rationale of using FTPL-D+\n\n   We clarify the motivation of using FTPL-D+ as follows. In the domain of online non-convex learning, previous approaches to minimizing dynamic regret often rely on prior knowledge about the dynamics of the loss landscape or require computationally intractable sampling oracles. Therefore, they cannot be used in our HIL-IL problem due to the unknown dynamics.\n\n  In contrast, FTPL-D+ does not depend on such prior knowledge, and its underlying offline optimization oracle is much more practical—stochastic gradient descent can serve as an effective approximation in practice and often leads to near-global optima. Given these advantages, we believe FTPL-D+ is currently the most suitable and only choice for addressing our problem setting.\n\n  We clarified the computational overhead may not be a issue in practice (see Respone to W1)."}, {"review_id": "669RC6sjmD", "reviewer": "Reviewer_DSG8", "summary": "This paper proposes FaithDaIL, a new Human-in-the-Loop (HIL) imitation learning framework that integrates dynamic regret minimization (via FTPL-D+) and an unbiased imitation objective inspired by DICE to faithfully learn from human interventions. The method is evaluated on MetaDrive and CARLA, and empirically outperforms existing HIL baselines in safety and task success.", "questions": "[Q1] What is the fundamental novelty in FaithDaIL beyond combining DICE and FTPL-D+?\n\n[Q2] What is the underlying rationale for FTPL-D+ being capable of solving the online IL problem well?\n\n[Q3] Can the method generalize to non-driving domains?\n\n[Q4] How sensitive is the method to the choice of time windows in FTPL-D+?\n\nMinor comment: It is better to formally define the meaning of \"faithful imitation\" in the paper.", "limitations": "The paper lacks a thorough discussion of potential limitations and corresponding solutions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n[S1] The work addresses a meaningful problem: improving data efficiency and safety in HIL imitation learning.\n\n[S2] The paper contributes some new insights to the community.\n\n[S3] The empirical results on driving benchmarks are competitive.\n\n**Weaknesses:**\n\n[W1] The key introduced technique - DICE - is well-established in the literature, like [1]. Also, the combination of DICE and FTPL-D+ does not introduce a fundamentally new learning principle; the methodological contribution is more of a composition of known tools than an innovation.\n\n[W2] There is a missing explanation for why FTPL-D+ can solve the online learning problem well. The paper does not provide new theoretical results, convergence analysis, or regret bounds tailored to the proposed problem. The use of dynamic regret is borrowed directly from prior work, and its connection to IL is not formalized beyond applying existing techniques.\n\n[W3] The evaluation is mainly limited to driving simulators. There is no discussion or demonstration of applicability to other HIL settings (e.g., robotics, dialogue systems), making it unclear whether the method generalizes.\n\n**Reference:**\n\n[1] How to Leverage Diverse Demonstrations in Offline Imitation Learning. ICML. 2024.", "was_revised": true, "final_justification": "After rebuttal, the authors have addressed my concerns, and I increase my score to 4.", "rebuttal": "### W1 & Q1: Novelty Clarification\n We clarify our key contribution to the community of human-in-the-loop imitation learning (HIL-IL) are:\n\nThis is the *first work to formulate HIL-IL as an online non-convex learning problem with dynamic regret as the optimization objective*.  Existing HIL-IL methods either use static regret or treat all imitation data  indiscriminately. We are *the first to identify the problem of shifting data distributions from evolving novice policy*, which has been totally overlooked in HIL-IL.\n\n We are also *the first to introduce the faithful imitation objective for leveraging hybrid data from both experts and the learning policy*.  Existing methods (e.g., HACO, IWR, PVP) optimize on mixed novice-expert data without distinction, hindering performance by causing the agent to imitate suboptimal novice actions.\n\nWe solve the dynamic regret problem using FTPL-D+ to adapt the dynamic non-convex loss and encode the faithful imitation objective into an unbiased proxy reward using DICE. These two techniques can be replaced with other methods (if any) as long as  they can slove dynamic regret problem with non-convex loss  and can optimize the unbiased imitation objective. \n\nIn addition, it is true that using a discriminator to estimate a density ratio and using weighted behavior cloning to extract policy is a common technique in IL. However, the key difference is how the weights are computed. ILID [1] (mentioned by the reviewer) trains a discriminator to identify expert states and selects preceding actions based on trajectory backtracking. It requires access to contiguous trajectories, while in our human intervention data consists of fragmented trajectories from experts. \n\n\n\n\n\n\n\n\n\n  ### W2 & Q2: Rationale of using FTPL-D+ & Theoretical Analysis\n The FTPL-D+ algorithm is an extension of the FTPL method. While FTPL is effective for optimizing static regret, it is not suited for dynamic distributions where environmental variation is unknown, a typical scenario in online imitation learning problems. \n\n FTPL-D+ addresses this challenge by using an ensemble learning framework. It maintains a collection of learners, each associated with a different interval parameter, and partitions the time horizon to restart FTPL periodically. A meta-algorithm, Hedge, is then used to adaptively select among these learners. This enables the method to adapt to environmental changes and achieve a strong regret bound $\\mathcal{O}\\left(T^{2/3}(V_T + 1)^{1/3}\\right)$, thereby being able to solve the online IL problems. \n\n \n We will provide the following theoretical analysis on regret guarantees. Our theoretical analysis shows that our algorithm is no-regret and can eventually converge to the optimal policy：\n\n(Rendering issues with long equations led us to split them and add new notations (e.g., $k = i-1$), slightly reducing readability.)\n\nWe aim to prove that our FaithDaIL is no-regret, i.e., \n\n$\n\\frac{1}{M} \\mathbb{E}[\\mathcal{R}_D] \\to 0\n$\n\n\nWe first give some assumption:\n\n+ **Bounded Parameter Space**: The policy parameter space is effectively bounded due to regularization and optimizer constraints in practice.\n+ **Lipschitz Loss**: The imitation loss $\\ell_i(\\pi)$ is Lipschitz in $\\theta_\\pi$, assuming (1) $\\pi(a|s)$ is Lipschitz, (2) importance weights $\\omega_i^\\star(s,a)$ are bounded, and (3) $\\pi(a|s)$ is bounded away from zero.\n\nFollowing the analysis from FTPL-D+, the **expected dynamic regret** of FaithDaIL satisfies:\n\n$\n\\mathbb{E}[\\mathcal{R}_D] \\le \\mathcal{O}\\left( M^{\\frac{2}{3}} (V_M + 1)^{\\frac{1}{3}} \\right),\n$\n\nwhere:\n\n- $M$ is the number of rounds,\n- $V_M =  \\sum_{i=2}^{M} \\max_{\\pi \\in \\mathcal{K}} |l(\\pi, \\mathcal{D}_i^B, \\mathcal{D}_i^H) - l(\\pi,\\mathcal{D}_k^B, \\mathcal{D}_k^H)|$, $k=i-1$ is the total variation of the loss sequence.\n\nTo establish that the dynamic regret is sublinear, **it suffices to show that $V_M$ grows sublinearly with $M$**. We formalize the implicit intervention rate $\\beta_i$ and state the core assumption underlying its use:\n\n+ **Definition (Implicit Intervention Rate $\\beta_i$)** :\n\n$\n\\beta_i = \\mathbb{E}_{\\pi^N_i } \\left[ \\frac{|\\mathcal{D}_i^{H,\\text{new}}|}{|\\mathcal{D}_i^{H,\\text{new}}| + |\\mathcal{D}_i^N|} \\right]\n$\n\n+ **Assumption (Intervention Rate Decay)** : This reflects the intuition that as the agent improves, fewer human interventions are needed over time.\n\n$\n\\lim_{i \\to \\infty} \\beta_i = 0\n$\n\nTo relate $\\beta_i$ to the trajectory distribution shift, we invoke the following result (adapted from DAgger, Lemma 4.1):\n\n+ **Lemma (Distribution Divergence Bound)**: With $T$ as the MDP horizon,\n\n$\n\\|d_i^B - d_i^N\\|_1 \\le 2T\\beta_i\n$\n\nWe now bound the single-step loss variation $\\Delta_i = \\max_{\\pi} | \\ell_i(\\pi) - \\ell_{i-1}(\\pi) |$.\n\n$\n|\\ell_i(\\pi) - \\ell_{i-1}(\\pi)|\n\\leq |A - B| + |B - C|\n$\n\nwhere $k=i-1$ :\n\n$\nA = \\mathbb{E}_{(s,a) \\sim d_i^B}[w_i^\\star(s,a) \\log \\pi(a|s)]\n$\n\n$\nB = \\mathbb{E}_{(s,a) \\sim d_k^B}[w_i^\\star(s,a) \\log \\pi(a|s)]\n$\n\n$\nC = \\mathbb{E}_{(s,a) \\sim d_k^B}[w_k^\\star(s,a) \\log \\pi(a|s)]\n$\n\n\n\n1. **Term 1 (due to distribution shift):**：\n   \n$\n   \\left| A-B \\right| = \\left| \\int_{(s,a)\\in\\mathcal{S}\\times\\mathcal{A}}[d_{i}^B(s,a) - d_{k}^B(s,a)] w_i^\\star(s,a) \\log \\pi(a|s) ds  da  \\right|\n$\n\n$\n\\leq N_1 \\cdot N_2 \\cdot \\|d_i^B - d_k^B\\|_1 \n$\n\nwhere: \n\n$\nN1 = \\|w_i^\\star \\|_{\\infty}\n$\n\n$\nN2 = \\|\\log \\pi\\|_{\\infty}\n$\n\n2. **Term 2 (due to weight shift):**：\n \n$\n   \\left| B-C \\right| = \\left| \\int_{(s,a)\\in\\mathcal{S}\\times\\mathcal{A}}[w_{i}^\\star(s,a) - w_{k}^\\star(s,a)] d_k^B(s,a) \\log \\pi(a|s) ds  da  \\right|\n$\n\n$\n\\leq N_2 \\cdot N_3 \\cdot \\|w_{i}^\\star - w_{k}^\\star\\|_1 \n$\n\nwhere: \n\n$\nN2 = \\|\\log \\pi\\|_{\\infty}\n$\n\n$\nN3 = \\|d_k^B \\|_{\\infty}\n$\n\n**Step 1: Bounding distribution shift:** Using triangle inequality and Lemma (Distribution Divergence Bound):\n\n$\n\\|d_i^B - d_k^B\\|_1 \\le 2T(\\beta_i + \\beta_k) + \\|d_i^N - d_k^N\\|_1\n$\n\nAssuming the novice policy changes smoothly: \n\n$\\sum_{i=2}^M \\|d_i^N - d_{i-1}^N\\|_1 = o(M)$\n\n and $\\beta_i \\to 0$, we have:\n\n$\n\\sum_{i=2}^M \\|d_i^B - d_{i-1}^B\\|_1 = o(M)\n$\n\n**Step 2: Bounding weight shift:** Assuming that the variation in importance weights is primarily induced by the shift in behavior distributions, and that $V_i^\\star$ evolves smoothly under $d_i^B$, we have:\n\n$\n\\|w_i^\\star - w_{i-1}^\\star\\|_1\n$\n\n$\n\\le L_w \\cdot \\|d_i^B - d_{i-1}^B\\|_1 + o(1)\n$\n\nThus:\n\n$\n\\sum_{i=2}^M \\|w_i^\\star - w_{i-1}^\\star\\|_1 = o(M)\n$\n\nThen under the earlier Lipschitz assumptions, we assume relevant terms of N1, N2, N3 are uniformly bounded.\n\n**Conclusion: Sublinear functional variation.**\n\n$\nV_M = \\sum_{i=2}^M | \\ell_i(\\pi) - \\ell_{i-1}(\\pi) | = o(M)\n$\n\n**Final Result: No-regret Guarantee**\n\nFrom FTPL-D+ analysis:\n\n$\n\\mathbb{E}[\\mathcal{R}_D] = \\mathcal{O}(M^{2/3}(V_M + 1)^{1/3}) = o(M), \\quad\n\\frac{1}{M} \\mathbb{E}[\\mathcal{R}_D] \\to 0\n$\n\nHence, **FaithDaIL is a no-regret algorithm** in non-convex dynamic environments, with the learned policy converging to the round-wise optimal.\n\n\n\n\n\n  ### W3 & Q3: Applicability to Other HIL Settings\nWe will add a section of discussion about applicability to other HIL settings as follows.\n\nThe core challenges FaithDaIL addresses—data efficiency and the impact of suboptimal novice data—are prevalent in many safety-critical domains beyond autonomous driving.\n\nRobotic Manipulation: In tasks like robotic arm manipulation, a human teleoperator may intervene to prevent errors. These interventions, coupled with the robot's evolving policy and potential reaction delays, create the exact type of dynamic, mixed-quality dataset that FaithDaIL is designed to handle. We plan to extend our work to this area using simulators to tackle tasks such as pick-and-place and object manipulation.\n\nDialogue Systems: High-stakes dialogue systems for customer service or medical consultation also rely on human oversight. An operator may correct or override the system's responses, leading to a mix of optimal and suboptimal conversational paths. As the dialogue agent learns and its response distribution shifts, FaithDaIL's dynamic regret framework can be applied to improve its performance and reduce the need for intervention over time.\n\n\n\n\n\n\n  ### Q4: Time Window Sensitivity\n\n FTPL-D+ employs an ensemble of base learners with different time windows ($\\tau_k = 2^{k-1}$ where $k$ denotes the $k$-th learner) and leverages a Hedge-style meta-algorithm to adaptively adjust their weights based on performance. This design inherently mitigates the sensitivity problem to different window settings. In practice, we observe that during training, the meta-algorithm tends to assign more weight to policies with shorter time windows in later rounds, suggesting that the method self-adjusts to favor recent, high-quality behavior distributions.\n\n\n\n### Q5:  Faithful Imitation Definition\n\n \"faithful imitation\" refers to explicitly imitating the human expert's policy, even when using a mixed dataset of expert and novice trajectories for efficiency. This is accomplished through our imitation objective (Equations 2-3), which optimizes the KL divergence towards the human policy distribution. We will formally define this concept in the next version of the paper.\n\n\n\n\n\n\n### Q6: Discussion on Limitation\n  We will discuss limitations as follows:\n  - Evaluation Scope: Validate the framework on other safety-critical platforms, such as robotic arms, by collecting and evaluating data from human teleoperator interventions.\n  - Expert Variability: Improve the model's robustness to diverse and potentially suboptimal interventions from different human experts. We can learn weight for expert data based on individual performance.\n  - Intervention Modality: Generalize the framework to support multi-modal feedback (e.g., verbal commands, gestures) instead of only direct control. This would involve encoding varied inputs into a consistent format for the learning loop."}]

Author rebuttal:

Meta-review: This paper studies imitation learning with online expert annotation, and formulates it as a dynamic regret minimization problem.  Experimental results under driving benchmarks were convincing to the reviewers. At the rebuttal stage the reviewers were convinced about the theoretical analysis, which strengthened the paper -- please include them in the final version.
Decision: Accept (poster)