Title: Faithful Dynamic Imitation Learning from Human Intervention with Dynamic Regret Minimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Human-in-the-loop Imitation Learning

Dynamic Regret

Online Learning

Off-policy Imitation Learning
Abstract: Human-in-the-loop (HIL) imitation learning enables agents to learn complex behaviors safely through real-time human intervention. However, existing methods struggle to efficiently leverage agent-generated data due to dynamically evolving trajectory distributions and imperfections caused by human intervention delays, often failing to faithfully imitate the human expert policy. In this work, we propose Faithful Dynamic Imitation Learning (FaithDaIL) to address these challenges. We formulate HIL imitation learning as an online non-convex problem and employ dynamic regret minimization to adapt to the shifting data distribution and track high-quality policy trajectories. 
To ensure faithful imitation of the human expert despite training on mixed agent and human data, we introduce an unbiased imitation objective and achieve it by weighting the behavior distribution relative to the human expert's as a proxy reward.
Extensive experiments on MetaDrive and CARLA driving benchmarks demonstrate that FaithDaIL achieves state-of-the-art performance in safety and task success with significantly reduced human intervention data compared to prior HIL baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FaithDaIL, a new Human-in-the-Loop (HIL) imitation learning framework that integrates dynamic regret minimization (via FTPL-D+) and an unbiased imitation objective inspired by DICE to faithfully learn from human interventions. The method is evaluated on MetaDrive and CARLA, and empirically outperforms existing HIL baselines in safety and task success.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n[S1] The work addresses a meaningful problem: improving data efficiency and safety in HIL imitation learning.\n\n[S2] The paper contributes some new insights to the community.\n\n[S3] The empirical results on driving benchmarks are competitive.\n\n**Weaknesses:**\n\n[W1] The key introduced technique - DICE - is well-established in the literature, like [1]. Also, the combination of DICE and FTPL-D+ does not introduce a fundamentally new learning principle; the methodological contribution is more of a composition of known tools than an innovation.\n\n[W2] There is a missing explanation for why FTPL-D+ can solve the online learning problem well. The paper does not provide new theoretical results, convergence analysis, or regret bounds tailored to the proposed problem. The use of dynamic regret is borrowed directly from prior work, and its connection to IL is not formalized beyond applying existing techniques.\n\n[W3] The evaluation is mainly limited to driving simulators. There is no discussion or demonstration of applicability to other HIL settings (e.g., robotics, dialogue systems), making it unclear whether the method generalizes.\n\n**Reference:**\n\n[1] How to Leverage Diverse Demonstrations in Offline Imitation Learning. ICML. 2024.\n\nQUESTIONS:\n[Q1] What is the fundamental novelty in FaithDaIL beyond combining DICE and FTPL-D+?\n\n[Q2] What is the underlying rationale for FTPL-D+ being capable of solving the online IL problem well?\n\n[Q3] Can the method generalize to non-driving domains?\n\n[Q4] How sensitive is the method to the choice of time windows in FTPL-D+?\n\nMinor comment: It is better to formally define the meaning of \"faithful imitation\" in the paper.\n\nLIMITATIONS:\nThe paper lacks a thorough discussion of potential limitations and corresponding solutions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces faithful dynamic imitation learning for human in the loop imitation learning. It propose a unbiased imitation learning objectives that  faithfully imitates human behavior and  contains a novel problem formulation by framing HIL as an online learning problem to tackle the non-stationary of the data distribution as agent’s policy improves.  Through experiments on the MetaDrive and CARLA simulators, the paper demonstrates that FaithDaIL achieves state-of-the-art performance in terms of safety and task success, while requiring significantly less human intervention data compared to existing HIL baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. As not an expert in HIL, I think the faithful imitation objective is very interesting and the way of converting it to the tracble form using odice through convex conjugate is clever.  For me the basic idea is similar to GAIL and reminds me of IQ-learn. \n2. By observing this instability, the paper employs FTPL-D+ to frame the task as an online learning problem with a dynamic regret objective. This is a much more principled approach than simply aggregating all historical data, as it allows the model not be held back by suboptimal data from early in training.\n3. the claims are supported by comprehensive empirical validation. The method outperforms a suite of baselines on two challenging driving simulators. The ablation studies clearly demonstrate the importance of both core contributions, and the qualitative case studies provide convincing visual evidence of the agent's superior safety and stability.\n\nWeakness:\n\n1.  The FTPL-D+ framework requires maintaining an ensemble of K policies, value functions, and discriminators. This approach seems computationally expensive and may pose scalability challenges, especially as the model complexity or the length of the training horizon (M) increases.\n2. The faithful imitation objective relies on a discriminator to estimate the density ratio, which serves as a proxy reward. I am not sure if this would introduce an adversarial training dynamic similar to that in GANs. The stability of such training could be a practical issue.\n3. The paper would benefit from a more detailed and intuitive explanation of the FTPL-D+ algorithm. While the reference is provided, a brief, self-contained overview of why this ensemble-based, multi-interval approach is well-suited to minimizing dynamic regret would make the paper more accessible to a broader audience.\n\nQUESTIONS:\n1. The non-stationarity you address is a common issue that also persists in general reinforcement learning (e.g., when the policy updates change the state visitation distribution). Do you believe the core ideas of your method, particularly the dynamic regret framework, could be generalized to improve sample efficiency or stability in standard RL settings beyond HIL?\n2. Can you explain the rationale behind selecting the FTPL-D+ algorithm for minimizing dynamic regret? Were alternative online learning methods evaluated during this decision process? Additionally, what practical trade-offs—such as the computational overhead introduced by the ensemble—does this choice entail compared to more straightforward approaches?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This work proposes Faithful Dynamic Imitation Learning (FaithDaIL), a framework for Human-in-the-loop imitation learning that enhances data efficiency by integrating evolving novice trajectories through dynamic regret minimization. The authors start their paper by motivating the need for their approach and discussing the drawbacks and limitations of prior work. The authors then formulate learning from active human intervention with evolving policy trajectories as a dynamic regret minimization problem. Afterward, the authors propose FaithDaIL and describe several components, including the loss function and algorithm. Finally, FaithDaIL is evaluated in two challenging driving simulators. The authors find that FaithDaIL achieves SOTA safety and task success with much-reduced human intervention.

STRENGTHS AND WEAKNESSES:
Strengths:
+ This paper deploys their algorithm in very challenging domains. As the algorithm is able to perform well in these domains, as well as require less human intervention, this lends support to the significance of FaithDaIL.
+ The contributions in this paper are significant and theoretically grounded.
+ The experiments in this paper utilize real human data collected from participants.

Weaknesses:
- Figure 2 is difficult to interpret and could use some enhancing and further captioning. The support text is helpful.
- Could you provide further information regarding the experiment procedure and HIL experiments? Was data for HG-DAgger collected in the same manner?

QUESTIONS:
+ Please respond to the weaknesses noted above.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of imitation learning with online expert annotation, in which the learner can access the environment and generate states, meanwhile the expert can provide action for each state. The paper formulates it as a dynamic regret minimization problem, and proposes a learning approach based on the follow-the-perturbed-leader (FTPL) paradigm. Experimental results under autonomous driving benchmarks are provided to verify the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths:\n\n1. The paper suggests studying dynamical regret minimization in imitation learning, which is less-studied to my knowledge. This may inspire online learning theory researchers to consider imitation learning as a practical applicational area for dynamic regret studies. \n\n- Weaknesses:\n\n1. The learning problem may not be properly formulated. The dynamic regret actually assumes that the optimal policy changes over time, while in the paper, the optimal policy remains the same as the oracle policy. Even though in Eq. 1, $\\pi^*$ seems to be related to time index $i$, while the paper does not assume that the oracle's policy can change. Instead, the paper claims that the reason to consider dynamical regret is that the learner's policy can change. While this situation is common for all online learning problems, even when the static regret is considered. Actually, the motivation for the classical DAGGER algorithm is to consider the effect of changing learner policy. \n\n2. In the abstract and introduction part, it is said the expert reaction delay can be an issue. While the paper seems to lack further discussions on the problem.\n\n3. The notion of faithfulness needs further clarification. It seems that the paper utilizes this as a measurement on how good the learner can imitation the expert. But the relationship of faithfulness to the dynamic regret problem still lacks clear explanations. \n\n4. The paper lacks theoretical analysis on regret guarantees, which is important for online learning researches.\n\nQUESTIONS:\nPlease feel free to respond to the points in the weaknesses part.\n\nLIMITATIONS:\nThe discussions of technical limitations seem to be lacking.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}