Title: Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning
Venue: NeurIPS 2025
Accepted: yes
Keywords: imitation learning

online learning

reinforcement learning
Abstract: Imitation learning (IL) is a paradigm for learning sequential decision-making policies from experts, leveraging offline demonstrations, interactive annotations, or both. Recent advances show that when annotation cost is tallied per trajectory, Behavior Cloning (BC)—which relies solely on offline demonstrations—cannot be improved in general, leaving limited conditions for interactive methods such as DAgger to help. We revisit this conclusion and prove that when the annotation cost is measured per state, algorithms using interactive annotations can provably outperform BC. Specifically: (1) we show that Stagger, a one‑sample‑per‑round variant of DAgger, provably beats BC under low-recovery-cost settings; (2) we initiate the study of hybrid IL where the agent learns from offline demonstrations and interactive annotations. We propose Warm-Stagger whose learning guarantee is not much worse than using either data source alone. Furthermore, motivated by compounding error and cold‑start problem in imitation learning practice, we give an MDP example in which Warm-Stagger has significant better annotation cost; (3) experiments on MuJoCo continuous‑control tasks confirm that, with modest cost ratio between interactive and offline annotations, interactive and hybrid approaches consistently outperform BC. To the best of our knowledge, our work is the first to highlight the benefit of state‑wise interactive annotation and hybrid feedback in imitation learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes a DAGGER-style algorithm which has access to both expert data, as well as interaction with the environment with access to a state-labeling oracle. The paper demonstrates theoretically how having access to both sources of data can lead to algorithms with better sample complexity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n - The paper furthers our understanding of DAgger-style algorithms which are fundamental to modern data pipelines for robotics. \n - The presentation is excellent, with a clear and intuitive proof sketch for the core result. The main Theorem 6 and proof sketch provide a clear framework for how to think about the tradeoffs in interaction algorithms.\n\nWeakness:\n - The paper makes use of Hellinger/TV-style distances throughout and uses a very strong indicator-based loss. It is unclear if these results transfer to a continuous-control setting, which is the principal setting of the experiments and presumably downstream applications. In particular bounds for continuous settings by necessity require much stronger assumptions on the expert (c.f. \"The Pitfalls of Imitation Learning when the Action Space is Continous\", Simchowitz et. al.). I believe there is considerable value in studying the tabular MDP case, but that the general claims need to be narrowed to better reflect that this only applies to discrete MDPs.\n\n - The original DAgger paper considers probabilistic switching between the expert and the learned policy on data collection rollouts, whereas WARM-STAGGER/STAGGER appear to roll out the learned policy directly after collecting the initial data. Properly tuned probabilistic switching between $\\pi^E$ and $\\pi^n$ should prevent the \"warm start issues\" mentioned in Proof Sketch of Theorem 6 (see the \"beta\" parameter in the original DAgger Ross et. al. paper). Slow mixing DAgger should explore E with higher probability, and therefore I believe would have similar properties to the WARM-STAGGER algorithm presented in this work.\n\nQUESTIONS:\n- The experiments run are all in continuous state and time, while the loss and theory uses Hellinger distances and indicator functions. Considering that continuous imitation learning can be exponentially harder than its tabular MDP equivalent (\"The Pitfalls of Imitation Learning when Actions are Continuous,\" Simchowitz et al), would it make more sense to consider discrete/language style tasks instead of continuous control tasks?\n\n- The paper uses \"hybrid imitation learning\" to describe the problem setting, yet similar algorithms like \"vanilla\" DAGGER and DART are usually described as \"online\" or \"interactive\" imitation learning algorithms and have the same premises as the hybrid setting described in the paper--access to an expert oracle and further rollouts. It would be helpful for the authors to clarify what they view as \"offline\" vs \"hybrid\" vs \"online\" imitation learning. Perhaps \"interactive\" and \"non-interactive\" would be better terminology to avoid overloading with terminology from Reinforcement Learning?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes new interactive algorithms for imitation learning. Motivated by several recent theoretical findings surrounding behavior cloning, the authors look into whether low-cost interaction can benefit behavior cloning, including in the case of function approximation. Specifically, the authors focus on how to minimize the number of expert queries, such that expert performance is achieved or some theoretical gap can be proven to occur with high probability.\n\nSpecifically, the proposed algorithm, STAgger, executes a mixture policy at each step (effectively can query a different policy at each state it seems along the trajectory), obtaining a single sample and using it to update the set of likelihoods over a finite policy class, of which the expert is assumed to be in.\n\nExperiments across a few MuJoCo tasks show that STAgger outperforms behavior cloning in terms of sample efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is pretty theoretically sound -- the underlying theory is classical no-regret theory from the original DAgger paper for the most part, which is well-known. The experiments also show good results of the proposed method, which is pretty neat for imitation learning. As for the method, it is a pretty straightforward extension to DAgger in the case of having a finite set of policies -- you effectively want to reduce your space down through queries on the mixture distribution, and update the mixture to bias towards the (expert) labels given at each query.\n\nThere are no theoretical weaknesses in the paper (e.g. nothing is really wrong from the theory side), but from the perspective of significance, I'm not sure how much this has from a practical standpoint. There weren't many details given about the experimental setup (my big concern are posed in the Questions section), and it seems that with something like this (if the set of policies given to you is reasonably small), a simple baseline could just be to find the policy in the (known) class that does the best (e.g. some type of test-set logic here) on the expert dataset (although I may be misunderstanding this), in which no interactions are required. It seems that interactive IL is great for theory, but not great from a practical perspective, and I'm not sure how much theoretical contribution this brings for now.\n\nQUESTIONS:\nThere are a few things I would like to clear up personally -- how many members of the policy class do you experiment with when you run these MuJoCo tasks? The task seems like you have a set of deterministic or Markovian policies, and you're effectively learning some sort of router/mixture of them that should in principle converge to a Dirac over the true expert. The appendix didn't give much in terms of the experimental paradigm, so I was wondering how many policies this allows for.\n\nTo scale this to resemble something close to what we may have in practice, how does this scale to a (finite) set, but which has a large number of policies? This is in effect something that is more realistic, where we would want to choose the expert from this set as quickly and with as little interaction with the oracle as possible. Maybe an ablation study can be done for that.\n\nLIMITATIONS:\nThe authors do mention some limitations of their work, including expert realizability (strong assumption) and lack of modeling of the environment query cost. It is clear (from an IL practitioner's perspective) that one can only be as good as an expert on any IL task at optimum, so design choices around the expert and the data collection process before IL takes place should be carefully considered from a societal standpoint. Maybe these can be put together as an addition to the last section of the paper (e.g. a \"Limitations\" section).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores theoretical underpinnings of when interactive imitation learning (i.e. variants of DAgger) can provably outperform vanilla behavior cloning (BC), and when a hybrid approach can provably beat both pure DAgger and BC, given the same budget of \"expert annotations\". To this end, the paper considers an amortized version of DAgger denoted STAgger, which instead of querying expert labels for each state along an on-policy rollout, simply draws a single sample from the on-policy averaged state visitation distribution. A guarantee for STAgger is established, showing that the performance upper bound can be smaller than that of BC as long as the expert is sufficiently recoverable and the relative cost of state-wise oracle queries is not too large. Given this result, a natural extension denoted Warm-STAgger is considered for hybrid settings, where an offline dataset is available in addition to online expert queries. Warm-STAgger sets the initial rollout policy is to an empirical risk minimizer over the offline dataset and performs STAgger on top. This algorithm is demonstrated to satisfy a bound no worse than either pure offline BC or pure online STAgger. Some lower bound constructions and numerical experiments are provided to demonstrate the benefits of online interaction.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, this paper introduces an interesting dimension to imitation learning. The paper is generally well-written. Thus, I am leaning toward acceptance. My main concerns with the paper are regarding the actual efficacy of the proposed algorithm, and the degree to which the theorems convincingly establish the uniform benefit of hybrid imitation learning.\n\n**Strengths**\n\nThe focus on reducing the number of online expert oracle queries is a perennially relevant one, and this paper achieves an interesting result that a reduced form of DAgger may actually perform on par or better to offline BC and DAgger, as indexed by the budget of expert queries. This has practical relevance when on-policy expert queries may be expensive compared to collecting expert policy rollouts.\n\n\n**Weaknesses**\n\nIn terms of theoretical soundness, Theorems 2 (BC), 3 (STAgger), 4 (Warm-STAgger) are technically comparing worst-case performance upper bounds. To be clear, the authors do not claim otherwise, but this technically renders claims of one method being generically better than another difficult to conclude. Theorem 6 partially closes the gap with instance constructions. However, it remains unclear when Warm-STAgger generically instance-wise dominates the performance of (ST,D)Agger or BC.\n\nIt is somewhat precarious to attribute the experimental results in continuous control as support for the theoretical bounds in this paper. In particular, it is known that imitation learning in control systems suffers unique problems that otherwise do not manifest in discrete-action settings [1, 2]. Notably, imitation learning in control systems can suffer from exponential-in-horizon compounding errors. This does not conflict with, e.g. the loglossBC bound, because such bounds assume policy estimation in losses such as indicator or log-loss, which are not feasible in continuous action spaces. I am not aware of analysis of DAgger in these settings, though I imagine for expert annotation along a exponentially unstable policy rollout likely yields exponentially low \"coverage\" around on-expert trajectories.\n\nMinor comments\n\n- Some inconsistent use of $\\ln$ versus $\\log$, e.g. line 108\n- Line 210: the equality should be an inequality\n- Should the loss used for fitting the BC policy $\\hat\\pi$ technically be the log-loss to instantiate Theorem 2? The set of empirical risk minimizers is the same for deterministic realizable experts, but it does matter with respect to which loss generalization is proven in order to yield the horizon-independent bound in Theorem 2.\n\n[1] Tu et al. \"On the Sample Complexity of Stability Constrained Imitation Learning\"\n\n[2] Simchowitz et al. \"The Pitfalls of Imitation Learning when Actions are Continuous\"\n\nQUESTIONS:\nPlease refer to Weaknesses above. My main questions can be summarized as:\n\nWhen can sparse corrections like in Warm-STAgger typically outperform both BC and a BC warm-started DAgger (as is typically done in practice)? In other words, in what regimes is Warm-STAgger actually expected to outperform both BC and (Warm-)DAgger, rather than \"falling back\" to the behavior of one or the other? When DAgger also gets a warm-start, when is expert annotation along the full-trajectory provably \"wasteful\" compared to state-wise annotation?\n\nClarifying these points will alleviate some of my hesitation about interpreting the results.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper investigates the benefits of state-based annotations for imitation learning. A new imitation algorithm with state-based annotation, STAGGER, is developed and settings with better cost-sample efficiency tradeoff than offline imitation (behavioral cloning) are analyzed. A hybrid method that initializes from offline imitation, called WARM-STAGGER, is also developed to outperform both purely offline and the base STAGGER algorithms. An MDP is constructed to demonstrate this. Experiments in MuJoCo also demonstrate the benefits of the approach.

STRENGTHS AND WEAKNESSES:
1. This paper makes an important contribution to imitation learning by establishing the theoretical benefits of interactive imitation learning.
2. The paper is nicely organized and well written in motivating, defining, and supporting its main claims.  
3. The description of related work makes the originality and significance of the paper's contributions clear.  
4. Despite being heavily based in sample complexity theory, the developed algorithms are practical.
5. The experiments effectively demonstrate the benefits of the developed approach.

Overall, this is a solid paper with both theoretical and experimental contributions.

QUESTIONS:
Q1. Are the theoretical bounds meaningful for the MuJoCo environment? In other words, can R and \mu be provided/estimated for each environment (and the resulting ratios discussed)?

Q2. How reliant do you anticipate the performance of your algorithms to be on the assumptions of the expert policy being realizable and deterministic?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-11