Efficient Active Imitation Learning with Random Network Distillation
Authors: Emilien Biré
Anthony Kobanda
Ludovic Denoyer
Rémy Portelas
Venue: ICLR 2025 Poster (2025)
Keywords: Active Imitation Learning
Imitation Learning
Interactive Learning
Navigation
Primary area: reinforcement learning
TL;DR: Human in the loop interactive training during which the agent is deciding when expert demonstration is needed

Abstract: Developing agents for complex and underspecified tasks, where no clear objective exists, remains challenging but offers many opportunities. This is especially true in video games, where simulated players (bots) need to play realistically, and there is no clear reward to evaluate them. While imitation learning has shown promise in such domains, these methods often fail when agents encounter out-of-distribution scenarios during deployment. Expanding the training dataset is a common solution, but it becomes impractical or costly when relying on human demonstrations. This article addresses active imitation learning, aiming to trigger expert intervention only when necessary, reducing the need for constant expert input along training. We introduce Random Network Distillation DAgger (RND-DAgger), a new active imitation learning method that limits expert querying by using a learned state-based out-of-distribution measure to trigger interventions. This approach avoids frequent expert-agent action comparisons, thus making the expert intervene only when it is useful. We evaluate RND-DAgger against traditional imitation learning and other active approaches in 3D video games (racing and third-person navigation) and in a robotic locomotion task and show that RND-DAgger surpasses previous methods by reducing expert queries.
https://sites.google.com/view/rnd-dagger

Published as a conference paper at ICLR 2025
EFFICIENT ACTIVE IMITATION LEARNING WITH RANDOM NETWORK DISTILLATION
Emilien Biré'; Anthony Kobanda², Ludovic Denoyer', Rémy Portelas² 'Centrale Supelec Ubisofft La Forge 'H Company remy Leny.pozee porteerteleee aaeeelassuue aawelbissssssseeeee com
ABSTRACT
Developing agents for complex and underspecified tasks, where no clear objective exists, remains challenging but offers many opportunities. This especially true in video games, where simulated players (bots) need play realistically. and there is no clear reward to evaluate them. While imitation learning has shown promise in such domains, these methods often fail when agents encounter out-of-distributon scenarios during deployment. Expanding the training dataset is a common solution, but it becomes impractical or costly when relying on human demonstrations. This article addresses active imitation learning, aiming to trigger expert intervention only when necessary, reducing the need for constant expert input along training. We introduce Random Network Distillation DAgger (RND-DAggen a new active imitation learning method that limits expert querying by using learned state-based -id-t--dditiittiinn measure to trigger interventions. This approach avoids frequent expert-agent action comparisons, thus making the expert intervene only when it is useful. We evaluate RND-DAge active approaches in 3D video games (racing and third-person navigation) and in a robotic locomotion task and show that RND--Agger surpasses previous methods by reducing expert queries. httpss: pettss//see tes. googl e. com/v hwwm/view/eiie wew/rndd wewpa/rnddaaeee dagger
against traditional imitation learning and other
INTRODUCTION
Imitation learning has increasingly become a favored approach for learning behaviors in complex environments, offering compelling alternative to classical scripted behaviors implemented by domain specialists (Schaal) 1999; Hussein et al., 2017). It is particularly well suited in problems where there is not clear performance measure reward). In video games, it is becoming more and more familiar to game developers that frequently address the problem of implementing bots in their games which must play in realistic ways (Harmer al. 2018 Yadgaroff al. 20244] Mao et 2024). Indeed, since the notion of realism is not well specified, it prevents the use of reinforcement learning-basse based approaches where reward signal is mandatory. By observing and replicating human players, these bots are trained to execute complex strategies and actions that are both efficient and human-like.
Imitation learning usually proceeds in two steps: first, dataset of behaviors is built by leveraging experts interacting with the dynamical system. Then, statistical model (e.g. a neural network) learned to imitate actions in that dataset, thus expecting this model to generalize to unseen state and to behave like the experts. Consider a use case where the goal is to train a driving policy in a video game context capable of controlling a is defined by sensor values at time f, which can include information such as the position of the car, its speed, and raycasts. In this context, Imitation Learning involves manually controlling the car for many laps to create a learning dataset, training a policy on this dataset using traditional algorithms like Behavioral Cloning (Bain & Sommut 1999). and expecting that the resulting bot will control the car correctly. However, this approach is known to be unreliable, particularly due to the risk of a shift in the state distribution between training and testing (Ross al., (01T), known as the problem of covariate shift (Nair et al., 2019). When covariate shift occurs, the agent struggles to determine the correct action, leading to compounding errors and ultimately undermining its performance.
track (see figure 1 In this scenario, the state
WWrk done during an intersship Ubisaft La Forge

---

Published as a conference paper at ICLR 2025
Badkonnaas
Autonomous diiving
Exprt demerstration
Autenommussdiiigg
The teris
The fesmnel
(a)
(b)
(c)
(d)
Racovery
Figure 1: RND-DAgger overview. (a) The learner's policy controls the agent until (b) our Random Network Distillation-ben based out-of-distribution (OOD) measure is triggered. Then (c) the expert takes control until the OOD measure gets lower than the threshold for at least W steps. (d) Finally, the current policy takes control of the agent to continue the episode, and can trigger the expert again later.
Facing this issue, the effectiveness of imitation learning hinges on the availability of extensive datasets comprised of player behaviors, often necessitating thousands of expert traces to achieve reasonable performance and exhibit credible behaviors (Vinyals et al. 2019 Mao et al.) 2024). This dependency on large volumes of data poses a challenge, particularly in scenarios where collecting such data is either impractical or costly.
In response to these limitations, some approaches involve numan-in-the-looe training framework. often referred to active imitation learning (Ross et al., 2011 Judah al. 2012: Menda al. 2019; Hoque et al. (021). Instead of gathering very large amounts of demonstrations, these methods focus the strategic incorporation of human feedback to selectively guide the learning process, thereby enhancing the efficiency of the training phase. By prioritizing the acquisition of relevant and impactful expert traces, these approaches seek expedite the agent's 's learning curve without relying on expansive datasets, and swiftly improve performance. In the aforementioned video game example, this corresponds to collecting a first set of player traces bootstrap the agent using classical imitation learning, and then to gather additional traces in particular race settings during training, i.e. requiring a player to control the car particular and relevant situations.
To tackle these challenges, several recent models have been proposed to identify when the expert's intervention is necessary to correct the agent's s suboptimal behavior (Zhang Cho 2017 Menda et al. 2019 Kelly et al., 2019 Hoque et al.] 2021), with DAgger (Ross et al.) (001]) being one of the most well wnoo- approaches. The primary objective of these methods is to minimize the time required for human feedback by concentrating interventions
the most critical situations where the agent is likely to make errors. This allows for a more efficient use of expert time, ensuring that corrective feedback is provided only when it is essential. But, explained in Section 2 many of these methods are not fully satisfying.
Despite these advancements, designing algorithms that effectively incorporate human feedback while reducing the overall expert effort remains an open research problem. Addressing this issue is the central focus of this article, as aim develop methods that optimize the interaction between the expert and the learning agent, thereby improving learning efficiency without overburdening the human supervisor.
Our main contributions are threefold: i) We propose a new method called RND-DAgger, a novel in- teractive imitation learning approach leveraging state- random network distillation. ii) We perform a comparative analysis of RND-DAgger and existing methods on 3 tasks: roboties scenario and two video-game environments. iii) Throughout these experiments, we demonstrate that RND-DAgger either outperforms matches existing approaches terms of final performance while significantly reducing expert burden.
identification through
2

---

Published as a conference paper at ICLR 2025
Algorithm 1 DAgger
Algorithm 2 Lazy/Ensemble DAgger
Require: K, (B.)e T, exp
Require: K, CONDITION, T, Texp
1: Instantiate dataset of expert trajectories
Instantiate D dataset of expert trajectories 2: Instantiate To BC policy trained on T)
2: Instantiate To BC policy trained D
3: t-0
nswitch <-
4: for 0, . K DAgger iterations do
4: t 2-- 0
5: Di <
5: D; +
6:
for T sampling steps do
for O, ..... K iterations do
7:
9; Enw(as-1) (Get new state)
7:
while # D; < do
8:
ar - BeTexp(St) (1 (s)
8:
8t EEw((a-1) (Get state) C+ < CONDITION(TTEP:: 8t)
9:
Di < Di [[at, Fexp Tezp(St)) (8t))
9:
+-1+1
10:
if if then
11:
Push D < DUD;
11:
a: 5 (ezp(Se) D; + D ((ae,
12: Train 1 on D using BC
12:
13:
return TK
13:
++t+1
14:
if not C.-1 then L nswilch + nswilch +
15:
Figure Above, the DAgger algorithm where the expert action added the training at each
16:
else
17:
a: t (.(s))
timestep. the agent being alternatively controlled
18:
t ++1+1
by the current policy and the expert one. On
19:
D DUD:
the right, in Lazy/Ensemble DAgger, the agent controlled by the expert policy only if given measure is higher than a threshold A. Only actions generated when the expert controls the agent are added to the training set.
20:
Train M441 on D using BC
21:
return Т
PRELIMINARIES AND RELATED MODELS
Notations Let us consider a Markov Decision Process described by a state space S and an action space A. The dynamics of the process are defined by an unknown transition function P(s ll,, where a, E A and 8; E S. Note that we do not consider any reward in this setting since the task to solve is not explicit (e.g.. driving car); objective is instead imitate human behaviors that may be suboptimal and diverse (e.g., considering multiple players or experts). An episode consists of sequence of states and actions, 7 (81, (9,,a1..... 8T, ar), where T is the length of the episode. Given policy (aa ss),, it is possible to sample an episode by sequentially executing the policy until reaching a stop criterion.
Distributional shift: Given dataset of expert demonstrations D (Ti..... Tn) common approach to policy learning is Behavioral Cloning (BC) (Bain Sammut 1995; Ding et al. 2019a), straightforward imitation learning algorithm. BC enables to learn policies by maximizing the log-likelihood of the expert actions within the dataset. However, learning a policy from a fixed dataset can result the policy encountering out-of-distribution (OOD) states during inference, where has not been trained to make accurate decisions (Ross et al. (201)) I). This phenomenon is referred as distributional shift. This shift in the distribution can cause the policy perform poorly, may not know how to handle unseen situations. For example, consider a driving policy in racing game. If the dataset is composed only of demonstrations from expert drivers, the learned policy might struggle to recover from rare events such as spin-out, since such situations are absent from the expert demonstrations. Therefore, the policy may fail generalize effectively to these unseen states.
This effect can be mitigated by providing more diverse and comprehensive training sets, ensuring that the policy is exposed to a wider range scenarios, including edge cases. However, this raises the challenge of how to efficiently collect such datasets while minimizing the burden on experts. Gathering enough data to cover all possible situations can be time-consuming and costly, especially if it relies heavily on expert demonstrations. Thus, the key question becomes how strategically collect diverse and representative data in a way that maximizes coverage of the state space while minimizing the amount of effort required from experts.
3

---

Published as a conference paper at ICLR 2025
Algorithm 3 Enneebbl-DAgger's CONDITION
Algorithm 4
CONDITION
Require: Tnoy mixture of N policies,
Require: ftS 6H, 3R
Teep: St. X,7
1: my < f(a) which estimates
1: o nov doubt of the mixture Now
Meep (8+) (sx))1
2: my & [stp($t) nno (s))l
2: m > BH then
3: if o X my then
3:
return TRUE
4: else
4: return TRUE
5: if mh < Bn then
5: else
6: return FALSE
6: L return FALSE
7: else
8: return TRUE
Figure 3: In Ensemble-DAggee the policy is a mixture, and the measure is based on both a dis- agreement between the models in this mixture, and a disagreement with the expert action. The computations of these measure require the expert to provide action at each timestep of the process. In Lazy-DAgger, the measure based on the disagreement between the current policy and the expert one, but to avoid the query of the expert action at each timestep, classifier f is trained to predict if the discrepancy measure [x)p(s)) n. )I² is above given threshold 3μ or not.
DAgger and variants: To address this issue, one can utilize the DAgger algorithm from Ross et al. (2011) (Algorithm [].. which iteratively expands the training dataset through the intervention of an expert, represented by a reference policy denoted as Texp- The reference policy serves two key roles: first, it determines the appropriate action to execute at each timestep based on specified decision rule (line 7) mixture between the current policy and the expert one. Second, helps augment the training set by adding new samples of states and corresponding expert actions, which are then used to improve the learned policy (line 11). At regular intervals, the policy is updated via imitation learning using the expanded dataset, and this process is repeated over multiple iterations. This method helps to mitigate the issue of distributional shift by continually refining the learned policy with fresh data that captures broader range of states.
To better sample relevant states, few variants of the DAgger algorithm have been proposed. One of the most relevant is Ensemble-DAgee (Menda et al. 2019) which introduces decision rule to decide if the agent is controlled by the current policy, or by the expert one. Moreover, pairs of state-actions are added to the learning dataset only when the action comes from the expert policy (see Figure - right ). The decision rule is a discrepancy measure that computes the distance between the expert action and the policy action (combined with a disagreement measure between mixture of experts) shown in Algorithm 3 This approach is not realistic when considering human experts. Indeed, let us revisit our car driving example, where the goal is learn an effective driving policy for video game. Using the DAgger and Mnssembee-DAggee approaches, the expert (i.e. the player) must continuously play the game during the active imitation learning process, while the car is only partially controlled by the current learned policy. This setup has two significant drawbacks: i) First, the player is placed in an unnatural setting, where they are expected to control car that they do not fully manage. This can create a disorienting experience, as the player must constantly adapt to actions taken by the policy, disrupting the natural flow of gameplay. ii) Second, the player remains engaged even when the car exhibiting good behavior, spending time providing feedback that may not significantly contribute to discovering a better policy. This leads to inefficiencies, as the expert's time not always optimally utilized, especially in situations where the current policy already performs well.
3 RANDOM NETWORK DISTILLATION-BASER DAGGER
To address the aforementioned limitations of Menda et al. (2019), Lazy-DAgger (Hoque et al.; 2021) proposes to involve the expert only during critical moments rather than requiring continuous control. Instead of relying the expert partially guide the agent throughout the learning process, this approach alternates between periods where the agent is completely controlled by the current policy and periods where it is fully controlled by the expert. During the phases controlled by the current policy, the expert's 's intervention is not required, thereby reducing the time the expert spends supervising the agent and making their interventions more targeted and efficient.
4

---

Published as a conference paper at ICLR 2025
Algorithm RND-DAgest
Figure 4: The RND-DAgger algorithm proceeds in K iterations. First policy is trained on a small expert dataset (line 2). Then at each iteration (line 6), it constructs a new dataset D; with T state- action pairs from the expert (line 7). At each timestep. the agent is by default controlled by the current policy (line 23). The OOD measure is computed by us- ing the prediction network freed and the target network ftarg (line 9). If this mea- sure is greater than a threshold (line 10), the expert takes over the policy to control the agent (line 16) and to add samples to the dataset (line 17). The number of con- text switches the number of times the expert has taken control (line 19). At the end of the iteration, the built dataset is aggregated with the current learning set (line 27) and the policy is retrained (line 28). addition, the fpred network is also updated (line 29) to allow the future detection of OOD states. The w counter is used to ensure that the algorithm waits at least W timesteps below the threshold before taking control back of the agent.
Require: K, mezps ftargs fpred, A, W,
1: Instantiate D dataset of expert trajectories
2: Instantiate To BC policy trained on D
3: nswitch
4: 1 4-0
5: Do f-0
6: for - O..... K DAgger iterations do
7:
while # D; T do
8:
Sc En)(ar-) (Get a new state)
9:
m. + [1 - [s)ell
10:
if ma > À or < W then
11:
if me S X then (Minimal demo time)
12:
+1
13:
else
14:
w 0-
15:
<- (se)
16:
Dj Dj s1)
17:
if m S X then nswitch + nswitch +
18:
19:
else
20:
a. - (s:t)
F t +1
23:
Push D 4- DUD:
24:
Train T++1 on D) using BC
25:
Train J Sprod D to predict ftory
26:
return TTK
Lazy-DAgger replaces the traditional action-based discrepancy measure between expert and policy actions with classifier-based approximation see Figure This approach allows the algorithm to predict when expert intervention is necessary without requiring an expert action at every timestep, thereby significantly reducing the expert burden compared to methods like Masemble-DAgeer How- ever, this method has notable drawback: similar to Ensemble-DAgger it relies on comparing the actions of the expert and the current policy for a given state. While this approach works well when the expert is optimal and acts deterministically. it becomes problematic when dealing with humans or imperfect experts. Human experts often exhibit variability and may choose different actions for the same state depending on context personal preference. This results in a noisy and unreliable measure of discrepancy. To address this issue, one could rely on a state- discrepancy measure that would focus meaningful divergences compared to action- one.
We propose the RND-DAgger algorithm, which builds upon Random Network Distillation (RND) (Burda et al., 2019). The core assumption of RND-DAgger is that expert feedback is only necessary when the agent encounters out-of-distribution (OOD) staess that are not well represented in the training set and where the policy more likely to fail. This is crucial because when an agent operates in OOD states, it faces a higher risk of taking suboptimal actions that could hinder learning or lead to unsafe outcomes. Once the agent returns to familiar, in-distribution states, the expert's feedback is no longer required, and the intervention is ended. Our method differs fundamentally from existing approaches like masemble-DAgee and Lazy-DAgger, which rely on action-based based discrepancy measures to determine when to intervene. As such, our approach remains robust to variations in expert behavior, ensuring that interventions only occur when the agent genuinely faces unfamiliar and potentially risky states.
To further improve the stability of expert interventions, we introduce a mechanism called minimal demonstration time. This concept addresses the issue of overly frequent and brief expert interventions, which can interrupt learning and increase the cognitive load on the expert. Minimal demonstration time defines lower bound on the duration for which the expert must maintain control once they start an intervention. The key idea is that the expert should provide a sufficient number of consecutive corrective actions guide the agent back stable state, instead of immediately handing control
5

---

Published as a conference paper at ICLR 2025
back to the policy after single correction. We describe these two components below. The detailed algorithm is provided in Algorithm 4 with a complete explanation of the different steps.
Random Network Distillation To measure if a state is OOD, RND-DAgger relies the Random Network Distillation technique, which is a classical approach developed initially for the problem of exploration in Reinforcement I Learning to detect new states to explore (Burda et al.] 2019). The principle of RND is to use a randomly initialized neural network as a fixed target ftorg and train second neural network fpred (the predictor) to approximate the output of the target network. As the predictor network improves over time, the error between the predictor and the target network decreases for familiar states (in-distribution), but remains high for unseen or distriibbiittoo states. This prediction error thus serves as a measure of novelty, allowing the agent to recognize when it has encountered a new or unfamiliar state. In RND-DAgger, the measure is used to decide when to trigger the expert, and when to trigger back the current policy, alternating phases where the expert is controlling the agent and when the agent controlled by the current policy. Note that, contrary the described baselines, this measure does not involve the expert action, thus avoiding the need to have an expert acting at each timestep.
Minimal demonstration time Relying exclusively on the OOD measure can lead to a behavior where the expert is asked too often for short demonstrations (see appendix Figure (C.2). To account for this limitation, we introduce the notion of minimal expert time (MET). It is defined as W the number of consecutive "in-distribution" frames required to switch control back from the expert to the autonomous policy (i.e. the number of frames where m, < Il.ttar - fprea(s))ll (s)))l²) This mechanism ensures that the expert will provide a minimal amount of information at each intervention, resulting in fewer, but longer, demonstration sequences.
4 EXPERIMENTS
The primary objectives of our experimental evaluation are twofold: first, to assess the efficiency of our approach in discovering an effective policy, and second, to evaluate its ability to reduce the expert's 's burden.
Environments: Our first environment HalfCheetah which a classical reinforcement learning unvironmen[[whee where the objective is to learn running strategy for the agent. The goal of the agent is to locomote as fast as possible. The HalfCheetah is controlled by applying motor torques, and the agent manipulates a b-degree-of-freeedon (6-DOF) motor joint vector. The observation space consists of 18 values, which include the position and velocity of the agent's 's body, the angles and angular velocities of its six joints, as well as the angle of the center of mass. We also propose and open-source two new environments developed for video game research. RaceCar (see Figure (5) features physics-based car controller that must complete single lap on a given track. After each lap, the car is reset to a random position on the starting line. The track presents several challenges, including speed bumps, a ramp front of a pillar, and sharp 90-degree turns, making the optimal driving behavior noi-trivial. Additionally, crossing the ramp and red sloped walls provides speed boost, adding complexity to the strategy. The car is controlled using four discrete actions: forward, backward, left, and right. The observation space consists of 22 dimensions, including the car's 3D position, linear velocity (3D), angular velocity (3D), rotation (encoded as the cosine and sine of the angle), and data from seven raycasts to detect obstacles. Finally, the 3D Maze environment allows us to study our strategy in goal-conditioned navigation scenarios. A classical human-like character is spawned at random location within the maze and tasked with reaching randomly assigned goal. The agent is controlled using four movement actions: walk forward, walk backward, strafe left, and strafe right, as well as one-dimensional rotation action change its facing direction. The observation space consists of the agent's is absolute position (3D), the goal's position (3D), and 131 raycasts, which provide low-resolution depth map of the surroundings. Since the objective in this environment is to navigate to any goal location, we replace the traditional Behavioral Cloning (BC) approach with a Goal-Conditiones Behavioral Cloning (GCBC) approach (Ding al.
2019b) as
the base learning algorithm. This adaptation ensures that the agent learns not just to imitate expert

---

Published as a conference paper at ICLR 2025
(a) RaceCar
(b) RaceCar (Player view)
(c) Maze
(d) HalfCheetah
Figure 5: Illustration of the three different environments used our experiments. The objective is to learn good driving, navigating, and walking policy while minimizing expert interventions.
trajectories, but also to generalize its navigation strategies based on varying goal positions within the maze.
We compare our approach RND-DDAger to multiple approaches from the active imitation learning literature: DAgger Ross et al. (011)) classical approach which queries the supervisor for an action in every state that the learner visits. Ensemble-DAgger Menda et al. (2019), which propose automated approach to measure OOD and trigger expert demonstration. Ensemble-DAgge relies on two metrics: A disagreement measure defined as the correlation between an ensemble of policies, and discrepancy measure computing the difference between the bot action and the expert action at every step. Lazy-DAgger uses the same discrepancy measure as Ensemble- DAgger and reduces bot-expert context switching by using additional threshold parameter define a hysteresis band. Note that, we rely on simplified version proposed in (Hoque et al.) 2021) that has been shown to be more efficient and generates less context switches than the original version. This variant discards the classifier for the benefit of the true discrepancy measure. Human-Gated DAgger (HG-DAgger) from Kelly al. (2019) involves human supervisor monitoring the agent's behavior in real-time and deciding when to take control. While similar in essence to our proposed RND-DAgger, HG-DAgger relies entirely on human judgment to identify when intervention is necessary, rather than using an automatic OOD measure. This requires the expert to continuously observe the agent's 's actions at every timestep. leading to a higher cognitive load and less efficient use of expert time. Finally, also consider Behavioral Cloning (BC) the classical baseline, which does not include any active learning mechanism.
Baselines:
The set of hyperparameters used for the different approaches is described in Appendix A For each method, a first training set is collected by the expert, without active imitation learning techniques. Then the different algorithms executed to collect additional examples and to update the policy.
Metrics: To assess the performance of the methods considered, rely several key metrics: i) The Task Performance measures whether the learned agent is successfully solving the environment. For the Race Car and Maze environments, a lap or reaches the goal). For HalfCheetah, we track the cumulative episode reward, which reflects the overall effectiveness of the agent's locomotion strategy. ii) The Dalaset Size indicates the number of expert actions added to the training dataset, providing insight into how much expert information is being utilized to train the agent. A smaller dataset size suggests the agent is learning efficiently with fewer expert interventions, while a larger dataset implics greater reliance on expert input. iii) The Context Switches which corresponds the nswitch variable Algorithms/ [ and measures the number of contiguous periods during which the expert is actually controlling the agent. This helps quantify how often the expert involved directly providing relevant training samples. For RND- DAgger, this measure accurately reflects the expert's involvement, as the agent is fully controlled by policy between switches. The metrics are averaged over seeded runs for each method.
the success rate (i.e., whether the agent completes
4.1 RESULTS
Oracle-based Performance: To conduct extensive experiments, we replace the human expert with predefined oracle policy. This substitution allows us multiple trials efficiently without depending on human participants, which would be impractical in terms of both time and resources (experiments using human experts reported separately later in the section). For the Race Car
7

---

Published as a conference paper at ICLR 2025
Task Performance
# Context Switch
Env
RC
HC
Maze
RC
HC
Maze
Method
BC
0.883 0.029
2455 1 424
0.367 0.043
DAgger
0.940 + 0.026
2343 + 209
0.450 + 0.074
Lazy-DAgger
0.939 + 0.025
2314 + 278
0.575 + 0.101
3437 + 89
312 + 45
1238 79
Ensemble-DAgee
0.952 = 0.018
2489 = 108
0.626 + 0.045
2785 + 41
1452 - 134
2871 + 94
RND-DAgger
0.944 = 0.014
2490 = 160
0.717 + 0.018
368
= 11
708 + 130
1214 + 25
Table Overall performance: This table illustrates the performance of the different algorithms at the end of the K iterations. It also provides the value of the nswitch variable that accounts for the number expert interventions. Best, Second best
environment, the oracle is learned through a separate active imitation learning process with large interaction budget, ensuring the agent has access high-quality demonstrations, In the 3D Maze environment, the oracle is NavMesh-baae -assed agent, which programatically solves any navigation scenario, making easy to generate trajectory data. For HalfCheetah, we use an open-source reinforcement learning agent (Kuznetsov et al.) (020)) from the RL Zoo repository (Raffin 2020) as the oracle, providing an optimal policy for locomotion tasks. This approach ensures consistency and scalability across experiments.
Table and Figure&presen the performance metrics across the evaluated environments. ensure fair comparison, we report results based on the hyperparameter settings that achieve the highest final performance for each model.
(a) Task performance RaceCar
(b) Task performance HalfChectah
Task performance Maze
(d) Context switches RaceCar
Context switches HalfCheetah
(1) Context switches Maze
Figure 6: Performance and context switches for the different environments. The X-axis corresponds to the size of the training set D that increases each iteration of the algorithms.
Our method, RND-DAgger, demonstrates competitive performance relative to Ensemble-DAgger and Lazy-DAgger, indicating that it effectively learns robust policies across different environments. For example, in the HalfCheetah environment, RND-DAgger achieves a cumulative reward of 2490, compared 2489 for Baaeembbe-DAgeee and 2314 for Lazy-DAgger. These results highlight that RND-DAgger is capable of discovering efficient policies, matching or surpassing the baselines in terms of overall task success. When analyzing performance relative to the size of the training set, RND-DAgger outperforms the other methods in the early stages of the active imitation learning process (Fig. (b). This suggests that RND-DAgger is more effective at focusing on critical states where expert guidance is most needed, thereby gathering valuable feedback more efficiently. By concentrating on key interventions, RND-DAgger accelerates policy improvement and requires fewer training samples achieve comparable superior results, making it especially advantageous in scenarios with limited expert availability. In summary, the experimental results demonstrate that RND-DAgger not only achieves strong final performance but also exhibits more sample-efficient
8

---

Published as a conference paper at ICLR 2025
(a) WRRND-DAA RND-DAgger
(b) Bazy-DAgger
(c) Ensemble-DAgee
Figure Visualization of agent's trajectories (blue) and expert intervention trajectories (red) in RaceCar. RND-DAgger predominantly requests expert intervention in challenging sections of the track, requiring fewer expert interventions overall compared to baseline methods.
learning curve compared existing methods (Fig. 6a
6b
6c)
validating its capability optimize
expert interventions and rapidly improve policy quality.
When examining the number of context switches between the different methods, RND-DAgger consistently results in significantly fewer context switches compared to Ensemble-DAgger across all environments. This indicates that RND-DAgger is more stable and requires fewer handovers between the expert and the policy, reducing the burden on the expert. When compared to Lazy- DAgger, RND-DAgger shows mixed results. the RaceCar environment, RND-DAgger generates fewer context switches (Fig. (dd). highlighting its efficiency in minimizing expert interventions. However, in the HalfCheetah and Maze environments, RND-DAgger produces a similar number of switches (Fig. 6e and 6f This similarity in HalfCheetah can be attributed to the use of an oracle expert policy, which performs consistently without generating diverse actions. Under these circumstances, the discrepancy measure used by Lazy-DAgget which relies comparing expert and policy actions, remains effective for detecting when to switch. In the Maze environment, despite having comparable number of context switches I Lazy-DAgger (Fig. Gf), RND-DDAger achieves higher final performance (Fig. (6c). This suggests that RND-DAgger intervention criterion leads to more informative expert interventions and is better able leverage the expert. Thus, in scenarios like Maze, where achieving high task performance is critical, RND-DA RND-DAgger is the better choice due to its ability drive the agent towards optimal behavior.
Overall, these findings indicate that RND-DAgget strikes a favorable balance between reducing context switches and achieving strong policy performance, making it more efficient and reliable choice.
Qualitative study: gain a deeper understanding of when our algorithm requests expert supervi- sion, Figure 7 visualizes the context switches between the current policy and the expert (additional figures are provided in Appendix [B. As seen the figure, both Ensemble-DAge higher number of context switches com-
and Sazy-DAgger
query the expert significantly more frequently, leading pared to RND-DAgger. The visualization also reveals that RND-DAgger identifies and focuses on critical areas of the track that challenging for the agent during early learning stages. Specifically, it requests expert intervention primarily in the problematic zones, such as the bottom section of the track and the obstacle immediately following the speeding ramp (top-left corner of the track). These are regions where the bot initially struggles, making precise interventions crucial for improving performance. This observation indicates that RNDD-Agger not only minimizes the number of expert queries but also targets the most relevant segments of the environment where guidance is essential. By focusing on challenging states rather than spreading expert effort across all states, RND-DAgger makes more strategic use of expert supervision, resulting in fewer yet more impactful interventions.
Human Expert-based performance: We conducted a series experiments (Figure using real human expert instead of an oracle policy. In this setup, collecting expert actions while the agent was being controlled by its own policy (rather than having the expert take direct control) felt unintuitive for the human participants, making it challenging apply traditional baselines effectively.

---

Published as a conference paper at ICLR 2025
14-
69.
0.8
0.5
RMG-Ologe
AND-0tage
BC
2000
200c
4002
5220 t593
2003
2050
3000
4902
5200
1002
(a) Performance
(b) Performance (time spent)
Figure 8: Task performance on RaceCar with a human expert. (a) RND-DAgger achieves similar task performance HG-DAgger with comparable number of demonstrations. (b) However, by reducing the need for constant expert monitoring, RND-DAgger significantly lowers the total expert time required.
As such we focused comparison on HG-DAgger, which allows the human expert decide when to intervene and take control from the agent.
Our results show that RND-DAgger performs comparably to HG-DAgger, but with a significant advantage: it does not require the human expert to continuously monitor the agent's ' behavior. Instead, RND-DAgger automates the decision of when request expert input based state novelty, reducing the cognitive load on the expert. Figure [egure@hddemonttiee demonstratees the relationship between policy performance and the actual time the expert spends observing the screen. In RND-DAgger, the agent can run accelerated speeds without the need for constant supervision, as the expert only needs to intervene at specific moments rather than watching the agent's 's entire trajectory.
Additional experiments Appendix presents ablation studies of RND-DAgger, including an analysis of the impact of the Minimal Expert Time mechanism (showing how it reduces expert context switches). Appendix D focuses on studying the impact of using Ensemble-DAgger compared to RND-DAgger regarding expert time spent (the metric used Figure (8b), showing how RND-DAgger better minimizes expert burden.
CONCLUSION AND LIMITATIONS
In this work, we presented RND-DAgger, a novel approach to active imitation learning that efficiently minimizes the need for expert interventions by leveraging state-based measure derived from Random Network Distillation (RND). Unlike traditional methods that rely on action- basie discrepancies to detect when to seek expert guidance, RND-DAgger focuses on identifying -atto--distribution states where the agent is most at risk of making errors. This allows our method to selectively request expert feedback only when it is truly necessary, thereby reducing the number of context switches and optimizing the allocation of expert time. Overall, our method provides step forward in developing more practical and numan--fffiiiee imitation learning algorithms, making valuable tool for training autonomous agents in complex environments.
Limitations RND-DAgger requires the expert to immediately take control of the agent, which may be impossible or dangerous in complex real-time environments. Predictive approaches to alert the expert before intervention is an interesting area of improvement. Studying how to scale RND-DAgger to more complex environments such as Partially observable MDPs e.g. pixel-based environments would be a natural extension of our work. In such scenarios, noisy-TV TV problems i.e. environments featuring distracting random or diverse observations, bring important new challenges, which may be addressed through representation learning or explicit high-entrop: filtering. Lastly, incorporating richer forms of expert feedback is another valuable future line of work.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENTS
This work was granted access to the HPC resources of IDRIS under the allocation 2024- AD011015218 made by GENCI.
7 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide detailed pseudo-code in section and section A comprehensive open-source codebase, including all environments, datasets, oracle model checkpoints, active learning algorithms, and detailed guide on how to reproduce our experiments and results is available htlttts:///steee google com/ www.////www/ww//////oeeeeeeeeeee dagger
REFERENCES
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Michael Bain and Claude Sommut. A framework for behavioural cloning. In Machine Intelligence 15, volume 15, pp. 103. Oxford University Press, 1999.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019. URL /attps: met/forund id-iddd-i H11JJnR5
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning, Advances neural information processing systems, 32, 2019a.
Yiming Ding, Carlos Florensa, Mariano Phielipp, and Pieter Abbeel. Goal-conditioned imitation learning. Advances in Neural Information Processing Systems, 2019b. URL http:: /ttttp///rxiic
org hog//bb/19906. 05838
Jack Harmer, Linus Gisslén, Jorge del Val, Henrik Holst, Joakim Bergdahl, Tom Olsson, Kristoffer Sjöö, and Magnus Nordin. Imitation learning with concurrent actions 3d games. In 2018 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2018.
Ryan Hoque, Ashwin Balakrishna, Carl Putterman, Michael Luo, Daniel S Brown, Daniel Seita, Brijen Thananjeyan, Ellen Novoseller, and Ken Goldberg. Lazydagger: Reducing context switching in interactive imitation learning. In 2021 IEEE 17th international conference on automation science and engineering (case), pp. 502-509. IEEE, 2021.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Flyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
Kshitij Judah, Alan Paul Fern, and Thomas Glenn Dietterich. Active imitation learning via reduction to iid active learning. In 2012 AAAI fall symposium series, 2012.
Michael Kelly, Chelsea Sidrane, Katherine Driggs Driggs-Campbell and Mykel J Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. 2019 International Conference on Robotics and Automation (ICRA), pp. 8077-8083. IEEE, 2019.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture of continuous distributional quantile crities, In International Conference on Machine Learning, pp. 5556-5566 PMLR, 2020.
Yihuan Mao, Chengjie Wu, Xi Chen, Hao Hu, Ji Jiang, Tianze Zhou, Tangjie Lv, Changjie Fan, Zhipeng Hu, Yi Wu, al. Stylized offline reinforcement learning: Extracting diverse high-quality behaviors from heterogeneous datasets. In The Twelfth International Conference on Learning Representations, 2024.
11

---

Published as a conference paper at ICLR 2025
Kumal Menda, Katherine Triggs-Cambbell and Mykel J Kochenderfer. Ensembledagger: A bayesian approach to safe imitation learning. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5041-5048. IEEE, 2019.
Nimisha G Nair, Pallavi Satpathy, Jabez Christopher, al. Covariate shift: review and analysis classifiers. In 2019 Global Conference for Advancement in Technology (GCAT), pp. 1-6. IEEE, 2019.
Antonin Rafiin, RI baselines3 zoo. 2020.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 2011.
JMLR Workshop and Conference Proceedings,
Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6):
233-242 242, 1999.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782)3350-354 2019.
Derek Yadgaroff, Alessandro Sestini, Konrad Tollmar, Ayça Özęelikkale, and I. Linus Gisslén. Improv- ing generalization in game agents with data augmentation in imitation learning. In IEEE Congress on Evolutionary Computation, CEC 2024, Yokohama, Japan, June 30 July 5, 2024, pp. 1-8. IEEE, 2024. doi: 10.1100/CEEC6000..0004444 10612178 UtLAttpss//aoi.o////00 CEC60901.0 2024.02220 10612277
Jiakai Zhang and Kyunghyun Cho. Query-efficient imitation learning for end-to-end simu- lated driving. In AAAI Conference on Artificial Intelligence, 2017. URL https::/app
semanticscho.ee org/ CrC.Coops
12

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL DETAILS
Dataset Figure 9a and 9b showcase examples of generated trajectories for the RaceCar and Maze
environments.
Episode traces
Episade traces
(a) RaceCar dataset examples
(b) Navigation dataset examples
Figure 9: Dataset examples: the trajectories of the expert from which bootstrapped train our
initial BC policies and out BC baselines.
Hyperparameters
For each decision rule, several key hyperparameters had to be tuned:
DAgger
The probability /3 of frame be controlled by the bot. The probability is decreased at each DAgger epoch by 3 88-1
RRND-DAgger
Threshold of OOD detection
The historic context length, that the number of frames in the past we take along with the current state to detect its OOD) nature
The Minimal Expert Time W
The size of the random network, represented by the number of layers and neurons per layer the predictor and target networks have.
Ensemble-DAgger
Threshold 7 for discrepancy measure
Threshold X for doubt measure
The number of models N
Lazy-DAgger
<Threshold 3Tr for discrepancy measure
-Threshold 3R for the backward controlled loop (i.e. the criterion to switch back from expert to autonomous)
Computing thresholds For all the methods that necessitate at least one threshold (RND-DAgger, Lazy-DAges Ensemble-DAggent we computed them following the following methodology: the threshold was be positive factor of the mean measure the training set. In other words, before each new sessions, the measure was ran over the training dataset, and the threshold (x and for Basemble-DAggeee 34 for Lazy-DAge and À for RND-DA were calculated
A MEASURE (lar, 8t) E Derain)) )) L
(1)
13

---

Published as a conference paper at ICLR 2025
Table 2: Hyperparameter search
(a) Ensemble Ensemble-DAgger
Hyperparameter
Values
RaceCar
HalfCheetah
Maze
factor
0, 1, 1.5, 2, 3,4 00,1, 1.5, 2. 3,4 [2, 3,
1.5
0
0.5
factor
1.5
1.5
N
5
5
(b) RND-DAgger
Hyperparameter
Values
RaceCar
HalfCheetah
Maze
À factor
1, 2, 3, 4
Hidden size
[32, 128
32
128
32
Number of layers
[0, 1.
0
Historic context length
00, 5, 10, 15]
10
0
2
W
1,2, 5, 10, 15, 30, 50]
30
5
5
(c) Lazy-DAgee
Hyperparameter
Values
RaceCar
HalfCheetah
Maze
31 factor
0, 2, 1.5 2, 4
1.5
2
Bn divider
2
1.5
2.5
(d) DAgger
Hyperparameter
Values
RaceCar
HalfCheetah
Maze
B
[0.2, 0.5, 0.6, 0.7 0...0.0...88.0.900099
0.5
0.7
0.8
With L being a positive factor, a hyperparameter to be tuned. For the second threshold of Lazy- DAgger, set it to be factor of the first one: 3R B2 LL, with L being the hyperparameter tuned. That's a different method compared to other methods, such as Zhang & Cho (2017) or Hoquue et al. (2021) who set their threshold so that approximately 20% of the initial dataset is unsafe, or Dessmble--Agger//
et al.] (2019) that directly grid searched the value. The Table summarizes the values used for our grid search.
B INTERACTIVE SESSION VISUALIZATIONS
Typical interactive situations In Figure one can understand better how, in the RaceCar environment, an oracle (and by extend a human taken as an expert) would interact with the learner. In (a). the current learned policy has control of the car, and the OOD measure (green circle) below the threshold: the learner is confident. In (b), the car crashes into wall, so the measure (red circle) queries the expert to take control (transparent red capsule). From (c) to (e), the expert demonstrates how to get back on track, until (f) the measure falls below the threshold again (green circle) for at least W consecutive frames (see Section 3 for further explanations)
Qualitative study On Figure 10 we reported the context switches and full trajectories on both the RaceCar and Maze environments. The results are taken from the same session and the same seed better compare the figures. From (a) to (b), it clear that Ensemble-DAgger exhibits the most context switches compared to Lazy-DAgger and RND-DA which show comparable counts. However, the context switches Lazy-DAgger are more concentrated in specific areas, whereas those in RND-DAgger are more evenly distributed. Then, from (d) to (f) we can see the corresponding episodes with expert demonstrations (in red) added to the dataset during that session. We can see that the expert demonstrations are either way longer (the whole episode) or way shorter (only few frames) for Lazy- -DAgeer compared to our method, providing more consistency throughout the map. The difference is clearer the RaceCar environment. Indeed, on both context switch figures, that Sazy-DAgger and Ensemble- -Dgger are comparable, and that the queries for RND-DA DDD-Aggeer are concentrated around only zones that generated most of the failure cases.
14

---

Published as a conference paper at ICLR 2025
(a) RND-DDAger
(b) Lazy-DAgger
(c) Ensemble- DAgger
(d) RND-DAgger
(e) Lazy-DAgger
(f) Ensemble-DAge -Dgger
RND-DAgger
(h) Lazy-DAggee
ii Ensemble CeeumbleeDDggerr
(i RND-DAgger
(k) Lazy-DAgger
(I Ensemble-DAre -Agger
Figure 10: We conduct a qualitative analysis the the different query methods. (a)-(d) and (g)-(i): Each dot represents context switch transition from autonomous to expert control. (e)-(h) and (i):(): In red, the expert demonstrations, in blue the novice segments.
15

---

Published as a conference paper at ICLR 2025
(c) 2
(d)
(e)
(f)
Figure 11: (a) The agent has control. (b) The OOD measure is triggered (RND), the expert given control. (c)-(e) The expert demonstrates how to recover. (f) The state is in distribution, the expert keeps the control until the Minimal Expert Time reached.
C ABLATION STUDY
C.I RND-DAGGER ON HALFCHBETAI
In this ablation study, we evaluate RND-DAgger, focusing on the additive advantages of three key enhancements: Minimal Expert Time (MET) of size W, historical context, and the architecture of the random networks (see Section 2 for insights on the values tested). The addition of a Minimal Expert Time should increase the quality the frames by letting the expert finish its demonstration and making sure the learner is fully recovered before letting it the control again. The historical context enriches decision-makinn them to the current state. This particularly helps in tasks where there is strong relation between the states. For example, a state where the car close to a wall is not necessarily bad state, unless the car is going towards it for several frames in row. However, historical context chosen too big detrimental, because it can overshadow useful dimensions of the sate, and thus preventing the predictor network seize useful information.
by incorporating temporal information from past actions and concatenating
Results that ablation are reported Table and Figure 12 We note that the Minimal Expert Time W helps reducing the context switching, without impacting that much the performance the end of the sessions of RND-DAgger. In other words, helps decreasing the expert burden, without impacting the task performance.
300
100
Cogger
1500
I6D0
(a) HalfCheetah (Task perf)
(b) # context switches
Figure 12: Ablation study on our method. RND-DAgger compared to: (light blue) RND w/o MET, (orange) bigger historical context, (dark blue) smaller architecture for the random networks.
16

---

Published as a conference paper at ICLR 2025
Task Performance
# Context Switch
BC
2455 + 424
RND-DAgger
2490 + 160
708 + 130
RND-DAgger MET
2432 + 175
1985 215
RND-DAgger small
2167 + 140
1136 + 76
RND-DAgger historic
2172 + 290
534 90
Lazy-DAgger
2314 + 278
312 + 45
Table 3: Performance and context switches the end of the sessions. We report the results of the key ablations on RND-DAgger, along with the results of Lazy-DAgger for further comparisons
C.2 MINIMAL. EXPERT TIME FOCUS
Ablation study on all the methods We further investigated the impact of the Minimal Expert baselines, across all the tasks. We reported the results of that
Time (MET) on method and study in Figure 13 and T In particular, we can see that LageDAgger MET isn't appropriate in HalfCheetah. The combination of the Lazy mechanism (introduction of a second threshold create hysteresis band) and the MET forces the expert to always have control over the agent, which explains the extremely low value for the context switches (which happens only once per episode) and the poor on RaceCar,
performance learning curve (Fig. D3b.. The same remark holds for Ensemble-DAge but this case, this behavior doesn't 't seem impact the task performance of the method (Fig. 13a
Soon
Suger
(a) Task performance RaceCar
(b) Task performance HalfCheetah
(c) Task performance Maze
(d) Context switches RaceCar
(c) Context switches HalfChectah
(f) Context switches Maze
Figure 13: Results for the ablation study with focus on the addition of the Minimal Expert Time (MET) on the baselines Ensemble-DAgger and LazyDAgger. RND-DAgger remains the same as in Table T
Qualitative interpretation On Figure 14 we report the context switches on the RaceCar environ- ment, the second DAgger epoch of given seed with and without a MET. We can clearly see that the number of context switches increases and a third critical zone appeared at the top right of the map for RND-DAgger (Fig. 14ah. We interpret that the bot fairly confident in the top straight line the course, but the last turn is difficult to manage when the velocity of the car is too high, inducing query to the expert. There are also more context switches at the entrance of that straight line, because the learner has be precise pass without touching the walls the speed ramps the sides. We conclude that the MET increases the quality of the demonstrations without impacting the quality of the measure, that still grasps the relevant critical areas of the track.
17

---

Published as a conference paper at ICLR 2025
Task Performance
Context Switch
RC
HC
Maze
RC
HC
Maze
Lazy-DAgger + MET
0,943 1 0.011
2374 = 325
0.575 1 0.101
759 1: 39
78 1: 0.0
876 + 71
Lazy-DAggee
0,939 0.025
2314
0.590 + 0.057
3437 :
312 45
1238
Ensemble-DAge Dagabeee--Dggrraaa
1: 0.023
Ensemble-DAge DDgger
0,952 +: 0.018
2489 108
0.626 1 0.045
2785 1 41
1452 134
2871 1:
RNDD-DAgger
0.944 + 0.014
2490 160
0.717 + 0.018
368
11
708
130
1214 + 25
RND-DAgger w/o MET
2432 175
1985 + 215
Table 4: We conduct an ablation study on the impact of the Minimal Expert Time (MET) on all baselines across our three environments. We report the final task performance and number of context switches. We conducted the same grid search for the value of W as for RND--Agger (see section A for more details).
AND-DAgger W
(b) Misembbe-DAggee W
(c) LazeDAgger W
L
(d) RND-DAgger W 30
(e) Ensemble Dassebble-DAgger W 30
LazyDAgger W
Figure 14: Ablation study: a focus on the effect of the MET on the RaceCar environment. We see an
increase of the context switches and shorter episode traces as there no Minimal Expert Time for all the methods. RND-DAgger seems to have better grasp the difficult zones of the track, whereas the two other measures trigger the expert rather uniformly across the track.
D TOTAL EXPERT TIME COMPARISONS
While focusing solely on task performance and context switches provides important insights on the efficiency of method and on given task, it doesn't tell the complete story about expert burden. We conducted an additional analysis the Total Expert Time, as we did in the Fig 8b on human experiments. The Table TableB]shows shows that Ensemble- -Dgger is highly time consuming for the expert without statistically outperforming RND-Dagger. For example, on Maze, it is three times longer to achieve a similar performance than with RND-DAgger, even with the addition our MET mechanism on that baseline. This demonstrates that are efficient in real human expert
18

---

Published as a conference paper at ICLR 2025
scenarios. Moreover, and especially in a fast-paced environment like RaceCar, that baseline and its improved MET variant requires the expert to input actions at the same time as the agent to be able to compute the discrepancy measure, which is not natural in practice.
Task Performance
Total Expert Time HC
RC
HC
Maze
RC
Maze
RND--Agger
0.944 0.014
2400 160
0.717 + 0.018
(27.8 + 0.1 mins)
(6.60 + 0.06 mins)
(27.0 0.1 mins)
313394 = 506
21535 774
51434 = 1507
DAgger
mins)
mins)
mins)
18216 +: 165
1.4474 = AS1
48125 9IXT
Ensemble-DAgger MET
0.935 0.019
2502 228
0.707 + 0.023
(30.4 = 0.3 mins)
(7:40 +: 0.35 mins)
(80.2 mins)
Table The results considering the Total Expert Time i.c. the total number of frames which the expert had to be involved to provide an action. In the case of Ensemble-DAgger, the expert action needed at each time step compute the discrepancy measure between the mean action of the ensemble of policies and the expert action. The time spent in minutes is computed with typical frame rate 30 HalfCheetah (arbitrary) and 60 with one action taken every frames, for the two game environment
19


Reviews:
[{"review_id": "FwIg1R0FkO", "reviewer": "Reviewer_USwa", "summary": "This work presents a method to more effectively integrate human feedback with imitation learning and specifically the IL method dagger. The idea is to only request human feedback when the policy is out of distribution. The data collected by the expert is then added to a buffer for training. In doing so, the algorithm is able to more effectively utilize the experts time and feedback.", "questions": "I wonder are there other comparisons here? Do they all have to be dagger variants? Is there another type of “expert takeover” method that you could compare to? If I missed this in the text I apologize.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The research problem this work focuses on is good. Increasing the efficiency of imitation learning is an open problem and the most effective way to utilize expert data within IL. Improving the interaction between expert and policy increases the ability of IL to be used with real world problems.\n\nThe contribution of this work is ok, they propose their method and perform a study to verify their claims.\n\nThe algorithm is clearly defined and could be implemented from the information given.\n\nThe experiments run are reasonable and seem to demonstrate their method well.\n\nThe baseline comparison are all dagger variants (and BC). They are definitely reasonable comparisons.\n\nThe empirical results are good. They show a balance between expert interventions and performance which is the goal of this work.\n\nThe clarity is great. This paper was very easy to read and very clear.", "weaknesses": "The novelty of this work is minimal. As far as I understand it the methods generally used in this work are all previously known. Dagger and the OOD classifier seem to be like the main components used but are previous work.\n\nThe statistical rigor needs improvement. The metrics are only averaged over 8 seeds. Is there a reason for only this many? I feel like it should be many more.\n\nAs well, I want to see confidence intervals on Table 1.\n\nThere is no failure analysis. I wish there was one on the times the method does not perform better than the previous methods. What about those tasks makes it not as good? \n\nThe future work and conclusion is ok. I wish there was a better future work section. I think a limitation of this work is that it works well in simulated environments but on an actual self-driving car how would this work? The car couldn’t just stop running. This could be an interesting direction of future work some like predicting beforehand that you’re going to get an out of distribution state and request input.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HHZA20nNw8", "reviewer": "Reviewer_r6sZ", "summary": "This paper presents Random Network Distillation DAgger (RND-DAgger), an active imitation learning method that leverages RND to define an out-of-distribution states, enabling selective expert intervention and minimize the frequency of transitions between human experts and learning agent through a minimal demonstration time mechanism. The approach is evaluated across Race Car, Maze, and Half Cheetah environments.", "questions": "They are mentioned in Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper presents a well-founded motivation, aiming to optimize the timing of expert interventions to reduce overall costs associated with human expertise and minimize the frequency of transitions between human experts and learning agent.\n- The paper is straightforward, particularly for readers with a background in the domain of unsupervised RL.", "weaknesses": "- **Novelty:** The paper's novelty appears constrained, as it predominantly builds upon an established novelty measure within the unsupervised RL domain. This integration approach mirrors Ensemble-Dagger, which relies on a principle similar to Disagreement in the unsupervised RL domain.\n\n- **Limitations:** The limitations specific to this method are not sufficiently addressed. Certain limitations noted by the authors for other approaches may also apply here. For instance, the paper states, “While this approach works well when the expert is optimal and acts deterministically, it becomes problematic when dealing with humans or imperfect experts.” It is unclear how this method manages these challenges, and if not addressed, this issue should be explicitly acknowledged.\n\n- **Baselines:** The study lacks comparisons with more recent methods that similarly leverage human input for out-of-distribution states, such as RLIF [1], PATO [2], and Sirius [3].\n\n- **Time Mechanism:** This approach is reliant on two factors: RND and Minimal Expert Time. It would be valuable to assess how Minimal Expert Time operates alongside other baseline methods - a potentially straightforward inclusion. This would provide a clearer understanding of RND’s advantages over alternative metrics.\n\n- **Experiments:** Additional experiments are needed to support the authors' claims regarding task performance and context switching. From the current results (Table 1), Ensemble-Dagger performs better in RC settings but with higher context switches, whereas Lazy Dagger shows the opposite trend in HC settings. Further experiments in more challenging environments, such as Adroit [4], would provide deeper insights and strenghten the paper’s claims. Moreover, incorporating the Time Mechanism into other methods may enhance their context-switching capabilities, offering a compelling comparison.\n\n- **Ablation Study:** The ablation study section would benefit from greater detail, as the current presentation makes it challenging to interpret the impact of various factors on learning outcomes.\n\n**References**\n\n[1] Luo, Jianlan, et al. \"RLIF: Interactive Imitation Learning as Reinforcement Learning.\"arXiv preprint arXiv:2311.12996 (2023).\n\n[2] Dass, Shivin, et al. \"Pato: Policy assisted teleoperation for scalable robot data collection.\"arXiv preprint arXiv:2212.04708 (2022).\n\n[3] Liu, Huihan, et al. \"Model-based runtime monitoring with interactive imitation learning.\"2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.\n\n[4] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cOddHTEXUd", "reviewer": "Reviewer_dV7L", "summary": "The paper focuses on addressing the covariate-shift problem in behavioural cloning, where the agent suffers from compounding error from the predicted actions. The paper considers DAgger-like algorithms where the process includes querying an expert to add extra demonstrations, thereby increasing state-action coverage. The paper identifies a gap where existing approaches require the said expert to be present for a long period of time, which can be costly for some applications. The paper thus propose a method that considers out-of-distribution in states through random network distillation. The paper conducts experiments on three simulated environments, demonstrating that the proposed algorithm achieves similar performance as existing DAgger-like baselines, as well as the reduced number of expert queries.", "questions": "- Experiments:\n\t- In Figure 6, do the algorithms use same initial datasets? Does increasing in x-axis mean starting at a larger dataset? Or this corresponds to increasing size due to the conditions to query from the expert?\n\t- How do Figure 6e and 6f demonstrate that RND-DAgger is better in \"sample-efficiency\"? Context switching does not totally correspond to number of samples queries from the expert. Can the paper clarify this point?\n\t- On page 9, there is a claim \"..., RND-DAgger is more effective at leveraging expert feedback to improve the policy.\" How is this statement true exactly? The policy training at the end is the same but I suppose the aggregated data is somewhat different?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea is very simple and focuses on a gap that previous approaches do not consider (i.e. OOD states rather than action mismatch).\n- The algorithm appears to be practical compared to existing approaches.", "weaknesses": "I am happy to increase the scores if these comments are addressed.\n\n**Comments**\n- I am unsure why the formulation is POMDP rather than MDP. The environments used in this paper appears to be using state-based information, in the sense that I cannot really guarantee that they are partially-observable, as opposed to using images that will be way more convincing.\n\t- I further believe that the current method assumes the observation aliasing is not a problem. Self-driving environment can dramatically different scene while similar actions.\n\t- I suggest including the CARLA environment to make this result more convincing.\n- Regarding the approach, experimentally how is this algorithm different from the earlier variants since $f_{targ}$ can be $\\pi_{exp}$? In other words, is LazyDAgger essentially doing RND but have a slightly higher-dimensional output space? If that is the case, then is the benefit coming from the additional \"minimal demonstration time\" mechanism?\n- The paper can improve upon its writing quality:\n\t- Algorithms 3 and 4: Include the actual definition of *measure* for clarity.\n\t- Table 1: What do bolded and underlined terms mean? Is this taking the average or some other statistics? What about the standard error or other statistics?\n\t- Figure 6: What do the solid line and shaded area correspond to?\n\t\t- Also recommend the top curve to use different line style to differentiate variants easier.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0TzwMuiPhK", "reviewer": "Reviewer_Qs6v", "summary": "The authors address efficient expert intervention in active imitation learning for complex, reward-free tasks like video game bots and robotic control. The proposed RND-DAgger framework uses a state-based out-of-distribution (OOD) measure to trigger expert input selectively, reducing intervention needs by focusing on critical states. Experiments in RaceCar, 3D Maze, and HalfCheetah environments demonstrate that RND-DAgger outperforms traditional methods, achieving high performance with fewer expert queries. Ablation studies highlight the importance of the stability window and historical context in reducing context switches and expert burden.", "questions": "1. The paper lacks necessary comparisons with methods specifically designed to address out-of-distribution scenarios, aiming to generalize to states unobserved during training without requiring expert intervention [1, 2]. Including these comparisons is essential to demonstrate the completeness of the proposed method in handling such challenges and I am willing to raise the scores.\n\n2. While the paper presents results from four environments, the setup may lack sufficient diversity to fully assess RND-DAgger’s adaptability. Would the authors consider evaluating on additional environments, such as robot manipulation or hand dexterity tasks, which could reveal the method’s robustness and effectiveness across a broader range of complex control scenarios?\n\n3. In which types of tasks or environments does RND-DAgger face challenges, especially with regard to state-based out-of-distribution detection?\n\n4. How does the RND mechanism manage computational efficiency, and is it feasible to optimize it further for large-scale applications?\n\n5. What role does the \"minimal demonstration time\" play in balancing expert workload and agent training quality, especially in complex environments?\n\n[1]  Shang-Fu Chen, Hsiang-Chun Wang, Ming-Hao Hsu, Chun-Mao Lai, and Shao-Hua Sun. Diffusion model-augmented behavioral cloning. In International Conference on Machine Learning, 2024.\n\n[2] Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robotic Learning, 2022.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. **Motivation and Intuition**: The motivation for reducing expert intervention in active imitation learning is convincing, particularly in applications where continuous expert availability is costly or impractical.\n\n2. **Novelty**: The paper introduces the idea of leveraging Random Network Distillation (RND) to manage expert interventions, which is innovative and well-suited for identifying critical OOD states.\n\n3. **Clarity**: The paper is well-organized, with clear explanations of both theoretical foundations and implementation details. Figure 4 effectively illustrates the RND-DAgger algorithm, and ablation studies clarify the effect of each component.\n\n4. **Experimental Results**: The visualizations of experimental results are clear. Figure 6, for instance, provides an accessible overview of task performance and the frequency of context switches across environments, highlighting RND-DAgger’s reduced reliance on expert intervention.\n\n5. **Reproducibility**: The author's inclusion of pseudo-code, a detailed description of the experimental setup, and an appendix for hyperparameters strengthens reproducibility. They promise to release code upon acceptance also supports this.", "weaknesses": "1. **Experiment Setup**: The environments in the main experiment, although tested on four different setups, lack sufficient diversity, limiting the evaluation of RND-DAgger’s performance across a broader set of tasks. Incorporating additional environments, such as robot manipulation or hand dexterity tasks, would enrich the setup and test the method’s adaptability in more complex, fine-grained control scenarios.\n\n2. **Related work**: The paper needs to include comparisons to the methods that specifically designed to handle out-of-distribution scenarios, aiming to generalize to states unobserved during training without requiring expert intervention [1, 2]. \n\n[1]  Shang-Fu Chen, Hsiang-Chun Wang, Ming-Hao Hsu, Chun-Mao Lai, and Shao-Hua Sun. Diffusion model-augmented behavioral cloning. In International Conference on Machine Learning, 2024.\n\n[2] Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robotic Learning, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)