Efficient Active Imitation Learning with Random Network Distillation

## Abstract

Abstract Developing agents for complex and underspecified tasks, where no clear objective exists, remains challenging but offers many opportunities. This is especially true in video games, where simulated players (bots) need to play realistically, and there is no clear reward to evaluate them. While imitation learning has shown promise in such domains, these methods often fail when agents encounter out-of-distribution scenarios during deployment. Expanding the training dataset is a common solution, but it becomes impractical or costly when relying on human demonstrations. This article addresses active imitation learning, aiming to trigger expert intervention only when necessary, reducing the need for constant expert input along training. We introduce Random Network Distillation DAgger (RND-DAgger), a new active imitation learning method that limits expert querying by using a learned state-based out-of-distribution measure to trigger interventions. This approach avoids frequent expert-agent action comparisons, thus making the expert intervene only when it is useful. We evaluate RND-DAgger against traditional imitation learning and other active approaches in 3D video games (racing and third-person navigation) and in a robotic locomotion task and show that RND-DAgger surpasses previous methods by reducing expert queries. https://sites.google.com/view/rnd-dagger

## 1 Introduction

Imitation learning has increasingly become a favored approach for learning behaviors in complex environments, offering a compelling alternative to classical scripted behaviors implemented by domain specialists (Schaal, ; Hussein et al., ). It is particularly well suited in problems where there is not a clear performance measure (or reward). In video games, it is becoming more and more familiar to game developers that frequently address the problem of implementing bots in their games which must play in realistic ways (Harmer et al., ; Yadgaroff et al., ; Mao et al., ).
Indeed, since the notion of realism is not well specified, it prevents the use of reinforcement learning-based approaches where a reward signal is mandatory. By observing and replicating human players, these bots are trained to execute complex strategies and actions that are both efficient and human-like.

Imitation learning usually proceeds in two steps: first, a dataset of behaviors is built by leveraging experts interacting with the dynamical system. Then, a statistical model (e.g. a neural network) is learned to imitate actions in that dataset, thus expecting this model to generalize to unseen state and to behave like the experts. Consider a use case where the goal is to train a driving policy in a video game context capable of controlling a car on a track (see figure 1). In this scenario, the state is defined by sensor values at time $t$, which can include information such as the position of the car, its speed, and raycasts. In this context, Imitation Learning involves manually controlling the car for many laps to create a learning dataset, training a policy on this dataset using traditional algorithms like Behavioral Cloning (Bain & Sommut, ), and expecting that the resulting bot will control the car correctly. However, this approach is known to be unreliable, particularly due to the risk of a shift in the state distribution between training and testing (Ross et al., ), known as the problem of covariate shift (Nair et al., ). When covariate shift occurs, the agent struggles to determine the correct action, leading to compounding errors and ultimately undermining its performance.

Facing this issue, the effectiveness of imitation learning hinges on the availability of extensive datasets comprised of player behaviors, often necessitating thousands of expert traces to achieve reasonable performance and exhibit credible behaviors (Vinyals et al., ; Mao et al., ). This dependency on large volumes of data poses a challenge, particularly in scenarios where collecting such data is either impractical or costly.

In response to these limitations, some approaches involve human-in-the-loop training framework, often referred to as active imitation learning (Ross et al., ; Judah et al., ; Menda et al., ; Hoque et al., ). Instead of gathering very large amounts of demonstrations, these methods focus on the strategic incorporation of human feedback to selectively guide the learning process, thereby enhancing the efficiency of the training phase. By prioritizing the acquisition of relevant and impactful expert traces, these approaches seek to expedite the agent’s learning curve without relying on expansive datasets, and swiftly improve performance. In the aforementioned video game example, this corresponds to collecting a first set of players traces to bootstrap the agent using classical imitation learning, and then to gather additional traces in particular race settings during training, i.e. requiring a player to control the car in particular and relevant situations.

Figure: Figure 1: RND-DAgger overview. (a) The learner’s policy controls the agent until (b) our Random Network Distillation-based out-of-distribution (OOD) measure is triggered. Then (c) the expert takes control until the OOD measure gets lower than the threshold for at least W steps. (d) At last, the current policy takes control of the agent to continue the episode, and can trigger the expert again later.
Refer to caption: x1.png

To tackle these challenges, several recent models have been proposed to identify when the expert’s intervention is necessary to correct the agent’s suboptimal behavior (Zhang & Cho, ; Menda et al., ; Kelly et al., ; Hoque et al., ), with DAgger (Ross et al., ) being one of the most well-known approaches. The primary objective of these methods is to minimize the time required for human feedback by concentrating interventions on the most critical situations where the agent is likely to make errors. This allows for a more efficient use of expert time, ensuring that corrective feedback is provided only when it is essential. But, as explained in Section 2, many of these methods are not fully satisfying.

Despite these advancements, designing algorithms that effectively incorporate human feedback while reducing the overall expert effort remains an open research problem. Addressing this issue is the central focus of this article, as we aim to develop methods that optimize the interaction between the expert and the learning agent, thereby improving learning efficiency without overburdening the human supervisor.

Our main contributions are threefold: i) We propose a new method called RND-DAgger which is a novel interactive imitation learning approach leveraging state-based out-of-distribution identification through random network distillation. ii) We perform a comparative analysis of RND-DAgger and existing methods on 3 tasks: a robotics scenario and two video-game environments. iii) Throughout these experiments, we demonstrate that RND-DAgger either outperforms or matches existing approaches in terms of final performance while significantly reducing expert burden.

## 2 Preliminaries and Related Models

#### Notations

Let us consider a Partially Observable Markov Decision Process described by a state space $\mathcal{S}$, an observation space $\mathcal{O}$ and an action space $\mathcal{A}$. The dynamics of the process is defined by two unknown distributions: i) the observation distribution $P(o_{t}|s_{t})$ with $(o_{t},s_{t})\in\mathcal{O}\times\mathcal{S}$ and ii) the transition function $P(s_{t+1}|a_{t},s_{t})$ where $a_{t}$ lies in $\mathcal{A}$. Note that we do not consider any reward in this setting since the task to solve is not explicit (e.g. driving a car), our objective being typically to imitate human behaviors that may be suboptimal and diverse (e.g. if considering multiple players or experts). An episode $\tau$ is composed of a sequence of observations and actions $\tau=(o_{1},a_{1},...,o_{T_{\#\tau}},a_{T_{\#\tau}})$
where $\#\tau$ is the size of the episode. Given a policy $\pi(a_{t}|o_{t},a_{t-1},o_{t-1},....,o_{1})$, it is possible to sample an episode by sequentially executing the policy until reaching a stop criterion.

Figure: Algorithm 1 DAgger

#### Distributional shift:

Given a dataset of expert demonstrations $\mathcal{D}=(\tau_{1},\dots,\tau_{n})$, a common approach to policy learning is Behavioral Cloning (BC) (Bain & Sammut, ; Ding et al., ), a straightforward imitation learning algorithm. BC enables to learn policies by maximizing the log-likelihood of the expert actions within the dataset.
However, learning a policy from a fixed dataset can result in the policy encountering out-of-distribution (OOD) states during inference, where it has not been trained to make accurate decisions (Ross et al., ). This phenomenon is referred to as distributional shift. This shift in the distribution can cause the policy to perform poorly, as it may not know how to handle unseen situations. For example, consider a driving policy in a racing game. If the dataset is composed only of demonstrations from expert drivers, the learned policy might struggle to recover from rare events such as a spin-out, since such situations are absent from the expert demonstrations. Therefore, the policy may fail to generalize effectively to these unseen states.

This effect can be mitigated by providing more diverse and comprehensive training sets, ensuring that the policy is exposed to a wider range of scenarios, including edge cases. However, this raises the challenge of how to efficiently collect such datasets while minimizing the burden on experts. Gathering enough data to cover all possible situations can be time-consuming and costly, especially if it relies heavily on expert demonstrations. Thus, the key question becomes how to strategically collect diverse and representative data in a way that maximizes coverage of the state space while minimizing the amount of effort required from experts.

#### DAgger and variants:

To address this issue, one can utilize the DAgger algorithm from Ross et al. () (Algorithm 1), which iteratively expands the training dataset through the intervention of an expert, represented by a reference policy denoted as $\pi_{\text{exp}}$. The reference policy serves two key roles: first, it determines the appropriate action to execute at each timestep based on a specified decision rule (line 7) – a mixture between the current policy and the expert one. Second, it helps augment the training set by adding new samples of states and corresponding expert actions, which are then used to improve the learned policy (line 11). At regular intervals, the policy is updated via imitation learning using the expanded dataset, and this process is repeated over multiple iterations. This method helps to mitigate the issue of distributional shift by continually refining the learned policy with fresh data that captures a broader range of states.

To better sample relevant states, few variants of the DAgger algorithm have been proposed. One of the most relevant is Ensemble-DAgger (Menda et al., ) which introduces a decision rule to decide if the agent is controlled by the current policy, or by the expert one. Moreover, pairs of state-actions are added to the learning dataset only when the action comes from the expert policy (see Figure 2 - right ). The decision rule is a discrepancy measure that computes the distance between the expert action and the policy action (combined with a disagreement measure between a mixture of experts) as shown in Algorithm 3. This approach is not realistic when considering human experts. Indeed, let us revisit our car driving example, where the goal is to learn an effective driving policy for a video game. Using the DAgger and EnsembleDAgger approaches, the expert (i.e. the player) must continuously play the game during the active imitation learning process, while the car is only partially controlled by the current learned policy. This setup has two significant drawbacks: i) First, the player is placed in an unnatural setting, where they are expected to control a car that they do not fully manage. This can create a disorienting experience, as the player must constantly adapt to actions taken by the policy, disrupting the natural flow of gameplay. ii) Second, the player remains engaged even when the car is exhibiting good behavior, spending time providing feedback that may not significantly contribute to discovering a better policy. This leads to inefficiencies, as the expert’s time is not always optimally utilized, especially in situations where the current policy already performs well.

Figure: Algorithm 3 Condition for Ensemble-DAgger

## 3 Random Network Distillation-based DAgger

To address the aforementioned limitations of Menda et al. (), Lazy-DAgger (Hoque et al., ) proposes to involve the expert only during critical moments rather than requiring continuous control. Instead of relying on the expert to partially guide the agent throughout the learning process, this approach alternates between periods where the agent is completely controlled by the current policy and periods where it is fully controlled by the expert. During the phases controlled by the current policy, the expert’s intervention is not required, thereby reducing the time the expert spends supervising the agent and making their interventions more targeted and efficient.

Figure: Figure 4: The RND-DAgger algorithm proceeds in $K$ iterations. First a policy is trained on a small expert dataset (line 2). Then at each iteration (line 6), it constructs a new dataset $D_{i}$ with $T$ state-action pairs from the expert (line 7). At each timestep, the agent is by default controlled by the current policy (line 23). The OOD measure is computed by using the prediction network $f_{pred}$ and the target network $f_{targ}$ (line 9). If this measure is greater than a threshold (line 10), the expert takes over the policy to control the agent (line 16) and to add samples to the dataset (line 17). The number of context switches is the number of times the expert has taken control (line 19). At the end of the iteration, the built dataset is aggregated with the current learning set (line 27) and the policy is retrained (line 28). In addition, the $f_{pred}$ network is also updated (line 29) to allow the future detection of OOD states. The $w$ counter is used to ensure that the algorithm waits at least $W$ timesteps below the threshold before taking control back of the agent.

Lazy-DAgger replaces the traditional action-based discrepancy measure between expert and policy actions with a classifier-based approximation — see Figure 4. This approach allows the algorithm to predict when expert intervention is necessary without requiring an expert action at every timestep, thereby significantly reducing the expert burden compared to methods like Ensemble-DAgger. However, this method has a notable drawback: similar to Ensemble-DAgger, it relies on comparing the actions of the expert and the current policy for a given state. While this approach works well when the expert is optimal and acts deterministically, it becomes problematic when dealing with humans or imperfect experts. Human experts often exhibit variability and may choose different actions for the same state depending on context or personal preference. This results in a noisy and unreliable measure of discrepancy, causing the method to trigger unnecessary interventions or overlook critical corrections, ultimately reducing its effectiveness.

We propose the RND-DAgger algorithm, which builds upon Random Network Distillation (RND) (Burda et al., ). The core assumption of RND-DAgger is that expert feedback is only necessary when the agent encounters out-of-distribution (OOD) states—states that are not well represented in the training set and where the policy is more likely to fail. This is crucial because when an agent operates in OOD states, it faces a higher risk of taking suboptimal actions that could hinder learning or lead to unsafe outcomes. Once the agent returns to familiar, in-distribution states, the expert’s feedback is no longer required, and the intervention is ended. Our method differs fundamentally from existing approaches like Ensemble-DAgger and Lazy-DAgger, which rely on action-based discrepancy measures to determine when to intervene. Thus, our approach remains robust to variations in expert behavior and ensures that interventions only occur when the agent genuinely faces unfamiliar and potentially risky states.

To further improve the stability of expert interventions, we introduce a mechanism called minimal demonstration time. This concept addresses the issue of overly frequent and brief expert interventions, which can interrupt learning and increase the cognitive load on the expert. Minimal demonstration time defines a lower bound on the duration for which the expert must maintain control once they start an intervention. The key idea is that the expert should provide a sufficient number of consecutive corrective actions to guide the agent back to a stable state, instead of immediately handing control back to the policy after a single correction. We describe these two components below. The detailed algorithm is provided in Algorithm 4 with a complete explanation of the different steps.

#### Random Network Distillation

To measure if a state is OOD, RND-DAgger relies on the Random Network Distillation technique, which is a classical approach developed initially for the problem of exploration in Reinforcement Learning to detect new states to explore (Burda et al., ). The principle of RND is to use a randomly initialized neural network as a fixed target $f_{targ}$ and train a second neural network $f_{pred}$ (the predictor) to approximate the output of the target network. As the predictor network improves over time, the error between the predictor and the target network decreases for familiar states (in-distribution), but remains high for unseen or out-of-distribution states. This prediction error thus serves as a measure of novelty, allowing the agent to recognize when it has encountered a new or unfamiliar state. In RND-DAgger, the measure is used to decide when to trigger the expert, and when to trigger back the current policy, alternating phases where the expert is controlling the agent and when the agent is controlled by the current policy. Note that, contrarily to the described baselines, this measure does not involve the expert action, thus avoiding the need to have an expert acting at each timestep.

#### Minimal demonstration time

Just relying on the OOD measure can lead to a behavior where the expert is asked too often for short demonstrations (see appendix Figure 13(d)). To account for this expert burden, we introduce the notion of minimal expert time. It is defined as $W$ the number of consecutive "in-distribution" frames to switch control back from the expert to the autonomous policy (e.g. then number of frames where $m_{t}<\|f_{targ}(s_{t})-f_{pred}(s_{t})\|^{2}$). This ensures that the expert will provide a minimal of information at each intervention, thus demonstrating fewer sequences but of longer length.

## 4 Experiments

The primary objectives of our experimental evaluation are twofold: first, to assess the efficiency of our approach in discovering an effective policy, and second, to evaluate its ability to reduce the expert’s burden.

#### Environments:

Our first environment is HalfCheetah which is a classical reinforcement learning environment(^1^11https://github.com/araffin/pybullet_envs_gymnasium) where the objective is to learn a running strategy for the agent. The goal of the agent is to locomote as fast as possible. The HalfCheetah is controlled by applying motor torques, and the agent manipulates a 6-degree-of-freedom (6-DOF) motor joint vector. The observation space consists of 18 values, which include the position and velocity of the agent’s body, the angles and angular velocities of its six joints, as well as the angle of the center of mass. We also propose two new environments developed for video game research(^2^22The code of these environments that have been made using the Godot engine will be released upon acceptance). RaceCar (see Figure 5) features a physics-based car controller that must complete a single lap on a given track. After each lap, the car is reset to a random position on the starting line. The track presents several challenges, including speed bumps, a ramp in front of a pillar, and sharp 90-degree turns, making the optimal driving behavior non-trivial. Additionally, crossing the ramp and red sloped walls provides a speed boost, adding complexity to the strategy. The car is controlled using four discrete actions: forward, backward, left, and right. The observation space consists of 22 dimensions, including the car’s 3D position, linear velocity (3D), angular velocity (3D), rotation (encoded as the cosine and sine of the angle), and data from seven raycasts to detect obstacles. At last, the 3D Maze environment allows us to study our strategy in goal-conditioned navigation scenarios. A classical human-like character is spawned at a random location within the maze and tasked with reaching a randomly assigned goal. The agent is controlled using four movement actions: walk forward, walk backward, strafe left, and strafe right, as well as a one-dimensional rotation action to change its facing direction.The observation space consists of the agent’s absolute position (3D), the goal’s position (3D), and 131 raycasts, which provide a low-resolution depth map of the surroundings. Since the objective in this environment is to navigate to any goal location, we replace the traditional Behavioral Cloning (BC) approach with a Goal-Conditioned Behavioral Cloning (GCBC) approach (Ding et al., ) as the base learning algorithm. This adaptation ensures that the agent learns not just to imitate expert trajectories, but also to generalize its navigation strategies based on varying goal positions within the maze.

Figure: (a) RaceCar
Refer to caption: extracted/5975707/figures/maps/racecar2.png

#### Baselines:

We compare our approach RND-DAgger to multiple approaches from the active imitation learning literature: DAgger Ross et al. (), a classical approach which queries the supervisor for an action in every state that the learner visits. Ensemble-DAgger Menda et al. (), which propose an automated approach to measure OOD and trigger expert demonstration. Ensemble-DAgger relies on two metrics: A disagreement measure defined as the correlation between an ensemble of policies, and a discrepancy measure computing the difference between the bot action and the expert action at every step. Lazy-DAgger uses the same discrepancy measure as Ensemble-DAgger and reduce bot-expert context switching by using an additional threshold parameter to define a hysteresis band. Note that, we rely on a simplified version proposed in (Hoque et al., ) that has been shown to be more efficient and generates less context switches than the original version. This variant discards the classifier for the benefit of the true discrepancy measure. Human-Gated DAgger (HG-DAgger)(Kelly et al., ) involves a human supervisor monitoring the agent’s behavior in real-time and deciding when to take control. While similar in essence to our proposed RND-DAgger, HG-DAgger relies entirely on human judgment to identify when intervention is necessary, rather than using an automatic OOD measure. This requires the expert to continuously observe the agent’s actions at every timestep, leading to a higher cognitive load and less efficient use of expert time. Finally, we also consider Behavioral Cloning (BC) as the classical baseline, which does not include any active learning mechanism.

The set of hyperparameters used for the different approaches is described in Appendix A. For each method, a first training set is collected by the expert, without active imitation learning techniques. Then the different algorithms are executed to collect additional examples and to update the policy.

#### Metrics:

To assess the performance of the methods considered, we rely on several key metrics: i) The Task Performance measures whether the learned agent is successfully solving the environment. For the Race Car and Maze environments, we use the success rate (i.e., whether the agent completes a lap or reaches the goal). For HalfCheetah, we track the cumulative episode reward, which reflects the overall effectiveness of the agent’s locomotion strategy. ii) The Dataset Size indicates the number of expert actions added to the training dataset, providing insight into how much expert information is being utilized to train the agent. A smaller dataset size suggests the agent is learning efficiently with fewer expert interventions, while a larger dataset implies greater reliance on expert input. iii) The Context Switches which corresponds to the $nswitch$ variable in Algorithms 1 and $\omega$ measures the number of contiguous periods during which the expert is actually controlling the agent. This helps quantify how often the expert is involved in directly providing relevant training samples. For RND-DAgger, this measure accurately reflects the expert’s involvement, as the agent is fully controlled by the policy between switches. The metrics are averaged over 8 seeded runs for each method.

### 4.1 Results

**Table 1: Overall performance: This table illustrates the performance of the different algorithms at the end of the $K$ iterations. It also provides the value of the $nswitch$ variable that accounts for the number of expert interventions.**
|  | Task Performance | # Context Switch |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method<br>Env | RC | HC | Maze | RC | HC | Maze |
| BC | 0.883 | 2455 | 0.367 | - | - | - |
| DAgger | 0.940 | 2343 | 0.450 | - | - | - |
| Lazy-DAgger | 0.939 | 2314 | 0.575 | 20990 | 312 | 1238 |
| Ensemble-DAgger | 0.952 | 2489 | 0.626 | 18391 | 13883 | 2871 |
| RND-DAgger | 0.944 | 2490 | 0.717 | 368 | 708 | 1214 |

#### Oracle-based Performance:

To conduct extensive experiments, we replace the human expert with a predefined oracle policy. This substitution allows us to run multiple trials efficiently without depending on human participants, which would be impractical in terms of both time and resources (experiments using human experts are reported separately later in the section). For the Race Car environment, the oracle is learned through a separate active imitation learning process with a large interaction budget, ensuring the agent has access to high-quality demonstrations. In the 3D Maze environment, the oracle is a NavMesh-based agent, which programatically solves any navigation scenario, making it easy to generate trajectory data. For HalfCheetah, we use an open-source reinforcement learning agent (Kuznetsov et al., ) from the RL Zoo repository (Raffin, ) as the oracle, providing an optimal policy for locomotion tasks. This approach ensures consistency and scalability across experiments.

Table 1 and Figure 6 present the performance metrics across the evaluated environments. To ensure a fair comparison, we report results based on the hyperparameter settings that achieve the highest final performance for each model. Additionally, in the Appendix D, we provide a separate set of results where the hyperparameters are adjusted to align the number of context switches between methods.

Figure: (a) Task performance RaceCar
Refer to caption: x2.png

Our method, RND-DAgger, demonstrates competitive performance relative to Ensemble-DAgger and Lazy-DAgger, indicating that it effectively learns robust policies across different environments. For example, in the HalfCheetah environment, RND-DAgger achieves a cumulative reward of 2490, compared to 2489 for Ensemble-DAgger and 2314 for Lazy-DAgger. These results highlight that RND-DAgger is capable of discovering efficient policies, matching or surpassing the baselines in terms of overall task success. When analyzing performance relative to the size of the training set, RND-DAgger outperforms the other methods in the early stages of the active imitation learning process. This suggests that RND-DAgger is more effective at focusing on critical states where expert guidance is most needed, thereby gathering valuable feedback more efficiently. By concentrating on key interventions, RND-DAgger accelerates policy improvement and requires fewer training samples to achieve comparable or superior results, making it especially advantageous in scenarios with limited expert availability. In summary, the experimental results demonstrate that RND-DAgger not only achieves strong final performance but also exhibits a more sample-efficient learning curve compared to existing methods, validating its capability to optimize expert interventions and rapidly improve policy quality.

When examining the number of context switches between the different methods, RND-DAgger consistently results in significantly fewer context switches compared to Ensemble-DAgger across all environments. This indicates that RND-DAgger is more stable and requires fewer handovers between the expert and the policy, reducing the burden on the expert. When compared to Lazy-DAgger, RND-DAgger shows mixed results. In the RaceCar environment, RND-DAgger generates fewer context switches, highlighting its efficiency in minimizing expert interventions. However, in the HalfCheetah and Maze environments, RND-DAgger produces a similar number of switches. This similarity in HalfCheetah can be attributed to the use of an oracle expert policy, which performs consistently without generating diverse actions. Under these circumstances, the discrepancy measure used by Lazy-DAgger, which relies on comparing expert and policy actions, remains effective for detecting when to switch. In the Maze environment, despite having a comparable number of context switches to Lazy-DAgger, RND-DAgger achieves a higher final performance. This suggests that even with similar intervention frequencies, RND-DAgger is more effective at leveraging expert feedback to improve the policy. Thus, in scenarios like Maze, where achieving high task performance is critical, RND-DAgger is the better choice due to its ability to make more meaningful corrections and drive the agent towards optimal behavior.

Overall, these findings indicate that RND-DAgger strikes a favorable balance between reducing context switches and achieving strong policy performance, making it a more efficient and reliable choice.

#### Qualitative study:

To gain a deeper understanding of when our algorithm requests expert supervision, Figure 7 visualizes the context switches between the current policy and the expert (additional figures are provided in Appendix B). As seen in the figure, both Ensemble-DAgger and Lazy-DAgger query the expert significantly more frequently, leading to a higher number of context switches compared to RND-DAgger. The visualization also reveals that RND-DAgger identifies and focuses on critical areas of the track that are challenging for the agent during early learning stages. Specifically, it requests expert intervention primarily in the problematic zones, such as the bottom section of the track near the bumpers and the obstacle immediately following the speeding ramp (top-left corner of the track). These are regions where the bot initially struggles, making precise interventions crucial for improving performance. This observation indicates that RND-DAgger not only minimizes the number of expert queries but also targets the most relevant segments of the environment where guidance is essential. By focusing on challenging states rather than spreading expert effort across all states, RND-DAgger makes more strategic use of expert supervision, resulting in fewer yet more impactful interventions.

Figure: (a) RND-DAgger
Refer to caption: extracted/5975707/figures/traces_visu/racecar_rnd.png

#### Human Expert-based performance:

We conducted a series of experiments (Figure 8) using a real human expert instead of an oracle policy. In this setup, collecting expert actions while the agent was being controlled by its own policy (rather than having the expert take direct control) felt unintuitive for the human participants, making it challenging to apply traditional baselines effectively. As such we focused our comparison on HG-DAgger, which allows the human expert to decide when to intervene and take control from the agent.

Our results show that RND-DAgger performs comparably to HG-DAgger, but with a significant advantage: it does not require the human expert to continuously monitor the agent’s behavior. Instead, RND-DAgger automates the decision of when to request expert input based on state novelty, reducing the cognitive load on the expert. Figure 8(b) demonstrates the relationship between policy performance and the actual time the expert spends observing the screen. In RND-DAgger, the agent can run at accelerated speeds without the need for constant supervision, as the expert only needs to intervene at specific moments rather than watching the agent’s entire trajectory.

Figure: (a) Performance
Refer to caption: x8.png

## 5 Conclusion

In this work, we presented RND-DAgger, a novel approach to active imitation learning that efficiently minimizes the need for expert interventions by leveraging a state-based measure derived from Random Network Distillation (RND). Unlike traditional methods that rely on action-based discrepancies to detect when to seek expert guidance, RND-DAgger focuses on identifying out-of-distribution states where the agent is most at risk of making errors. This allows our method to selectively request expert feedback only when it is truly necessary, thereby reducing the number of context switches and optimizing the allocation of expert time. Overall, our method provides a step forward in developing more practical and human-efficient imitation learning algorithms, making it a valuable tool for training autonomous agents in complex environments. Future work will explore how to extend RND-DAgger to more challenging tasks and incorporate different forms of expert feedback to further improve learning efficiency and applicability.

## 6 Reproducibility Statement

To ensure the reproducibility of our work, we provide detailed pseudo-code in section 2 and section 3. The code, along with necessary dependencies, will be made available to reviewers for evaluation. Upon acceptance, we will release a comprehensive open-source codebase, including all environments, datasets, oracle model checkpoints, active learning algorithms, and a detailed guide on how to reproduce our experiments and results.

## Appendix A Experimental Details

Dataset Figure 9(a) and 9(b) showcase examples of generated trajectories for the RaceCar and Maze environments.

Figure: (a) RaceCar dataset examples
Refer to caption: x10.png

#### Hyperparameters

For each decision rule, several key hyperparameters had to be tuned:

- •
DAgger
–
The probability $\beta$ of a frame to be controlled by the bot. The probability is decreased at each DAgger epoch by $\beta_{i}\leftarrow\beta^{i-1}_{0}$
- •
RND-DAgger
–
Threhsold $\lambda$ of OOD detection
–
The historic context length, that is the number of frames in the past we take along with the current state to detect its OOD nature
–
The stability window $W$
–
The size of the random network, represented by the number of layers and neurons per layer the predictor and target networks have.
- •
Ensemble-DAgger
–
Threhsold $\tau$ for discrepency measure
–
Threhsold $\chi$ for doubt measure
–
The number of models $N$
- •
Lazy-DAgger
–
Threhsold $\beta_{H}$ for discrepency measure
–
Threshold $\beta_{R}$ for the backward controlled loop (so the criterion to switch back from expert to autonomous)

#### Computing thresholds

For all the methods that necessitate at least one threshold (RND-DAgger, Lazy-DAgger, Ensemble-DAgger), we computed them following the following methodology: the threshold was set to be a positive factor of the mean measure on the training set. In other words, before each new sessions, the measure was ran over the training dataset, and the threshold ($\chi$ and $\tau$ for Ensemble-DAgger, $\beta_{H}$ for Lazy-DAgger and $\lambda$ for RND-DAgger) were calculated as :

$$ $\lambda\leftarrow\overline{\textsc{Measure}(\{(a_{t},s_{t})\in D_{train}\})}\times L$ (1) $$

With L being a positive factor, a hyperparameter to be tuned. For the second threshold of Lazy-DAgger, we set it to be a factor of the first one: $\beta_{R}=\beta_{H}/L^{*}$, with $L^{*}$ being the hyperparameter tuned. That’s a different method compared to other methods, such as Zhang & Cho () or Hoque et al. () who set their threshold so that approximately 20% of the initial dataset is unsafe, or Ensemble-DAgger Menda et al. () that directly grid search the value. The Table 2(d) summarizes the values used for the grid search.

**Table 2: Hyperparameter search**
| Hyperparameter | Values | RaceCar | HalfCheetah | Maze |
| --- | --- | --- | --- | --- |
| $\chi\text{ factor}$ | $[0,1,1.5,2,3,4]$ | 1.5 | 0 | 0.5 |
| $\tau\text{ factor}$ | $[0,1,1.5,2,3,4]$ | 1.5 | 1.5 | 3 |
| N | $[2,3,5]$ | 5 | 5 | 3 |

## Appendix B Interactive session visualizations

#### Typical interactive situations

In Figure 10, one can understand better how, in the RaceCar environment, an oracle (and by extend a human taken as an expert) would interact with the learner. In (a), the current learned policy has control of the car, and the OOD measure (green circle) is below the threshold: the learner is confident. In (b), the car crashes into a wall, so the measure (red circle) queries the expert to take control (transparent red capsule). From (c) to (e), the expert demonstrates how to get back on track, until (f) the measure falls below the threshold again (green circle) for at least $W$ consecutive frames (see Section 3 for further explanations)

Figure: (a) 0
Refer to caption: extracted/5975707/figures/interactive_sesion/minus_1.png

#### Qualitative study

On Figure 11, we reported the context switches and full trajectories on both the RaceCar and Maze environments. The results are taken from the same session and the same seed to better compare the figures. From (a) to (b) we can clearly see that Ensemble-DAgger got the most context switches, compared to Lazy-DAgger and RND-DAgger that have a comparable amounts, but we can see that the ones from Lazy-DAgger are more concentrated in specific area whereas is RND-DAgger they seem more spread out. Then, from (d) to (f) we can see the corresponding episodes with expert demonstrations (in red) added to the dataset during that session. We can see that the expert demonstrations are either way longer (the whole episode) or way shorter (only a few frames) for Lazy-DAgger compared to our method, providing more consistency of the demonstrations throughout the map. The difference is clearer on the RaceCar environment. Indeed, both on the context switch figures, we can see that Lazy-DAgger and Ensemble-DAgger are comparable, and that the queries for RND-DAgger are concentrated around only two zones that generated most of the failure cases of the bot.

Figure: (a) RND-DAgger
Refer to caption: extracted/5975707/figures/traces_visu/navbench_CS_rnd.png

## Appendix C Ablation Study

In this ablation study, we evaluate RND-DAgger, focusing on the additive advantages of three key enhancements: a stability window of size $W$, historical context, and the architecture of the random networks (see Section 2(d) for insights on the values tested). The stability window should increase the quality of the frames by letting the expert finish its demonstration and making sure the learner is fully recovered before letting it the control again. The historical context enriches decision-making by incorporating temporal information from past actions and concatenating them to the current state. This particularly helps in tasks where there is a strong relation between the states. For example, a state where the car is close to a wall is not necessarily a bad state, unless the car is going towards it for several frames in a row. However, a historical context chosen too big is detrimental, because it can overshadow useful dimensions of the sate, and thus preventing the predictor network to seize useful information.

Results are reported in Table 3 and Figure 12. We note that the stability window $W$ helps reducing the context switching, without impacting that much the performance at the end of the sessions of RND-DAgger. In other words, it helps decreasing the expert burden, without impacting the task performance.

Figure: (a) HalfCheetah (Task perf)
Refer to caption: x12.png

**Table 3: Performance and context switches at the end of the sessions. We report the results of the key ablations on RND-DAgger, along with the results of Lazy-DAgger for further comparisons**
|  | Task Performance | # Context Switch |
| --- | --- | --- |
| BC | 2455 | - |
| RND-DAgger | 2490 | 708 |
| RND-DAgger - stab. window | 2432 | 1985 |
| RND-DAgger - small | 2167 | 1136 |
| RND-DAgger - historic | 2172 | 554 |
| Lazy-DAgger | 2314 | 312 |

#### Stability window focus

On the Figure 13 we report the context switches and session trajectories on the RaceCar environment, at the second DAgger epoch of a given seed with and without a stability window. We can clearly see that the number of context switches increases and a third critical zone appeared at the top right of the map. That is due to the fact that the bot is fairly confident in the top straight line of the course, but the last turn is difficult to manage when the velocity of the car is too high, inducing a query to the expert.There are also more context switches at the entry of that straight line, because the learner has to be precise to pass without any troubles with the walls or the speed ramps on the sides. We conclude that the stability window increases the quality of the demonstrations without impacting the quality of the measure, that still grasps the relevant critical areas of the track.

Figure: (a) $W=30$
Refer to caption: extracted/5975707/figures/traces_visu/racecar_rnd_cs_withstab.png

## Appendix D Additional comparisons

In this section we provide additional comparisons on some experiments obtained during the grid search of our hyperparameters. In Figures 14(a) and 14(b) we aligned the experiments that had comparable context switches throughout the sessions. We see that RND-DAgger clearly outperformed LazyDAgger. On Figures 14(c) and 14(d), we reported the experiments aligned with the best context switches. Again, RND-DAgger outperforms LazyDAgger on the early sessions, while reducing the number of context switches.

Figure: (a) Perfs
Refer to caption: x16.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: I wonder are there other comparisons here? Do they all have to be dagger variants? Is there another type of “expert takeover” method that you could compare to? If I missed this in the text I apologize.
soundness: 3
strengths: The research problem this work focuses on is good. Increasing the efficiency of imitation learning is an open problem and the most effective way to utilize expert data within IL. Improving the interaction between expert and policy increases the ability of IL to be used with real world problems.

The contribution of this work is ok, they propose their method and perform a study to verify their claims.

The algorithm is clearly defined and could be implemented from the information given.

The experiments run are reasonable and seem to demonstrate their method well.

The baseline comparison are all dagger variants (and BC). They are definitely reasonable comparisons.

The empirical results are good. They show a balance between expert interventions and performance which is the goal of this work.

The clarity is great. This paper was very easy to read and very clear.
summary: This work presents a method to more effectively integrate human feedback with imitation learning and specifically the IL method dagger. The idea is to only request human feedback when the policy is out of distribution. The data collected by the expert is then added to a buffer for training. In doing so, the algorithm is able to more effectively utilize the experts time and feedback.
weaknesses: The novelty of this work is minimal. As far as I understand it the methods generally used in this work are all previously known. Dagger and the OOD classifier seem to be like the main components used but are previous work.

The statistical rigor needs improvement. The metrics are only averaged over 8 seeds. Is there a reason for only this many? I feel like it should be many more.

As well, I want to see confidence intervals on Table 1.

There is no failure analysis. I wish there was one on the times the method does not perform better than the previous methods. What about those tasks makes it not as good? 

The future work and conclusion is ok. I wish there was a better future work section. I think a limitation of this work is that it works well in simulated environments but on an actual self-driving car how would this work? The car couldn’t just stop running. This could be an interesting direction of future work some like predicting beforehand that you’re going to get an out of distribution state and request input.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: They are mentioned in Weaknesses.
soundness: 3
strengths: - The paper presents a well-founded motivation, aiming to optimize the timing of expert interventions to reduce overall costs associated with human expertise and minimize the frequency of transitions between human experts and learning agent.
- The paper is straightforward, particularly for readers with a background in the domain of unsupervised RL.
summary: This paper presents Random Network Distillation DAgger (RND-DAgger), an active imitation learning method that leverages RND to define an out-of-distribution states, enabling selective expert intervention and minimize the frequency of transitions between human experts and learning agent through a minimal demonstration time mechanism. The approach is evaluated across Race Car, Maze, and Half Cheetah environments.
weaknesses: - **Novelty:** The paper's novelty appears constrained, as it predominantly builds upon an established novelty measure within the unsupervised RL domain. This integration approach mirrors Ensemble-Dagger, which relies on a principle similar to Disagreement in the unsupervised RL domain.

- **Limitations:** The limitations specific to this method are not sufficiently addressed. Certain limitations noted by the authors for other approaches may also apply here. For instance, the paper states, “While this approach works well when the expert is optimal and acts deterministically, it becomes problematic when dealing with humans or imperfect experts.” It is unclear how this method manages these challenges, and if not addressed, this issue should be explicitly acknowledged.

- **Baselines:** The study lacks comparisons with more recent methods that similarly leverage human input for out-of-distribution states, such as RLIF [1], PATO [2], and Sirius [3].

- **Time Mechanism:** This approach is reliant on two factors: RND and Minimal Expert Time. It would be valuable to assess how Minimal Expert Time operates alongside other baseline methods - a potentially straightforward inclusion. This would provide a clearer understanding of RND’s advantages over alternative metrics.

- **Experiments:** Additional experiments are needed to support the authors' claims regarding task performance and context switching. From the current results (Table 1), Ensemble-Dagger performs better in RC settings but with higher context switches, whereas Lazy Dagger shows the opposite trend in HC settings. Further experiments in more challenging environments, such as Adroit [4], would provide deeper insights and strenghten the paper’s claims. Moreover, incorporating the Time Mechanism into other methods may enhance their context-switching capabilities, offering a compelling comparison.

- **Ablation Study:** The ablation study section would benefit from greater detail, as the current presentation makes it challenging to interpret the impact of various factors on learning outcomes.

**References**

[1] Luo, Jianlan, et al. "RLIF: Interactive Imitation Learning as Reinforcement Learning."arXiv preprint arXiv:2311.12996 (2023).

[2] Dass, Shivin, et al. "Pato: Policy assisted teleoperation for scalable robot data collection."arXiv preprint arXiv:2212.04708 (2022).

[3] Liu, Huihan, et al. "Model-based runtime monitoring with interactive imitation learning."2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.

[4] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Experiments:
	- In Figure 6, do the algorithms use same initial datasets? Does increasing in x-axis mean starting at a larger dataset? Or this corresponds to increasing size due to the conditions to query from the expert?
	- How do Figure 6e and 6f demonstrate that RND-DAgger is better in "sample-efficiency"? Context switching does not totally correspond to number of samples queries from the expert. Can the paper clarify this point?
	- On page 9, there is a claim "..., RND-DAgger is more effective at leveraging expert feedback to improve the policy." How is this statement true exactly? The policy training at the end is the same but I suppose the aggregated data is somewhat different?
soundness: 3
strengths: - The idea is very simple and focuses on a gap that previous approaches do not consider (i.e. OOD states rather than action mismatch).
- The algorithm appears to be practical compared to existing approaches.
summary: The paper focuses on addressing the covariate-shift problem in behavioural cloning, where the agent suffers from compounding error from the predicted actions. The paper considers DAgger-like algorithms where the process includes querying an expert to add extra demonstrations, thereby increasing state-action coverage. The paper identifies a gap where existing approaches require the said expert to be present for a long period of time, which can be costly for some applications. The paper thus propose a method that considers out-of-distribution in states through random network distillation. The paper conducts experiments on three simulated environments, demonstrating that the proposed algorithm achieves similar performance as existing DAgger-like baselines, as well as the reduced number of expert queries.
weaknesses: I am happy to increase the scores if these comments are addressed.

**Comments**
- I am unsure why the formulation is POMDP rather than MDP. The environments used in this paper appears to be using state-based information, in the sense that I cannot really guarantee that they are partially-observable, as opposed to using images that will be way more convincing.
	- I further believe that the current method assumes the observation aliasing is not a problem. Self-driving environment can dramatically different scene while similar actions.
	- I suggest including the CARLA environment to make this result more convincing.
- Regarding the approach, experimentally how is this algorithm different from the earlier variants since $f_{targ}$ can be $\pi_{exp}$? In other words, is LazyDAgger essentially doing RND but have a slightly higher-dimensional output space? If that is the case, then is the benefit coming from the additional "minimal demonstration time" mechanism?
- The paper can improve upon its writing quality:
	- Algorithms 3 and 4: Include the actual definition of *measure* for clarity.
	- Table 1: What do bolded and underlined terms mean? Is this taking the average or some other statistics? What about the standard error or other statistics?
	- Figure 6: What do the solid line and shaded area correspond to?
		- Also recommend the top curve to use different line style to differentiate variants easier.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. The paper lacks necessary comparisons with methods specifically designed to address out-of-distribution scenarios, aiming to generalize to states unobserved during training without requiring expert intervention [1, 2]. Including these comparisons is essential to demonstrate the completeness of the proposed method in handling such challenges and I am willing to raise the scores.

2. While the paper presents results from four environments, the setup may lack sufficient diversity to fully assess RND-DAgger’s adaptability. Would the authors consider evaluating on additional environments, such as robot manipulation or hand dexterity tasks, which could reveal the method’s robustness and effectiveness across a broader range of complex control scenarios?

3. In which types of tasks or environments does RND-DAgger face challenges, especially with regard to state-based out-of-distribution detection?

4. How does the RND mechanism manage computational efficiency, and is it feasible to optimize it further for large-scale applications?

5. What role does the "minimal demonstration time" play in balancing expert workload and agent training quality, especially in complex environments?

[1]  Shang-Fu Chen, Hsiang-Chun Wang, Ming-Hao Hsu, Chun-Mao Lai, and Shao-Hua Sun. Diffusion model-augmented behavioral cloning. In International Conference on Machine Learning, 2024.

[2] Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robotic Learning, 2022.
soundness: 2
strengths: 1. **Motivation and Intuition**: The motivation for reducing expert intervention in active imitation learning is convincing, particularly in applications where continuous expert availability is costly or impractical.

2. **Novelty**: The paper introduces the idea of leveraging Random Network Distillation (RND) to manage expert interventions, which is innovative and well-suited for identifying critical OOD states.

3. **Clarity**: The paper is well-organized, with clear explanations of both theoretical foundations and implementation details. Figure 4 effectively illustrates the RND-DAgger algorithm, and ablation studies clarify the effect of each component.

4. **Experimental Results**: The visualizations of experimental results are clear. Figure 6, for instance, provides an accessible overview of task performance and the frequency of context switches across environments, highlighting RND-DAgger’s reduced reliance on expert intervention.

5. **Reproducibility**: The author's inclusion of pseudo-code, a detailed description of the experimental setup, and an appendix for hyperparameters strengthens reproducibility. They promise to release code upon acceptance also supports this.
summary: The authors address efficient expert intervention in active imitation learning for complex, reward-free tasks like video game bots and robotic control. The proposed RND-DAgger framework uses a state-based out-of-distribution (OOD) measure to trigger expert input selectively, reducing intervention needs by focusing on critical states. Experiments in RaceCar, 3D Maze, and HalfCheetah environments demonstrate that RND-DAgger outperforms traditional methods, achieving high performance with fewer expert queries. Ablation studies highlight the importance of the stability window and historical context in reducing context switches and expert burden.
weaknesses: 1. **Experiment Setup**: The environments in the main experiment, although tested on four different setups, lack sufficient diversity, limiting the evaluation of RND-DAgger’s performance across a broader set of tasks. Incorporating additional environments, such as robot manipulation or hand dexterity tasks, would enrich the setup and test the method’s adaptability in more complex, fine-grained control scenarios.

2. **Related work**: The paper needs to include comparisons to the methods that specifically designed to handle out-of-distribution scenarios, aiming to generalize to states unobserved during training without requiring expert intervention [1, 2]. 

[1]  Shang-Fu Chen, Hsiang-Chun Wang, Ming-Hao Hsu, Chun-Mao Lai, and Shao-Hua Sun. Diffusion model-augmented behavioral cloning. In International Conference on Machine Learning, 2024.

[2] Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robotic Learning, 2022.

Meta-review: (a) Summary: This paper presents Random Network Distillation DAgger (RND-DAgger), an active imitation learning method that leverages RND to define out-of-distribution states, enabling selective expert intervention and minimizing the frequency of transitions between human experts and learning agents through a minimal demonstration time mechanism. 
(b) Strengths: The paper is generally well-written and easy to follow. The proposed approach seems interesting and reasonable. The experimental results seem to support the authors' claims.
(c) Weaknesses: The reviewers pointed out a few major concerns and questions. The novelty is somewhat incremental. Some comparisons with baselines are missing.
(d) The authors have addressed the majority of the concerns from reviewers. All reviewers gave a final rating of above borderline.

Decision: Accept (Poster)