Impossibly Good Experts and How to Follow Them
Authors: Aaron Walsman
Muru Zhang
Sanjiban Choudhury
Dieter Fox
Ali Farhadi
Venue: ICLR 2023 poster (2023)
Keywords: Imitation Learning
Reinforcement Learning
Experts
Distillation

Abstract: We consider the sequential decision making problem of learning from an expert that has access to more information than the learner.  For many problems this extra information will enable the expert to achieve greater long term reward than any policy without this privileged information access.  We call these experts ``Impossibly Good'' because no learning algorithm will be able to reproduce their behavior.  However, in these settings it is reasonable to attempt to recover the best policy possible given the agent's restricted access to information.  We provide a set of necessary criteria on the expert that will allow a learner to recover the optimal policy in the reduced information space from the expert's advice alone.  We also provide a new approach called Elf Distillation (Explorer Learning from Follower) that can be used in cases where these criteria are not met and environmental rewards must be taken into account.  We show that this algorithm performs better than a variety of strong baselines on a challenging suite of Minigrid and Vizdoom environments.

Published as a conference paper at ICLR 2023
IMPOSSIBLY GOOD EXPERTS AND How TO FOLLOW THEM
Aaron Walsman', , Muru Zhang', Sanjiban Choudhury?, Ali Farhadi', Dieter Fox!- Computer Science and Engineering, University of Washington' Computer Science, Cornell University? NVIDIA
{awa lawalsman, sman, nanamil7, ali, fox waashssswaahtnoonnnoooo washington.edi edu san bann]aaenecoreell edu
ABSTRACT
We consider the sequential decision making problem of learning from an expert that has access to more information than the learner. For many problems this extra information will enable the expert to achieve greater long term reward than any policy without this privileged information access. We call these experts "Impossi- bly Good" because no learning algorithm will be able reproduce their behavior. However, in these settings it is reasonable to attempt to recover the best policy pos- sible given the agent's 's restricted access to information. We provide set of neces- sary criteria on the expert that will allow learner to recover the optimal policy in the reduced information space from the expert's advice alone. We also provide a new approach called ELF Distillation (Explorer Learning from Follower) that can be used in cases where these criteria are not met and environmental rewards must be taken into account. We show that this algorithm performs better than variety strong baselines on a challenging suite of Minigrid and Vizdoom environments.
INTRODUCTION
Sequential decision making is one of the most important problems in modern machine learning theory and practice. Reinforcement learning from an environmental reward signal is powerful but unwieldy tool for attacking these problems. In contrast, imitation learning can be much more sample efficient and empirically easier to train than reinforcement learning, but requires powerful expert that can either provide an offline dataset of instructions or online supervision. settings, these experts have access to more information than the learning agent. This can occur when using human demonstrators train robots that have inferior sensors, or simulated environments where a synthetic expert uses hidden simulator information to train an agent. In these settings, it possible that the expert's additional information makes it more powerful than any learning agent that does not have access to the hidden information. We call these experts "Impossibly Good" and show that learning from them using techniques that do not incorporate environmental rewards can cause the agent to drastically under perform the optimal policy in the reduced information space.
many practical
For example, consider a simulated robot tasked with retrieving a cell phone in an unknown apartment consisting of multiple rooms. The robot observes the world using camera and is not given the location of the phone in advance, so it must explore each room in order to find it. Because this is a simulated environment, we can construct but also the exact layout of each room and can compute the shortest path from the robot to the phone. We can then use this expert to construct large corpus of training data across any number of apartments and phone locations. While these demonstrations may be optimal according to the expert that knows the phone's 's location, they crucially do not provide any demonstrations of the exploratory behavior that is necessary for the robot which must rely its more limited sensors. At test time, the robot may need to explore many empty rooms before finding the one that contains the phone, but the expert has always walked directly to the goal and so it has never shown the robot what to do when encountering an empty room. In this case the expert is impossibly good because on average, it can reach the phone much faster than any agent that does not have access to the map, but must explore each room by one. While may be able learn important skills from this expert,
expert that knows not only the location of the phone,

---

Published as a conference paper at ICLR 2023
crucially missing demonstrations of other necessary behavior, and so learning from this expert's advice alone may cause the robot to fail.
Our goal in these settings is to find an algorithm that retains the efficiency of imitation learning, while incorporating just enough reward feedback from the environment to achieve success. To address this, we introduce a new technique called ELF Distillation (Explorer Learning from Follower). The key insight of this approach is to train one follower policy using the advice of the impossibly good expert alone, and then use the estimated long-term value of this policy to drive exploration of a second explorer policy using reward shaping. These two policies are trained jointly so that the explorer policy can be used to inform the distribution of states from which the follower must learn.
In order to study these problems, we have constructed a suite of Minigrid(Chevalier Boiiee et al.) 2018) and Vizdoom Vizdoom/Wydmuch Wydmuch et al., (019) environments that clearly demonstrate the challenges of learning from impossibly good experts. While these are toy problems, they are quite challenging for many strong baselines and related approaches, and allow us to clearly demonstrate the necessary concepts in a setting that avoids confounding implementation details. Code for these experiments can be found at Mtltttps///githii com/ aawonwa sman wwa.sssmannnaaaaa wwwpaa//mmosssssyyooooooo
2 RELATED WORK
Recently many authors have identified the problem of learning from experts that have access to more information than the learning agent. This can occur in the self-driving domain, where a human expert may be able to see more than the car's perception system (de Haan et al. 2019 Bansal al.) 2018 Chen et al. 2019). It can also occur robot exploration, where the expert may already know map of the world (Choudhury expert may know which graph edges are valid (Bhardwaj et al. al.] 2017). In some cases it is possible to overcome these issues by considering history of recent observations instead a single step, but unfortunately this exacerbates the "latching" behavior identified by several practitioners in both self-driving (Muller et al., 2006 Kuefler et al. 2017 Bansal et al. 2018 Codevilla et al. 2019) and natural language processing (Ortega et al.
al. 2017 Jain et al.] 202T), in heuristic search where the
2021 which an agent becomes overly fixated on
repeating recent actions.
In these partial information settings, ing converges to the QMDP approximation of the expert's s policy (Littman et al. 1995). This likely explains the empirical success (Chen et al.] 2019: [ee et al., (020) of such techniques in settings where information naturally reveals itself over time. These interactive imitation learning techniques were originally designed to address covariate shift, a condition in which compounding single-step errors drive the learning agent into states not seen during test time. This effect was first noted by (Pomerleau] 1989) and has long been recognized as one of the most important challenges in imitation learning (Bagnell, 2015). Several approaches have been proposed to address this such as SEARN Daumé et al. (2009). DAgger (Ross et al.] 201T) and aggreVaTTè (Ross & Bagnelll 2014 Sun et 2017). Recent work (Spencer et al. 2222] has shown that covariate shift can be broken into realizable settings where off-policy methods such as behavior cloning work well with increas- ing data, and non non-realizable settings where on-policy methods with an interactive expert (Ross et al., 2010) or an interactive simulator Zicbart et al. 2008 Swamy et al.
settings,Choudhury Choudhury al. (018)) showed that interactive imitation learn-
2021) are necessary.
Some authors (Zhang et al., 2020 Kumor et al. 2021 Ortega et al. 2021 have proposed to address learning in partial in information settings using the causal reasoning Iramework of Pearl et al. (2016), while Swamy et al. (2022) has recently shown that on-policy imitation learning methods can more effective at recovering the expert's behavior than off-policy approaches in these situations, and has provided conditions under which an agent can asymptotically recover the expert's behavior.
However, in settings where information does not reveal itself, the learner has to actively gather information (Lee et al. (021). TTeneeholtz et al. (2021) examine similar settings but give the learner access to the confounder at test time. |Wariinttnn et al. (202T) proposes an asymmetric DAgger algorithm for this setting, but it requires a differrntiablee model of the expert, which is fre- quently unavailable. Nguyen et al. (2022) replaces the entropy term in Soft Actor Critic (Haarnoja et al. 2018) with a divergence between the agent and expert policies at each visited state. Weehh et al. 2021) interpolates between the policy gradient and an imitation learning signal using an estimate of how well the agent is able to follow the expert in each state. Our work builds on these ideas by encouraging the agent to visit states where following the expert leads to long term success rather than short-term ability to mimic the expert. This technique uses the tools of pol-
2

---

Published as a conference paper at ICLR 2023
icy distillation (Rusu et al. 2015 Czarnecki al., (0199) where an agent may learn from mix of single-step imitation and multi-step reward signals.
Imitation learning from impossibly good experts requires exploratory behavior to find access to hidden information. Deciding how to conduct effective exploration in unknown environments is problem with a rich history (Kao et al., 1996] Allbrs & Henzinger, [997] Yang et al.] [021]. Our approach is compatible with a wide range of exploration techniques. but we use the exploration inherent to training stochastic policies with (PhhSuaul et al. (2017) in this paper for simplicity.
IMPOSSIBLY GOOD EXPERTS
In order to study learning from an expert with more information than the learning agent. we use the framework of Partially Observable Markov Decision Processes (POMDPs), (Kaelbling et al. 1998), a generalization of Markov Decision Processes (MDPs) (Sutton & Barto 2018) to situations where an agent must make decisions using limited observations. The goal in these settings is to sequentially make decisions discrete time based on feedback from the environment in order to maximize reward signal. As in an MDP, a POMDP contains an underlying state space S and action space A. In an MDP, the agent receives the state s from the environment each time step. but in a POMDP, the learning agent instead receives some possibly lossy and non-Markovian observation o that is a function of the state the agent must reason over the history of past observations, actions and rewards up until the current time step T; {(11.a1:71) T1) ... (o.-1, (i-1, } referred to as trajectory.
(ss). Because the observations themselves are non-Markovian,
A learning agent's policy _e is a differentiable function mapping trajectory to a normalized distri- bution over actions. We define (T( as the agent's 's action distribution after witnessing the history We also define the model class II, be the set of all possible learnable policies. During training we assume access to an expert policy Te which is a non-di differentiable function that also maps a trajectory to normalized distribution over actions.
In order to reason about agents and experts that have different information, we consider a POMDP with two separate observation functions. The first og = Os(s) produces an observation that the agent sees, while the second og = Og(s) produces an observation that the expert sees. This allows us to reason about a trajectory in the underlying state space т and map it to a corresponding trajectory of observations for the agent Ta and separate trajectory of observations for the expert Tg.
TS = [(.1,, (1, (SN, aN, rx)) Jc ((0c(s)).,,.)) (O Eg = (000(s)),)).7)))))) (..
The observational trajectory Tg represents what the learning agent observes while interacting with the environment, while the state trajectory Ts represents the ground truth state of the world which is unknown. For notedeteriiiistic observation functions we use Oc(ogls) and Oc(Te(rs) to repre- sent the probability of a state or state space trajectory Ts producing an observation og or observa- tion trajectory TC.
We assume a model-free setting in which the states, observation functions, transition dynamics and reward dynamics are all unknown the learning algorithm. Instead during training, the algorithm interacts with the environment, and receives observations and a scalar reward the form (oc, og, r} with the assumption that og and og were generated from the same unknown state s. The learning algorithm also has access to an expert ag that can produce advice in the form of suggested action ag ~ (T()). ). At test time the agent will not receive the expert's 's observations og or have access to the expert's advice dg, and so it must learn to make decisions using trajectories of og alone.
We consider the episodic setting with a set of terminal states fermm When terminal state is reached, the environment informs the agent that the episode has ended using special observation ferm and resets using an unknown initial state distribution Srint, We write TC TE to refer to trajectory generated by initializing according to Sänit and repeatedly sampling actions from σæ until a terminal state is reached. We also write P (oc) and Pr.. (Tc) as the probability of observing Og and To. respectively when acting according to Tg from the initial state distribution. We will also use Px. to refer to the set of trajectories that have nonzero probability when acting according to Tे. from the initial state distribution.
The return of a trajectory G(T) = E/r T; is the sum of all rewards encountered during that trajec- Gi=1 tory. The value function of a policy Vn (s) Eyy la G(T) is the expected value of the return of
3

---

Published as a conference paper at ICLR 2023
trajectory that starts in state and continues by sampling actions according to Tg. We write Vinit to be ginii Vig (8), the expected value when acting according to TT after starting from states sam- pled from the initial state distribution. The goal of the learning algorithm to learn a policy (<(Te) using the reward signal and expert advice dg during training in order to maximize Vinit during test time. An agent-optimal policy =cc is one that achieves the greatest value when starting from the initial state distribution compared to all other policies in the model class Vinit 2 Vinit V Tg E IIg.
x2 Note that while this agent-optimal policy is the best policy achievable with limited information, may still underperform the expert Te that has more information.
We now have the tools formally state the impossibly good criterion. We say an expert Te is impossibly good iff'
Vinit
Vinit 22
This means that an expert is impossibly good if in expectation it achieves a greater value from the initial state distribution than the best policy in the model class IIc.
4 How NOT To FOLLOW THEM
In this section we consider learning from an impossibly good expert using Behavior Cloning and DAgger, two common techniques that expert demonstrations learning signal. We first de- scribe these methods, then show examples designed to demonstrate their success and failure modes when learning from impossibly good experts. We then provide formal criteria for determining when these methods are capable of finding the agent-optimal policy 22 from demonstrations provided by Tg. Note that Swamy et al. (2022) have also recently provided conditions under which an agent with restricted information may achieve expert level performance from demonstrations. Rather than sidering cases where expert performance may be achieved, our goal is instead to give conditions for learning the best policy possible in cases where achieving expert performance is impossible. Finally we show that in some environments, the agent-optimal policy T2 cannot be recovered from expert advice alone.
Behavior Cloning is off-policy technique in which the expert Ne generates a static dataset of trajectories prior to training. The algorithm then iteratively samples batches of state-action transi- tions from this dataset and trains the learning agent to match the expert's 's demonstrations, Behavior Cloning known to suffer from covariate shift (Ross et al., 2011). a condition in which errors made by the model during test time can quickly take the agent outside of the distribution of states encoun- tered during training. Despite this limitation, Behavior Cloning enjoys widespread popularity due to its simplicity and ability to perform well on some problems (Spencer et al. (021))
DAgger Ross et al. (201)) is an on-policy method designed to overcome the covariate shift of Be- havior Cloning. Recently Swamy et al.] (2022) has shown that on-policy methods such as DAgger can improve an agent's 's periormance when learning from an expert with more information. Instead of generating static dataset by acting according to the expert TE, DAgger iteratively collects batches of data by acting according to the learning agent Te and simultaneously recording the expert's ad- vice in each visited state. After collecting batch of data, the agent is trained to replicate the expert's 's advice on all data collected so far. This allows the agent to train on states that may not be visited by the expert, but are necessary to recovering from mistakes made by the learner.
Diagrams of the examples in this section are found in Figure I In these environments, the state space, action space and state-action transition function consist of a directed graph of nodes repre- senting the location of the agent. The state space also contains a random variable X that is uniformly sampled from {0, 1} at the start of each episode and remains fixed until termination. The expert ob- serves the value of X at the start of the episode, while the agent only observes X after reaching certain nodes in the graph. Transitions between nodes are deterministic, and both expert and agent observe the node they currently occupy. In all examples there are two terminal nodes, one which produces reward of 1 when X 0 and the other which produces reward of 1 when X = 1. The agent also receives a small reward penalty -c at each step to encourage faster solutions.
Due to the deterministic transition function and the fact that X is fixed for each episode, we can simplify σe to be a function of the current node and the remembered value of X E 10,1,, ?} rather than an entire trajectory of observations. We also assume the expert Tg acts optimally according to its observations.
4

---

Published as a conference paper at ICLR 2023
Example
Example II
Example III
Legend:
Starting
the covosed the agent when visliig this position Teriinal Stre. reward the
AMTO
lilden sampled antiveraly
mooee [0.3] beginning of Shows
ocper
expert
Cipes
epe qpeaeds
Figure 1: Three example environments. An agent may travel through states by taking actions as indicated by the arrows. The reward when reaching a terminal state depends on a hidden variable X that is revealed to the agent if it visits the indicated locations.
First consider Example I. Here, the agent sees the value of as soon as it visits B. This allows the agent-optimal policy #2 to always know which terminal reward to transition agent in this example must learn distributions for observations Tc(A, ?), Tc 0) and (B 1). Because the expert ge takes the same path as the agent-optimal policy, it will generate correct training examples for these distributions. Using Behavior Cloning to match these examples will recover n
from B. An
Example II in Figure I shows case where Behavior Cloning fails. To see this, note that when generating data, the expert will travel from A to B when X = and from A to C when X =1 Under the agent's observation function Oc, these correspond to (A, ?). (B, 0) and (C, 1). Now consider the agent-optimal behavior in this example. Because of its limited information, the agent must take the first step blindly, and so it will sometimes reach either (B, 1) or (C, 0). In these cases, the optimal behavior backtrack to where it will encounter either (A, or (A, 1), remembering the value of X. From these observations, the agent now has enough information to guarantee reaching (B, 0) or (C, 1) and achieve the terminal reward. Note though that data generated by acting according to the expert did not provide demonstrations for what to do when observing (B, 1) or (C,0) so we cannot hope to recover this optimal behavior.
On the other hand, we can show that DAgger can recover 7Ż in this example. Now data is generated by the learning agent Tg, and as we just saw, it has no way to decide whether to transition to B or C at the first step. This means that unlike Behavior Cloning, the observations (B, 1) and (C, 0) will exist in the training set. For these observations, the optimal behavior according to the expert ng is to backtrack to A and continue down the opposite branch so these labels will be provided along with the corresponding observations. These labels are also optimal according to o2. so this extra data will allow DAgger to recover the agent-optimal policy.
Finally Example III in Figure I shows case where DAgger will fail to recover T². Now the problem not only data coverage, but also the instructions received by the expert. Similar to Example II, the expert will always tell the learner to transition to B or C' from the initial observation (A, ?). However, due to the information asymmetry, the agent-optimal policy g/ must first visit D in order to discover the value of X before backtracking to A and continuing to B or C as appropriate. Even if some uncertainty in the agent's 's policy causes it to visit D while generating data, the labels for (A, ?) will never tell the agent that this is the correct behavior.
These examples have given us two intuitive conditions that must be met in order for these methods to recover 7... The first is that all observations that are encountered when acting according to the agent-optimal policy T2 must be encountered during training. The second condition is that the expert We must instruct the agent take correct actions according x²- We now generalize these intuitions into a formal set of conditions that must be met in order for these methods to recover the agent-optimal policy π in simplified learning model.
Consider a dataset D generated by acting according to an arbitrary generation policy TD and sam- pling labels ag from TE. Also consider a learning policy 7e that memorizes the empirical distribu- tion of expert recommendations for each trajectory Tc. in the dataset D.
E.Pl Tzy 1
(2)
Σ
Theorem 1. The empirical policy 7e will recover ni as [D] + 0o iff the following conditions hold:
5

---

Published as a conference paper at ICLR 2023
1. (Coverage) PxD (T2) 00 P.
2. (Correctness) Ergwen Te (e - m2 V E Prt
The proof is in Appendix A
The first condition states that all policy input trajectories that have nonzero probability of being visited under the agent-optimal policy =2 must also have non-zero probability under the dataset sampling policy dD- This corresponds to the intuition developed from the examples above that during training must visit all states that the agent-optimal policy needs to learn about.
The second condition states that the distribution of expert advice for the state trajectories 7s which map to agent observation trajectories 72 must be equal to =2 =²() for all trajectories with nonzero probability of being visited under the agent-optimal policy TT. This corresponds to the intuition developed earlier that an expert must tell the agent to act according to x2 in order to recover T2-
Next show it may not be possible to learn the agent-optimal policy =2c from expert demonstra- tions alone regardless the dataset generation policy TD
Theorem 2. There exist environments with impossibly good experts that violate the correctness condition in Theorem regardless of the dataset distribution policy ap.
Proof. Example III in Figure Lis proof by example.
In this environment (A - that can sample states mapping to that will cause (T())) to produce the labels required to learn 22
?)) 1.0. However, TE A - D(TE) V тe so there is no
This result shows that in some cases it is impossible to learn the agent-optimal policy T2 from the labels generated by impossibly good expert alone, regardless of the rollout policy TD that generates the dataset.
How TO FOLLOW THEM
In the previous section, we have shown that it possible construct an environment with an impos- sibly good expert such that the learner cannot recover the agent-optimal policy when learning from the expert's advice, regardless of the dataset generating distribution æ. Paradoxically, this means we must treat advice from these experts as sub-optimal from the agent's perspective, even though they come from expert that can achieve higher reward than any policy we can learn. In light of this result, is natural to ask if we can improve this situation by incorporating environmental reward into our learning algorithm.
The literature imitation learning (Rajeswaran al.) 2017) and policy distillation (Rusu et al., 2013) provides many useful tools that can be used to learn from sub-optimal experts. We describe these methods using the unifying distillation framework of Czarnecki et al. (2019). In this setting, distillation algorithms consist of two steps. The first step rolls out set of N state-action-rewarr
transitions D (T), d1,71) ... (TN. aN,TN) rx)} using a dataset generation policy TD (go in Czar- neckii al. (2019)) In the second step, the parameters 0 of the learner's policy He are then updated
in proportion to
EAD
Vo Vologac(as R. + "alln(aalti
These steps are repeated until performance converges or some other stopping criterion is met. The update rule in Equation can be thought of as mixing reinforcement learning objective based on a non-differentiable multi-step return RG E A with a single-step differentiable loss function 1. The reward term f.z can be the environmental reward rı, any other term we choose that assigns greater value to some behavior that we want to encourage. The loss / more restrictive because must be differentiable, but provides powerful way inject direct supervision into the training pro- cess. By varying our choice of Tp. F and I we can arrive several different distillation algorithms as shown in T Here the functions H (Tg, T2)x and K LLTE TC); respectively refer to the cross entropy and KI. divergence between the expert and agent for the current observational history т;

---

Published as a conference paper at ICLR 2023
Table 1: Algorithms.
Algorithm
TD
Policy Gradient
WC
Teacher Distill
TE
H(TE,
0
On-Policy Distill
H. ( TC)
On-Policy Distill+R
TC
H(TT,, TC)
Vi
N-Distill +R
TC
H(TT; TL)
-H (T T Tc) +
Expert Matching Reward +R
a ala,=00: 3 fla + ri
ADVISOR
WE
H (TTETTC))
ri w/) ri (T+ UF(
ELF
WC
Policy Gradient is a general class of reinforcement learning algorithms (Williams 1992 Kakade
200T Schulman et al. 2015; 2017). Applying policy gradient methods to our problems essentially throws away the expert advice and learns from environmental rewards alone.
Teacher Distill and On-Policy Distill are the names used by Czarnecki et al. (2019) for versions of the Behavior Cloning and DAgger Ross al.] (201T) algorithms adapted for the distillation frame- work where data is continually generated online. These algorithms do not include an fg term, and so will not be able to recover =cc in all cases.
On-Policy Distill+R Combines the single step loss term from On-Policy Distill with the environ- mental reward used in Policy Gradient. The motivation is that if On-Policy Distill attempts to follow the expert, and Policy Gradient seeks high environmental reward, then combining them seeks to do both.
N-Distill+R is also from Czarnecki et al. (2019) and is similar to COSIL (Nguyen et al., 2022). but adapted to the distillation framework. Unlike COSIL, N-Distill+R uses cross entropy rather than the KL Divergence, but these are identical for deterministic experts, which is the setting we assume in our experiments. The other difference is that N-Distill+R uses the cross entropy term from the first time step as a differentiable loss rather than rolling it into the reward signal, which improves performance. This method is also similar to On-Policy Distill+R except that it augments the reward using future agreement with the expert. This encourages the model to not only take actions that the expert immediately agrees with, but also to seek out states in which it is easy to agree with the expert in the future.
Expert matching Reward+R removes the single-step loss term and instead assigns a fixed scalar to the reward T; when the agent's actions agree with the expert and negative reward -9 when the actions disagree.
Teacher Distill + PPO is similar to DAPG (Rajeswaran et al. 2017) and first trains an agent with Teacher Distill for a fixed number of time steps, then refines this policy using reinforcement learning. This is a popular technique used to learn robot behavior from human demonstrations.
All methods mentioned so far that attempt to learn from both the expert advice and the reward signal suffer from a common problem: they all attempt to balance reward seeking and expert following behavior uniformly across all sequences of observations, and do not attempt to explicitly discover where following the expert yields high reward and where does not.
ADVISOR, from Weihs et al. (021)) attempts to address this by dynamically interpolating between imitation and reinforcement learning signals using a weighting factor w; = e -aK L(Te where Taur is an auxiliary policy trained to follow the expert in every state using the information available to the agent. This weighting factor estimates how closely the agent can follow the expert given the information it has at the current time step. Unfortunately, the ability to follow the expert at a given state is not strictly indicative of states that require exploration. An example demonstrating how this can lead to suboptimal behavior is shown in Appendix E
ELF Distillation is our new technique which overcomes the limitations of the previous methods. Pseudocode is shown in Algorithm [I] The key insight to train two policies jointly: a follower which attempts to learn how to follow the expert, and an explorer Te that attempts to maximize environmental reward using the follower's value function reward shaping.
7

---

Published as a conference paper at ICLR 2023
The follower is trained using distillation that samples data according to switching policy that rolls out experience according to the explorer #c for a random number of time steps, then allows the follower тF to take over. This rollout behavior is also used in Aggre VaTe Ross & Bagnell (2014) and allows us learn to follow the expert from states visited by the explorer that would not normally be visited by the follower. We also train a value function OF on the second half of the trajectory that is rolled out according to the follower. The follower's 's distillation uses cross entropy to the expert's s advice as the single-step loss function with no reward term.
The explorer is trained using policy gradients with reward shaping that encourages the explorer to visit states with high value according the follower's learned value function. The reward term f = r1 + FF 1 (T(TT) uses the follower's learned value function ex as potential-based reward shaping Ng et al. (1999). This is referred to Teacher V Reward in Czarnecki et al. (2019) and is related to Sun et al. (2018). The single-step loss function is unless the follower's 8 value
function U=(rr) is greater than a target value in which case it uses cross-entropy to the follower's distribution. This accelerates learning in states where the follower has already learned to reach high returns.
Training proceeds in alternating fashion, first training the follower switching policy, then training the explorer on new data generated using the explorer (on-policy). It is necessary to keep training the follower because the explorer may reach states in the middle of training that would not be visited otherwise, and it is important to build accurate value estimates for them as well.
data generated by the
At test time, the follower is no longer necessary and can be discarded. The only purpose of the follower is to discover where the expert is reliable and where it isn't in order to overcome the limi- tation of trying to find good global mixing rules for reward seeking and expert following behavior. In this way the follower policy similar to the auxiliary policy in ADVISOR, but it used by the algorithm in a much different way. Rather than guiding exploration using the follower's ability to replicate the expert in a given state, we instead use the follower's value estimate, which encourages the learner to visit states where following the expert leads to high long-term reward. This avoids important failure cases such the one shown in Appendix B
Algorithm ELF Distillation
Require: Nateps Expert Tg. Horizon T, Dataset Sizes Njoloows Nexplore: Target value v. Initialize follower πF.
Initialize follower value estimator uz. Initialize explorer TL. for 1 to N Neteps do Initialize Dialoow 0 for j = 1 to N follow do
Choose uniform random switching time E {L...., T}.
Add new trajectory T to D fattow by sampling Ti... ~ Fe and Tr...T ~ TT.
end for
Train ITF to match ae for all transitions in D follow using cross entropy.
Train ey to match the returns from all transitions in D Dollow after the switching time t.
Initialize D Deuplore== = ().
for j = 1 to Nexplore do
Add new trajectory T to Demplore by sampling nc. Reshape all rewards in 7 to be f = -r--- + uy - ()(
end for
Train Te using policy gradient with an additional cross entropy loss to TTFF when U)((T) 2 v.
end for
6 EXPERIMENTS
In order to evaluate ELF Distill, we compare it against the baselines in Section 5 on several challeng- ing Minigrid Chevalier-Boisvert et al.) 2018) and Vizdoom (Wydmuch et al. 2019) environments with partial information. Figure shows the results with diagrams of the environments.
The goal of the Minigrid environments is to reach a door that has the same color as a set of balls hidden in various locations in the environment. Observations provided as restricted top-down
8

---

Published as a conference paper at ICLR 2023
Single Branch
Backtrack
Backtrack
Open
Farly Explore
Early Explore
Early Explore
Late Explore
MonsterRooml
MonsterRoom2
PPO
Teacher Distill
On-Policy Distill
On Pulicy Distil++R
N-Distill
Export Matching-
Teacher
ADVISOR
Distill (Ours)
02
Figure 2: ELF Distill compared against seven baselines on eight Minigrid and two Vizdoom envi- ronments designed to require various levels of deviation from an impossibly good expert's advice. The inset image of each plot shows a diagram the environment.
view of the gridworld. In this setting memory is handled for the agent by automatically remembering the color of the last ball seen and presenting as an additional observation variable. See Appendix for details.
The goal of the Vizdoom environments is navigate safely up by a cyborg demon. Each map has two exits, one of which is randomly guarded by the monster. The agent must find a window that allows it to see which exit the monster is guarding and take the alternate route. Unlike Minigrid, agents in these Vizdoom environments are provided with first- person view from the player's perspective, and must remember relevant past observations using recurrent network. See Appendix C for details.
exit point without getting blown
From the experiments shown here, we can see ELF Distill performing comparably or better than all other baselines. Note that ADVISOR also performs quite well, except for few environments (Early Explore 2, Early Explore 3, Monster Room 2) where the issues pointed out Appendix F severely impact performance.
7 CONCLUSION
Imitation learning remains a powerful tool for sequential decision making problems. Unfortunately when experts have more information than the policies that learn from them, the experts may be impossibly good, and their behavior and performance cannot be replicated by any learning algo- rithm. We have shown that in these settings, it may still be possible to recover the optimal policy in the limited information space using only the expert's advice, but that this is not guaranteed have shown by providing counter-examples. To address this, we have introduced ELF Distillation, training method that uses the estimated value function of policy trained from expert demon- strations guide exploration for a second reinforcement learning agent. We have shown that this algorithm outperforms several distillation basclines that incorporate both environmental reward and expert demonstrations set of challenging minigrid and vizdoom environments.

---

Published as a conference paper at ICLR 2023
REFERENCES
Susanne Albers and Monika Henzinger. Exploring unknown environments, In Proceedings of the !wenty-ninth annual ACM symposium Theory of computing, pp- 416-425, 1997.
J Andrew Bagnell. An invitation to imitation. Technical report, Carnegie-Mellon Univ Pittsburgh Pa Robotics Inst, 2015.
Mayank Bansal, Alex Krizhevsky, and Abhijit S. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. CoRR, abs/1812.03079, 2018. URL. http:
//axx/v.org/abs//11122.
Mohak Bhardwaj, Sanjiban Choudhury, and Sebastian Scherer. Learning heuristic search via imita- tion. In Conference on Robot Learning. pp. 271-280. PMLR, 2017.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating, 2019. URL
abs/ 912 12294
Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for gymnasium. htttps:///gttt com http//araaaaauunattiiionoon wowaaaaaoooooaaaaaaoooaana.- 2018.
Sanjiban Choudhury, Ashish Kapoor, Gireeja Ranade, Sebastian Scherer, and Debadeepta Dey.
Adaptive information gathering via imitation learning. arXiv preprint arXiv:/705.0788 2017.
Sanjiban Choudhury, Mohak Bhardwaj. Sankalp Arora, Ashish Kapoor, Gireeja Ranade, Sebastian Scherer, and Debadeepta Dey. Data-driven planning via imitation learning. The International Journal of Robotics Research, 37(13-14):1632-1777 2018.
Felipe Codevilla, Eder Santana, Antonio M. López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. CoRR, abs/1904.08990, 2019. URL http://
arxiv. ottlv.orrg/aaa abs 904 08980
Wojeiech M Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. The 22nd International Conference on Arti- ficial Intelligence and Statistics, pp. 1331-1340. PMLR, 2019.
Hal Daumé, John Langford, and Daniel Marcu. Search-based structured prediction. Machine learn- ing, 75(3):297-325 2009.
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. Ad- vances Neural Information Processing Systems, 119999 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL https:
//arxiv. abs 11011.01210
Unnat Jain, Iou-Jen Liu, Svetlana Lazebnik, Aniruddha Kembhavi, Luca Weihs, and Alexander G Schwing. Gridtopix: Training embodied agents with minimal supervision. In Proceedings of the JEEE/CVF International Conference on Computer Vision pp. 15141-15151, 2021.
Leslie Pack Kaelbling, Michael L Littman, and Authony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-144, 1998.
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
Ming- Ming-Yang Kao, John H Reif, and Stephen R Tate. Searching in an unknown environment: An optimal randomized algorithm for the cow-path problem. Information and Computation, 131(1): 63-79, 1996.
Alex Kuefler, Jeremy Morton, Tim Wheeler, and Mykel Kochenderfer. Imitating driver behavior with generative adversarial networks. In 2017 IEEE Intelligent Vehicles Symposium (IV), pp- 204-211. IEEE, 2017.
Daniel Kumor, Junzhe Zhang, and Elias Bareinboim. Sequential causal imitation learning with unobserved confounders. Advances in Neural Information Processing Systems, 34: 14669-1466680, 2021.
10

---

Published as a conference paper at ICLR 2023
Gilwoo Lee, Brian Hou, Sanjiban Choudhury, and Siddhartha S Srinivasa. Bayesian residual pol- icy optimization: Scalable bayesian reinforcement learning with clairvoyant experts. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5611-5618. IEEE, 2021.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xeabc5986, 2020.
Michael Littman, Anthony R Cassandra, and Leslie Pack Kaelbling. Learning policies for partially observable environments: Scaling up. In Machine Learning Proceedings 1995, pp. 362-370.
Elsevier, 1995.
Urs Muller, Jan Ben, Eric Cosatto, Beat Flepp, and Yann L Cun. Off-road obstacle avoidance through end-to-end learning. In Advances in neural information processing systems, pp. 739- 746. Citeseer, 2006.
Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. leml, volume 99, pp. 278-287, 1999.
Hai Nguyen, Andrea Baisero, Dian Wang, Christopher Amato, and Robert Platt. Leveraging fully observable policies for learning under partial observability. arXiv preprint arXiv:2211.01991, 2022.
Pedro A Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Ve- ness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, al. Shaking the foundations: delusions in sequence models for interaction and control. arXiv preprint arXiv:2110.108/1 2021.
Judea Pearl, Madelyn Glymour, and Nicholas Jewell. Causal inference in statistics: A primer.
John Wiley & Sons, 2016.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. 1989.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.1000 2017.
Stephane Ross and Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret 2014.
learning. arXiv preprint
Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning, 2010. URL htttps://eexii.crrrrrre/rs /bs//
71011.0885
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distil- lation. arXiv preprint 151iv:1511. 1.06295, 2015.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and Andrew Bagnell. Feedback in imitation learning: The three regimes of covariate shift, 2021. URL https://
arxiv. hbtg/ab//// 22122 02872
Wen Sun, Arun Venkatraman, Geoffrey Gordon, Byron Boots, and Andrew Bagnell. Deeply ag- grevated: Differentiable imitation learning for sequential prediction. In International conference machine learning, pp. 3309-3318. PMLR, 2017.

---

Published as a conference paper at ICLR 2023
Wen Sun, Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining rein- forcement learning & imitation learning. arXiv preprint urXiv:18053 1240, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Gokul Swamy, Sanjiban Choudhury, Andrew Bagnell, and Steven Wu. Of moments and match- ing: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning. pp. 10022-10032. PMLR, 2021.
Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Zhiwei Steven Wu. Sequence model imitation learning with unobserved contexts. arXiv preprint arXiv:2208.0222 02222, 2022.
Guy Tennenholtz, Assaf Hallak, Gal Dalal, Shie Mannor, Gal Chechik, and Uri Shalit. On co- variate shift of latent confounders in imitation and reinforcement learning. arXiv preprint arXiv:2110.06539 2021.
Andrew Warrington, Jonathan W Lavington, Adam Scibior, Mark Schmidt, and Frank Wood. Robust asymmetric learning in pomdps. In International Conference Machine Learning, pp. 11013- 11023. PMLR, 2021.
Luca Weihs, Unnat Jain, Iou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, and Alex Schwing. Bridging the imitation gap by adaptive insubordination. Advances in Neural Information Processing Systems, 34:19134-19144 2021.
Ronald J Williams. Simple statistical grolient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-256 256, 1992.
Marek Wydmuch, Michal Kempka, and Wojciech Jaskowski. ViZDoom Competitions: Playing Doom from Pixels. IEEE Transactions Games, 11(3):248-259 2019. doi: 10.11097TG.2188 2877047. The 2022 IEEE Transactions on Games Outstanding Paper Award.
Tianpei Yang. Hongyao Tang. Chenjia Bai, Jinyi Liu, Jianye Hao, Zhaopeng Meng, and Peng Liu. Exploration in deep reinforcement learning: a comprehensive survey. arXiv preprint arXiv:2109. 06668, 2021.
Junzhe Zhang, Daniel Kumor, and Elias Bareinboim. Causal imitation learning with unobserved confounders. Advances in neural information processing systems, 15:11266-12274 2020.
Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp.
Chicago, IL, USA, 2008.
12

---

Published as a conference paper at ICLR 2023
APPENDIX
A PROOF OF THEOREM 1
Recall that the policy #e is computed using the empirical distribution of dataset D sampled from a rollout policy np:
Tc( = E1P1 Tr T2.0g:
(4)
Σ 121 TOITT
We want show:
The empirical policy Re will recover T2 as |D] - oo iff the following conditions hold:
(Coverage) PxD (T2) + 20 E P=
2. (Correctness) Ergwap Ne' ((((ss))))((())))
Proof. We first show that the coverage condition is necessary, and then show that the correctness condition necessary and sufficient when the coverage condition holds.
The coverage condition is necessary: the coverage condition does not hold, for at least some observation histories Tg, the denominator of Equation 4 will be:
Σ 1 =
This will result in an undefined distribution for 7e at some trajectories that are encountered while acting according to z². Undefined behavior is not optimal according 72 this shows that the coverage condition is necessary to recover T².
The correctness condition is necessary and sufficient when the coverage condition holds: Because the dataset D is sampled according to TD, as [D] oo, the empirical distribution of expert recom- mendations in Equation becomes:
# EE (Os (T)))(((()(((zz
Te PxD
If the coverage condition holds, then Pa C Pap. This allows us to extend the expression above to Prg. the set of trajectories with nonzero probability under Tē
# #T())) (egg) V V E Pag
This means that if the conrectness condition holds:
(c(Te) (e0== = =2(72) V V E Pag
This shows that the agent-optimal policy can only be recovered if the distribution over labels gen- erated by the expert Te under the sampling distribution of the state space, match the agent-optimal policy 72 wherever the states map T
B MINIGRID ENVIRONMENTS, MODEL AND TRAINING DETAILS
B. ENVIRONMENT
stated Section ( the goal of the minigrid environments is to reach the door that matches the color of one or more balls placed in the scene. At the beginning of each episode all balls in the scene be either blue or red, and they remain that color for the entire episode. There one door per color and it's location is fixed for all time. The expert knows in advance the color of the balls and provides the agent with direct supervision to reach the correct door, but does not consider
13

---

Published as a conference paper at ICLR 2023
Single Branch
Backtrack
Backtrack
Open
Early Explore
Farly Explore
Early Explore
Late Explore
Figure 3: Minigrid Maps
the agent's ignorance of the ball color. Without taking further exploratory action to find the balls first, the agent cannot determine which door to go to and will always underperform compared to the expert. In order succeed, the agent must deviate from the instructions provided by the expert in order to gather the information necessary complete the task.
The agent gets reward of 0.01[T| for entering the correct door for episode. During training the expert demonstrations are provided by computing a shortest line path from the agent to the correct door without bothering to visit locations where the balls are located.
Visibility is strictly limited and the agent can only view the 3 x 3 tiles directly in front of them. While these environments are small, they are quite challenging due to the sparse reward and the need to automatically learn the color-matching behavior.
B.2 MODEL
All minigrid methods train the same small model that takes 3x3 grid with two channels representing object type (wall, door, etc) and single color. The model automatically remembers the color of balls that has seen in the past. The field of view of the agent is a 3x3 grid of tiles. The input the network is provided as integer indices, 16-channel representation of the observed object type, another 3x3 16-chamnel representation of the observed object color and 1x1 16-channel representation of the remembered ball color (a third "grey" color used when the ball has been observed yet). The feature for the remembered ball color is tiled to 3x3 and these features are added together and flattened. The model then uses two fully connected layers with 256 channels each and ReLU activations. A policy head consists of another 256-channel linear layer followed by Tanh activation and a linear projection to the number of actions 7. A value head is the same except it projects single value. ELF distill trains two policies, so has one network for each. ADVISOR has an additional auxilliary head.
three embedding layers construct 3x3
B.3 TRAINING
Each method was trained on 220 25 1M frames. When training ELF Distill, both the follower and explorer were trained 219 s 500K frames, that the total frames observed during training was equal the other methods. All methods were trained with 10 different random seeds in each environment. PPO (Schulman et al. 2017) was used as the loss function to maximize reward in all algorithms with P. term.
14

---

Published as a conference paper at ICLR 2023
Monster Room
Monster Room
Window
Demon
Door
Alterate
Exit
Demon
Agent Start
Location
Figure 4: Vizdoom Maps
C VIZDOOM ENVIRONMENTS, MODEL AND TRAINING DETAILS
C.I ENVIRONMENTS
Our VizDoom (Wydmuch al. 2019) were built with the free SLADE Doom editor. The goal of these environments is to escape the room without getting destroyed by a cyborg demon. The agent must using first-person visual data, and memory of the past. The native resolution of the environment is 320 x 240 RGB pixels which we rescale to 84x84 grayscale channels. The game engine runs at very high frequency, so we use frame skip of 8 to avoid making decisions at fine of granularity. The agent access to four actions: WALK FORWARD, TURN LEFT, TURN RIGHT and USE which both opens doors and pushes the switch to end the level. The damage settings have been tuned that single hit from the monster guarantees instant death. Although the agent can be seen carying pistol, it purely decorative, and the agent has no ability to fight back against the monster.
The agent gets a reward for completing the level successfully, reward of -2 for getting blown up, and a reward of 0 if neither happens before the maximum number of 72 frames. We also give the agent an exploration bonus based how far it moves away from the nearest location it has been before, and a small negative reward of -0.001 at each time step. We also use early- carly-termination, stopping an episode if the agent tries walk forward, but makes progress (hitting forward while pointing directly at wall), switches between looking left and looking right three times in a row, or if the agent pushes the USE button twice in row when not in front of a door or switch. These help avoid long trajectories of meaningless behavior in early episodes. If any of these early termination conditions are triggered, the episode ends with a small penalty of -0.05 for the agent.
C.2 TRAINING
Each method was trained 221 2M frmes. As with Minigrid, train each component of ELF Distillwith half that number so that the total number of training frames are comparable. All methods were trained with 3 different random seeds in each environment, As with Minigrid, PPO was used as the loss function to maximize reward
algorithms with F; term.
D DISCUSSION OF RESULTS
D.1 MINIGRID
Our eight MiniGrid Environments belong to five categories. In all environments we colored the walls so that the agents with their very limited field of view could find their way around.
Single Branch was designed to mimic Example I in Figure I In order to reach any door, the agent must pass by the balls. This is the easiest case, and the only one in which Teacher Distill can do better than 0.5. PPO is often successful here, but often takes longer reach the goal than methods
15

---

Published as a conference paper at ICLR 2023
that the expert. Here we see common failure mode of Expert Matching+R in that the agent learns a state-action loop for which the agent agrees with most of the expert's actions, but disagrees with few of them. This allows the agent to continuously cycle and gain infinite reward according it's reshaped reward objective that attempts to maximize agreement with the expert while failing to complete the task.
Backtrack two environments, one larger than the other designed to mimic Example II in Figure T This time ball placed near each door so that if the agent heads to the wrong one initially, can backtrack to the other. On-Policy Distill often reaches the correct goal, but does not perform as well as reward seeking approaches due ambiguous expert advice before the balls have been observed. We also see Teacher Distill then PPO perform well in the smaller version, but it takes a long time to learn. PPO performs much worse this task. ELF Distill, ADVISOR, On-Policy Distill+R and N-Distill+R all perform very well on these problems, with ELF Distill underperforming the others in few cases.
Early Explore are three environments of different sizes designed to mimic Example III in Figure I Here the ball is placed on a separate hallway that is not on the path to any door. An algorithm must incorporate environmental rewards to be successful here. Here ELF Distill clearly dominates all other methods, although the problem becomes more difficult the futher the agent has to explore from states suggested by the expert. ADVISOR performs well on the first environment, but fails at the other two due to the issues pointed out in Appendix
Late Explore is similar to Backtrack in that the balls are near the doors. However in this case, they are far enough away that they will not be observed by agent that walks directly to the door. Here ELF Distill and ADVISOR dominate, with On-Policy Distill+R and N-Distill+R also making progress.
Open places the balls and doors in opposite corners of the room with most of the space freely navigable. ELF Distill outperforms all others, while ADVISOR also makes progress.
D.2
VIZDOOM
Our Vizdoom environments were designed to provide a more challenging visual scenario for training agents from impossibly good experts.
Monster Room 1 Here the agent must exit small room with two doors to enter a large room with windows where the agent can check for the existence of monsters. The agent then exit the level by going back to the original room and opening the door to the empty room and proceeding to the exit. Only ELF Distill and ADVISOR are able to reliably complete this task.
Monster Room the second Monster Room, the agent starts outside of the room with two doors. Because the expert knows where the monster is, it will always tell the agent to proceed to the room with two doors. This causes problems for ADVISOR, which will latch onto this predictable expert signal without knowing what to do once it has to make a decision about which door to enter. ELF Distill is the only method to reliably make progress on this task.
E ADVISOR FAILURE CASE
Example IV
D snU
exper
Figure 5: ADVISOR cannot recover the agent-optimal policy कद in Example IV.
Example IV in Figure providess a demonstration of a failure case for ADVISOR. In this example, nothing prevents the auxiliary policy Tauz from replicating expert behavior at A, meaning the AD- VISOR loss will strongly favor the imitation learning signal at this location which encourages the
16

---

Published as a conference paper at ICLR 2023
transition from A to B. Unfortunately, however, the agent-optimal action is first travel from A to C in order to learn the value X, then backtrack to and continue to the goal. Because ADVISOR is able to reproduce the expert's behavior at A, it will ignore the transition to C, and will be unable gather the necessary information.
F LIMITATIONS
Because the explorer policy relies on value estimates from the follower policy, it is possible for biases those estimates to cause poor explorer performance. This is especially true early in the training process before the follower's s value function has had time to learn to differentiate good and bad states. This be exacerbated by the fact that the follower's state coverage is determined by the explorer due to the switching policy used to collect training data for the follower. This can lead to situations where the follower underestimates the value of particular state, which causes the explorer to visit it less often, which then causes the follower to get less training data for that state and therefore fail to recover from its initial error.
This suggests that may be beneficial use a training schedule that devotes more samples the follower early on order to provide better estimates of the value function, and more samples to the explorer later once this value function is well modelled.
Taken to an extreme, this would suggest training the follower alone for half the training step budget, then freezing it and training the explorer. Unfortunately. this will not be ideal in many cir- cumstances, as the explorer's policy is used to determine which states necessary for the follower to learn. However, it may be possible to automatically decide when to allocate training resources to the follower or the explorer if the quality of the follower's 's value function can be estimated online.
G ACKNOWLEDGEMENTS
We would like to thank the members of the RAIVN and Robotics and State Estimation labs at the University of Washington for their support, discussion and feedback. We would also like to thank Luca Weihs for a valuable discussion the ADVISOR algorithm.
17


Reviews:
[{"review_id": "Q6han4CROt", "reviewer": "Reviewer_ZZSA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studied a interesting problem setting in imitation learning where experts have more information than the policies that learn from them. Thus, the experts may be impossibly good, and their behavior and performance cannot be replicated by any learning algorithm. This paper analyzed different existing approaches on this probelem setting and summarized one limitation: the existing approaches do not attempt to explicitly discover where following the expert yields high reward and where it does not. Based on this limitation, it proposed a training method, ELF Distillation, which uses the estimated value function of a policy agent trained from expert demonstrations to guide exploration for a second reinforcement learning agent. It further empirically demonstrates the power of the proposed method in minigrid environment. The experimental results show that the ELF Distillation outperforms several distillation baselines that incorporate both environmental reward and expert demonstrations.\n\nThe key insight in ELF Distillation is to train two policies jointly: a follower π_ϕ which attempts to learn how to follow the expert, and an explorer π_θ that attempts to maximize environmental reward using the follower’s value function as reward shaping. Training proceeds in an alternating fashion, first training the follower π_ϕ on new data generated by a combination of the explorer's action and the expert demonstration, then training the explorer π_θ on new data generated using the explorer.\n\nOverall, this work proposed a novel and effective method in the problem setting where experts have more information available than the policy to be learned when making decisions. The paper is well-written, and the proposed method is also well-motivated. My major concern is on the experimental side as all experiments are on conducted under one type of synthetic environment --- the minigrid. I would recommend this work to show the effectiveness of ELF Distillation in at least one real-world setting.", "summary_of_the_review": "Overall, this work proposed a novel and effective method in the problem setting where experts have more information available than the policy to be learned when making decisions. The paper is well-written, and the proposed method is also well-motivated. My major concern is on the experimental side as all experiments are on conducted under one type of synthetic environment --- the minigrid. I would recommend this work to show the effectiveness of ELF Distillation in at least one real-world setting.", "strength_and_weaknesses": "Strength:\n-\n\n[+] The proposed training method ELF Distillation is novel and well-motivated.\n\n[+] The work also empirically demonstrates that ELF Distillation performs better than a variety of strong baselines on a suite of minigrid environments.\n\nWeakness:\n-\n\n[-] Regarding the training of the follower, I wonder how to guarantee that the follower will learn to give high values to critical states (e.g., visit the ball) which the expert will never need to visit. I did not see a specific term to motivate the follower to learn to assign high values to such states (despite that these critical states will appear in the training data)\n\n[-] Regarding the experiments, current experiments are on minigrid with synthetic data, and I wonder how the proposed algorithm will perform in real-world scenarios.\n\nMinor comment：\n- Can you provide a concrete example of the problem-setting difference from Swamy et al. (2022). It is unclear from the description in the first paragraph of section 4\n- What is the H function in Table 1？", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FbLKz_imZg", "reviewer": "Reviewer_i2wK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper concerns itself with the situation in which some expert(s) may have access to more information than the learning agent. During training the expert has access to advice from the expert but this is not the case during testing.\n\nAfter shortly describing the Behaviour Cloning and Dagger techniques, the paper investigates under what conditions these techniques confined the agent-optimal policy, and it is shown that in some settings the expert advice may not be sufficient for this purpose. Next it is investigated how to/whether to incorporate environmental reward into the learning algorithm -- to this end a new technique called ELF Distill is presented which is also shown to outperform previously known techniques (in most cases).", "summary_of_the_review": "Overall I found the paper interesting. Given my background, I find it difficult to determine where exactly the results place in the landscape of current literature, but I found the studied paper, results and approach interesting and with the exception of some parts that I found hard to follow also generally well written. The results themselves and approaches employed are not very innovative, but in my point of view that is not necessarily required and the paper is still of interest.", "strength_and_weaknesses": "Strengths:\n- Interesting subject\n- Overall good quality of writing\n\n\nWeaknesses:\n- The paper is not completely self-contained. I come from a different area and although I have worked on experts I found it difficult to follow some part mostly because of missing context.\n- The example with the robot searching in an unknown environment reminds me of several settings from online algorithms. See for example \"Exploring Unknown Environments\" by Albers and Henzinger, SIAM J. Computing, 2000, or a sequence of papers on the \"Cow path problem\" and its generalisations. I personally would find it good if the current paper included a discussion on the differences/similarities between the models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8V1O82pKkER", "reviewer": "Reviewer_tPto", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the problem of imitating an expert with access to more information about the state space in a POMDP than a learner is considered -- these are called impossibly good experts. Existing methods for imitation learning like Behavior Cloning and the DAGGER algorithm are shown to fail with concrete counterexamples, and necessary and sufficient conditions are given for learning from an impossibly good expert. The paper proposes an algorithm to solve this problem called ELF Distillation, which makes use of existing techniques for making use of information from sub-optimal experts (the paper considers sub-optimal experts because environments can be constructed where the impossibly good expert is unhelpful) by combining expert demonstrations with reward signal from the environment itself. The main twist that the ELF algorithm proposes on top of these existing methods is to avoid balancing reward seeking behavior and expert following $\\textit{uniformly}$ across all sequences of behavior; ELF instead proposes to jointly train a follower that follows the expert and an explorer which uses the follower's learned value function as a reward-shaping mechanism. This algorithm is tested empirically on grid-world based POMDP tasks and against an expert given appropriate information commensurate with the problem setting, for which it performs favorably compared to existing methods.", "summary_of_the_review": "Overall, the paper is a decent contribution to the literature on following an impossibly good expert. My main concern is on the novelty/ sufficient amount of additional learnings present in the paper.\n\n===== Post-Rebuttal ======\n\nAfter seeing the update, I'm inclined to think that the additional experiments and framing of the contribution make the paper sufficiently interesting for publication. Thus I updated my score.", "strength_and_weaknesses": "* Strengths: The problem setting proposed is an interesting one, and a novel algorithm is provided which appears to perform favorably in some gridworld settings. The authors also provide some concrete examples of when various other natural algorithms to try may not work. They also provide a characterization of some conditions on environments for which an empirical distribution of expert demonstrations will converge to the optimal policy. Finally, the authors test their new algorithm on several conceptually sound environments that probe at natural difficulties involved with the problem setting.\n\n* Weaknesses: The primary contribution of this paper, in my opinion, is the novel algorithm that they propose and test -- the counter-examples are fairly straightforward. However, the ideas in the algorithm do not appear to be that novel by themselves. This in itself would be fine if it were shown that the resulting algorithm works significantly better than other approaches on many kinds empirically, or if there were some interesting theoretical insight to pair with the improved algorithm, but the experiments are fairly limited and do not necessarily show a significant improvement over existing methods, though it seems clear that a trend exists that the mean performance of ELF is better on all these tasks. Also, Theorem 1 is not that interesting in my opinion (the conditions are fairly clear to see). Thus I am more lukewarm on accepting the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster