Bridging the Imitation Gap by Adaptive Insubordination
Authors: Luca Weihs
Unnat Jain
Jordi Salvador
Svetlana Lazebnik
Aniruddha Kembhavi
Alex Schwing
Venue:  (2021)
Keywords: Privileged Experts
Imitation Learning
Reinforcement Learning
Actor-Critic
Behavior Cloning
MiniGrid
Knowledge Distillation

Abstract: When expert supervision is available, practitioners often use imitation learning with varying degrees of success. We show that when an expert has access to privileged information that is unavailable to the student, this information is marginalized in the student policy during imitation learning resulting in an ''imitation gap'' and, potentially, poor results. Prior work bridges this gap via a progression from imitation learning to reinforcement learning. While often successful, gradual progression fails for tasks that require frequent switches between exploration and memorization skills. To better address these tasks and alleviate the imitation gap we propose 'Adaptive Insubordination' (ADVISOR), which dynamically weights imitation and reward-based reinforcement learning losses during training, enabling switching between imitation and exploration. On a suite of challenging didactic and MiniGrid tasks, we show that ADVISOR outperforms pure imitation, pure reinforcement learning, as well as their sequential and parallel combinations.

Under review as a conference paper at ICLR 2021
BRIDGING THE IMITATION GAP BY ADAPTIVE INSUBORDINATION
Anonymous authors Paper under double-blind review
ABSTRACT
When expert supervision is available, practitioners often use imitation learning with varying degrees of success. We show that when an expert has access to privileged information that is unavailable to the student, this information is marginalized in the student policy during imitation learning resulting in an "imitation gap" and, potentially, poor results. Prior work bridges this gap via a progression from imitation learning to reinforcement learning. While often successful, gradual progression fails for tasks that require frequent switches between exploration and memorization skills. To better address these tasks and alleviate the imitation gap we propose 'Adaptive Insubordination' (ADVISOR), which dynamically weights imitation and reward-basse reinforcement learning losses during training, enabling switching between imitation and exploration. On a suite of challenging didactic and MINIGRID tasks, we show that ADVISOR outperforms pure imitation, pure reinforcement learning, as well as their sequential and parallel combinations.
INTRODUCTION
Imitation learning (IL) can be remarkably successful in settings where reinforcement learning (RL) struggles. For instance, IL succeeds in complex tasks with sparse rewards Booisseer-Boisserttt et al.) 2018a| Peng et al., 2018 Nair et al., 2018), and when the observations are nigh-dimensional, e.g.. in visual 3D environments (Kolve et al., 2019; Savva et al. 2019). In such tasks, obtaining a high quality policy purely from reward-based RL often challenging, requiring extensive reward shaping and careful tuning as reward variance remains high. In contrast, II. leverages expert which is generally less impacted by the environment's random state. However, designing an expert often relies on privileged information that is unavailable at inference time. For instance, it is straightforward to create navigational expert when privileged with access to a connectivity graph of the environment (using shortest-patt algorithms) (e.g., Gupta al. 2017 (b) leverages an available semantic map (e.g.. Shridhar et al. 2020 Das et al.) 2018b). Similarly, game experts may have the privilege of seeing rollouts (Silver et al. (016) vision-based driving experts may have access to ground-truth layout (Chen et 2020). Such graphs, maps, rollouts, or layouts aren't available to the student at inference time.
instruction-folllee followwig expert which
How does use of a privileged expert influence the student policy? We show that training an agent imitate such expert results in a policy which marginalizes out the privileged in- formation. This can result in a student policy which is sub-optimal, and even near-uniform, over large collection of states. We call this discrepancy between the expert policy and the student policy the imitation gap. A frequent strategy used in prior work to improve upon expert demonstrations (and implicitly to overcome the imitation gap when applicable) is stage-wise training: II. is used to 'warm start' learning and subsequent reward-based RL algorithms, such as proximal policy optimization (PPO), are used for fine-tuning (Lowe et al. 2020). While this strategy is often successful, the following example shows that it can fail dramatically.
4.
?
Example (Poisoned Doors). Suppose an agent is presented with N > 3 doors d1, dx. As illustrated in Fig. (for N = 4), opening di requires entering an unknown but fixed code of length M. Successful code entry results in a guaranteed reward of otherwise the reward is Since the code is unknown to the agent, it would have to learn the code. All other doors be opened without code.
-2
2
Consistent reward behind locked door
High-value door chosen at random
Figure PoisonedDoors

---

Under review as a conference paper at ICLR 2021
For some randomly chosen 2 S < N (sampled each episode), the reward behind d, is 2 but for all E .2..... .N} \ the reward behind d, is -2. Without knowledge of j. the optimal policy is to always enter the correct code to open dj obtaining an expected reward of 1. In contrast, if the expert is given the privileged knowledge of the door d3 with reward 2, it will always choose to open this door immediately. It is easy see that agent without knowledge of attempting to imitate such an expert will learn open door among dg,... dy uniformly at random obtaining an expected return of -2 (NV (/ - 1). Training with reward-basee RL after this 'warm start' is strictly worse than starting without it: the agent needs to unlearn its policy and then, by chance, stumble into entering the correct code for door d1, a practical impossibility when M is large.
To bridge the imitation gap, we introduce Adaptive Insubordination (ADVISOR). ADVISOR adap- tively weights imitation and RL losses. Specifically, throughout training we use an auxiliary actor which judges whether the current observation is better treated using an IL or RL loss. For this, the auxiliary actor attempts to reproduce the expert's 's action using the observations of the student at every step. Intuitively, the weight corresponding to the II. loss is large when the auxiliary actor can reproduce the expert's action with high confidence and is otherwise small. As we show empirically, ADVISOR combines the benefits of IL and RL while avoiding the pitfalls of either method alone. Most IL algorithms were designed with common-sense but strong assumptions which implicitly disallow a discrepancy between expert and student observations: it is when these assumptions are violated (as they often are in practice) that the imitation gap appears.
We evaluate the benefits of employing ADVISOR across ten tasks including the Poisoned Doors discussed above, 2D gridworld, and suite of tasks based on the MINIGRID environment (Chevalier Boisvert et al.) 2018a b). Across all tasks, ADVISOR outperforms popular IL. and RL baselines as well as combinations of these methods. We also demonstrate that ADVISOR can learn to ignore corruption in expert supervision. ADVISOR can be easily incorporated into existing RL pipelines. The code to do the same included in the supplement and will be made publicly available.
2 RELATED WORK
A series of solutions (e.g.. Mnih al., 2015 van Hasselt et al., 2016 Bellemare et al.) 2016 Schaul
jet al. 2016) have made off off-coliy deep Q Q-learning methods stable for complex environments like
Atari Games Several high-performance (on-policy) policy-gradient methods for deep-RL have
also been proposed (Schulman et al. 2015a Mnih et al. 2016: Levine et al.] [016:] Wang et al.
2017: (Slver et al.] 2016). For instance, Trust Region Policy Optimizationn (TRPO) (Schulman et al. 2015a) improves eunllceffficienyy by safely integrating larger gradient steps, but is incompatible with architectures with shared parameters between policy and value approximators. Proximal Policy Optimization (PPO) (Schulman et al. (017)) employs a clipped variant of TRPO's s surrogate objective and is widely adopted in the deep RL community. We also use
baseline in our experiments.
As environments get more complex, navigating the search space with only deep RL and simple heuristic exploration (such as e-greedy) is increasingly difficult, leading to methods that imitate expert information (Subramanian et al. (016). While several approaches exist for leveraging expert feedback, e.g.. Cederborg et al. (2015) consider policy shaping with human evaluations, a simple, popular, approach to imitation learning (IL) is Behaviour Cloning (BC), supervised classification loss between the policy of the learner and expert agents (Sammut et al., 1992 Bain & Sammut 1995). BC suffers from compounding of errors due to covariate shift, namely if the learning agent makes single mistake at inference time then it can rapidly enter settings where it has never received relevant supervision and thus fails (Ross & Bagnell 2010). Data Aggregation (DAgger) (Ross al.. 2011] 1) is the go-to online sampling framework that trains a sequence of learner policies by querying the expert at states beyond those that would be reached by following only expert actions. is further enhanced. e.g., via hierarchies (Le et al., 2018). by improving over the expert (Chang et al.] [ et al. 2015: et al. 2020). bypassing any intermediate reward function inference (Ho & Ermon [0066]. and/or learning from experts that differ from the learner (Gupta et al.) [2017a] Jiang [019 Gangwani & Peng. 2020). A sequential combination of IL and RL, i.e., pre-training a model on expert data before letting the agent interact with the environment, performs remarkably well. This strategy has been applied in a wide range of applicatioos - the game of Go (Silver et al. (0016). robotic and motor skills (Pomerleau 1991; Kober & Peters 2009; Peters & Schaal 2008| Rajeswaran et al. 2018). navigation in visually realistic environments (Gupta et al.]|20 [0176 Das et al.]||0 2018a), and web & language based tasks (He et al. 2016 Das et al. 2017 et al. 2017 Wang et al. 2018).
2

---

Under review as a conference paper at ICLR 2021
Recent methods mix expert demonstrations with the agent's own roll-outs instead of using sequential combination of IL followed by RL. Chemali & Lazaric (2013) perform policy iteration from expert and on-policy demonstrations. DQiD (Hester et al. 2018) initializes the replay buffer with expert episodes and adds roll-outs of (a pretrained) agent. They weight experiences based on the previous temporal difference errors (Schaul et al. 2016) and use supervised loss to learn from the expert. For continuous action spaces, DDPGID Vecerík et al.
(017)) is an analogous incorporation of IL
into DDPG (Lillicrap et al. 2016). POID (Kang et al. 2018) improves performance by adding demonstration-guiee
exploration term, i.e., the -hannnn-- divergence between the expert's and the learner's policy (estimated using occupancy measures). THOR uses suboptimal experts to reshape rewards and then searches over a finite planning horizon (Sun et al. 2018)) 8). Zhu et al. (2018) show that combination of GAIL (Ho & Ermon 2016) and RL can be highly effective for difficult manipulation tasks.
Critically, the above methods have, implicitly or explicitly, been designed under certain assumptions (e.g., the agent operates in an MDP) which imply the expert and student observe the same state. Different from the above methods, we investigate the difference of privilege between the expert policy and the learned policy. Contrary to sequential, static, rule-based combination of supervised loss or divergence, we train an auxiliary actor to adaptively weight IL and RL losses. To the best of our knowledge, this hasn't 't been studied before.
Our approach attempts to address imitation directly, assuming the information available to the learning agent is fixed. An indirect approach for reducing this gap is enrich the information available to the agent or to improve the agent's memory of past experience. Several works have considered this direction in the context of autonomous driving (Codevilla al. [0018] Hawke al. (020) and continuous control (Gangwani et al.) 2019). We expect that these methods can be fruitfully combined with the method that we discuss next.
3 ADVISOR
We first introduce notation to define the imitation gap and illustrate how it leads to 'policy averaging." Next, using the construct of an auxiliary policy, we propose ADVISOR bridge this gap. Finally, we show how to estimate the auxiliary policy in practice using deep nets.
3.1 IMITATION GAP
We want agent to complete task T in environment E. The environment has states E S and the agent executes an action a E A at every discrete timestep 2 O. For simplicity and w.l.o.g. assume both A and S are finite. For example, let È be a ID ---griwworld in which the agent is tasked with navigating to a location by executing actions to move left or right, as shown in Fig. 2a Here and below we assume states 8 E S encapsulate historical information so that s includes the full trajectory of the agent up to time & 2 0. The objective is to find a policy n, a mapping from states to distributions over actions, which maximizes an evaluation criterion. Often this policy search is restricted to a set of feasible policies iess, for instance 1e may be the set (=(=:) 0 E RD where (-: 0) is deep neural network with D-dimensional parameters In classical (deep) RL (Mnih et al. 2015 2016), the evaluation criterion is usually the expected dddicouutedd future return.
We focus on the setting of aurtaall--observe makes decisions without access to the full state information. We model this restricted access by defining a filtration function f S - Oj and limiting the space of feasible policies to those policies ITfens for which the value of x (s) depends on s only through f(s), i.e., so that f() = f(s)) implies We call any æ satisfying this condition an s-partial policy and the set of feasible /-partial policies ITleas In a gridworld example, f might restrict to only include information local to the agent's s current position as shown in Figs. 2c 2d
Markov decision processes (POMDPs) where an agent
If a f-partial policy is optimal among all and for any It is frequently the case that, during training, we have access to an expert policy which is able to successfully complete the task T. This expert policy may have access to the whole environment state and thus may be optimal among all policies. Alternatively, the expert policy may, like the student, only make decisions given partial information (e.g., human who sees exactly the same inputs as the student). For flexibility we will
other f-partial policies, we say it is fooptimll. We call o E O, partial-observaaiee
f-partial policy ; we write T (o) to mean π (s) if f(s)
3

---

Under review as a conference paper at ICLR 2021
Envirommentt start states
Hypotheticl epicode
8 Papatialll obs.
e S-pertiall obs.
00
2. Mose riglt
110 10
3. Moxe
Goal
Boundtary Agems
Actions A (lefl, right)
Gosl bou
(a)
(b)
(c)
(d)
Figure 2: Effect of partial observability in 1-dimensional gridworld environment. (a) The two start states and actions space for ID-Lighthuuse with N = 4. (b) A trajectory of the agent following hypothetical random policy. At every trajectory step display output probabilities as per the shortest-path expert (xxP) for each state. (c/d) Using the same trajectory from (b) we highlight the -ortivlloberrvtiions available to the agent (shaded gray) under different filtration function ft f² Notice that, under f², the agent does not see the goal within its first four steps. The policies =!.l II Tii T2, learned by imitating TXP,, show that imitation results in sub-optimal policies iee. aT)l Tl2 # oxp.
define the expert policy as ग yop you, denoting it is a f policy for some filtration function faP For simplicity, we will assume that π you is poxp-optimal. Subsequently, we will drop the subscript foup unless we wish to explicitly discuss multiple experts simultaneously.
In IL (Osa et al. 2018: Ross et al. 201T), Ty is trained to mimic xh by minimizing the (expected) cross-entropy between f and nh over set of sampled states 8 E S:
min E,E"E ,Ty) E
(1)
where CF(x"T, (T) (S) @ (S) @ denotes the usual dot-product, and S is a random variable taking value & E 8 with probability measure ; - [0, 1). Often ju(s) is chosen to equal the frequency with which an exploration policy (e.g., random actions or TP*)) visits state in
randomly initialized episode. When it exists, we denote the policy minimizing Eq. m u."""
When j and TSP are unambiguous, we write - =x un.""
What happens when there is a difference of privilege (or filtration functions) between the expert and the student? Intuitively, if the information that an expert uses to make a decision is unavailable the student then the student has little hope of being able to mimic the expert's decisions. As we show in our next example, even when optimizing perfectly, depending on the choice of f and foup, IL may result in m being uniformly random over large collection of states. We call the phenomenon that Il A π ees the imitation gap.
Example 2 (ID-Lighthouse). We illustrate the imitation gap using a gridworld spanning [-N,... ....... .]. The two start states correspond to the goal at -N or N, while the agent is al- ways initialized at 0 (see Fig. 2a Clearly, with full state information, nTP maps states to an 'always left' or 'always right' probability distribution, depending on whether the goal on the left right, respectively. Suppose now that the agent's visibility is constrained to a radius of i (Fig. 2c shows i = 1), i.e., an /'-partial observation is accessible. An agent following an optimal policy with visibility of radius i will begin to move deterministically towards any corner, w.l.o.g. assume right. When the agent sees the rightmost edge (from position N - i), will either continue to move right if the goal is visible or, if it's not, move left until it reaches the goal (at -N). Now we may ask: what the best f' fa-partial policy that can be learnt by imitating cxP (i.e., what is x//l Tragically, the cross-entrops loss causes THE to be uniform in a large number states. In particular, an agent following policy ThE will take left (and right) with probability 0.5, until is within a distance of i from one of the corners. Subsequently, it will head directly to the goal. See the policies highlighted The intuition for this result is straightforward: until the agent observes one of the corners it cannot know if the goal is to the right or left and, conditional on its observations, each of these events is equally likely under μ (assumed uniform). Hence for half of these events the expert will instruct the agent to go right. For the other half the instruction is to go left. See App. A.I for a rigorous treatment of this example. In Sec. 4 and Fig. 5 train f ---aattial policies with opttimll experts for a 2D variant of this example. We empirically verify that a student learns a better policy when imitating teachers whose filtration function is closest to their own.
in Figs. 2c 2
4

---

Under review as a conference paper at ICLR 2021
The above example shows: when student attempts to imitate an expert that is privileged with information not available to the student, the student learns a version of Th in which this privileged information is marginalized out. We formalize this intuition in the following proposition.
Proposition 1 (Policy Averaging). In the setting of Sectiin suppose that I contains all f- partial policies. Then, for any 8 with f()), we have that (0) (- [(5) o|.
Proofs are deferred to Appendix A.2
The imitation gap provides theoretical justification for the common practical observation that an agent trained via IL can often be significantly improved by continuing train the agent using pure RL (e.g., PPO) (Lowe et al. [020]] Das et af.) 2018b). Obviously training first with IL. and then via pure RL techniques is ad hoc and potentially sub-optimal
discussed in Ex. Ix.[aand empirically shown in Sec. 4 To alleviate this problem, the student should imitate the expert policy only in settings in which the expert policy can, in principle, be exactly reproduced by the student. Otherwise the student should learn via 'standard' RL methods. To this end, we introduce ADVISOR.
3.2 ADAPTIVE INSUBORDINATION (ADVISOR) WITH POLICY GRADIENTS
To close the imitation gap. ADVISOR adaptively weights reward-based and imitation losses, Intu- itively, it supervises student to imitate an expert policy only in those states E S for which the imitation gap small. For all other states, it trains the student using reward-bases based RI. To simplify notation, denote the reward- based RL loss via [,,[L(( S)] for some loss function L This formulation is general and spans all policy gradient methods, including A2C and PPO. The imitatiin loss is the standard cross-entropy loss E, EC ( (S)) x (S;0)) Concretely, ADVISOR loss is:
This loss
ADDV (0) = (w)((()) CE(T (S), (S; ()) + (1 - w()) L(O, S)]
(2)
Our goal is to find weight function w: S x θ + [0, 1] where w(s) Rs 1 when the imitation gap is small and w()) Rs 0 otherwise. For this we need an estimator of the distance between xa and x at a state s and a mapping from this distance to weights.
We now define d (TT, x (s), a distance estimate between a policy π and an f-partial policy T at a state s. We can use any common non-negativy distance (or divergence) d between probability distributions on A, e.g., the KL-divergence (which we use in our experiments). While there are many possible strategies for using dl to estimate d' (TT, (s),, perhaps the simplest of these strategies to define d (s) d( (s)). Note that this quantity does not attempt to use any information about the fiber (f()s which may be useful in producing more holistic measures of 2 distances Appendix A.3 considers how those distances can be used in lieu of dº, Next, using the above, we need estimate the quantity (1 (s).
Unfortunately it is, in general, impossible to compute (~<T,,,, (s) exactly as it is intractable to compute the optimal minimizer n.f. Instead we leverage an estimator of TT which we term mrxx, and will define in the next section.
Given T.th we obtain the estimator d' (TTTP, , m.) of d ( , ..) Additionally, we make use of the monotonically decreasing function ma,8 Rzo + [0, 1], where a, B 2 0. We define our weight function wxs) for E S as:
w(s) = ma.9 (d' ( a (s))
with
(3)
(()) = e-eao az 1(050))
(4)
Together Eq. 2 define ADVISOR.
FFor readability, we implicitly make three key simplifications. First, computing the expectation Ex is generally intractable, hence we cannot directly minimize losses such as F,, S)]. Instead, approximate the expectation using rollouts from js and optimize the empirical loss. Second, recent RI. methods adjust the measure ya over states as optimization progresses while we assume it to static for simplicity. Our final simplification regards the degree to which any loss can be, and is, optimized. In general, losses are often optimized by gradient descent and generally no guarantees are given that the global optimum can be found. Extending our presentation encompass these issues is straightforward but notationally dense.
2 *Measures using such information include max, e (s(() d(r((s)) (s'',, J,my(s) (s)) or a corresponding expectation instead of the maximization, i.e., F,, (S() n, (S)) f(S) = o].
5

---

Under review as a conference paper at ICLR 2021
New_partial ial observation
Action
Env
sonplle
Actor Cg
(sy
Agent's viow
ADVISOR Iooe
egocentziee
Hi DE His RE
Cornect
Encoder
LSTM
Critic Uy
Value
Wall ebstacles
PPO
f(e) partial observation
Aus- Actur
(s; 0*)
Goal
lppert
Og"
CB
(a) WALSCROSSING
(b) LAVACROSSING
(c) Model overview
Figure 3: MINIGRID base tasks and model overview. (a) WC: Navigation with wall obstacles,
with additional expert and environmental challenges. We test up-to 25 x 25 grids with 10 walls. (b) LC: Safe navigation, avoiding lethal lava rivers. We test up-to 15 x 15 grids with 10 lava rivers.
(c) An auxiliary actor is added and trained only using IL. The 'main' actor policy is trained using the
ADVISOR loss defined in Section 3.2 3.3
3.3 THE AUXILIARY POLICY π AUX; ESTIMATING T.. IN PRACTICE
In this section we describe how we can, during training, obtain an auxiliary policy xy which estimates 7.. Given this auxiliary policy we estimate (=<',,,,,))) (s) using the plug-in estimator ( (s). While plug-in estimators
intuitive and simple to define, they need not be statistically efficient. In Appendix A.4 we consider possible strategies for improving the statistical efficiency of our plug-in estimator via prospective estimation.
In Fig. 3c we provide an overview of how we compute the estimator n nhx via deep nets. As is common practice (Mnih et al.) 2016 Heess et al. 2017 Jaderberg et al. 2017: Pathak et al. 2017 Mirowski -eevalier-Boiveettt Boisvert al., 2018a), the policy net x ( 8) composed via a, P with V. A), where a, is the actor head (possibly complemented in actor-critic models by critic head v.) and rx is called the representation network. Generally, a is lightweight, for instance linear layer or shallow MIP followed by soft-max function. Instead, yx is a deep and possibly recurrent neural net. We add another actor head ay to our existing network which shares the underlying representation тX, i.e., F ay r'x As both actors share the representation yA they benefit from any mutual learning. While our instantiation covers most use-cases, ADVISOR can be extended to estimating . via two separate networks, i.e., = (v', X). In practice we train Ty 0) and mux (-; 0) simultancously using stochastic gradient descent, as summarized Algorithm A.1
al. 2017] Chevalier
4 EXPERIMENTS
We rigorously compare ADVISOR to IL methods, RL methods and their popularly-adopted (yet ad hoc) combinations. In particular, we evaluate 14 methods. We do this over ten tasks realizations of Ex. T & Ex. 2 and eight navigational tasks of varying complexity within the fast, versatile MINIGRID environment
al., 2018a b). Furthermore, for robustness, we train 50 hyperparameter variants for complex tasks. For all tasks, we find ADVISOR- based methods outperform or match performance of all baselines. In a final study, we train agents to complete the PointGoal Navigation task in the visually rich AIHABITAT environment, In this task we compare ADVISOR to prior work (using PPO) and pure-IL IIL baselines.
Sitevalier-oiswwee
4.1 TASKS
Succinct descriptions of our tasks follow. Experts always take globally optimal actions. We defer further details and description of experts to Appendix A.5]
POISONEDDOORS (PD). As defined in Ex. [ in Sec. with N = 4, 10, see Fig. I The agent's ' observation is integer identifying the agent's s state: (1) the agent must pick door, (2) the agent has picked the first door and has either not started entering a code or has just previously correct code input. The agent's 's action space includes action for each door (used to pick among them) and three additional actions corresponding input code values (only operational after choosing door 1).
entered wrong input, (3) the agent has picked the first door and its last action

---

Under review as a conference paper at ICLR 2021
PoIsoNEnDoORS
LAVACROSSING (LC) 13.24 JABY
LC ONCE SWITCH
WC CORRUPT
Previous methods
11. ADV
10
ADV
ADV
ouly
[8 B[eewardd)
0.00
055
IL-RI. HL-RI PPO
0.23
BC PPG
0:01
0.14
0.3
BC PPCp
PRO
-1:35
-0.90
0.2
0.30
112
Hused on cotonntaations DC
BC-7000 (ears)
10. 30t PPO
-825
110
ADVISOR (ours)
# training TUES
# training cure
training rins
Lraining runs
AIV
12: 1- ABV
(a)
(b)
(c)
(d)
EC ADV
AUV 2913
Figure 4: Evaluation following (Dogge et al. 2019). As described in Section 4.3 we plot expected validation reward of best-found model (y-axis) over an increasing budget of # training runs, each with random hyperparameter values (x-axis). Clearly, larger E[Reward] with fewer # training runs is better. We mark the best performing (sthod(s) at the top of each plot.
WALLCROSSING/LAAACRSOS
3b an agent is tasked to (WC/LC). As illustrated in Fig.
3a
navigate to the goal using local observations. In doing so, it must avoid walls deadly (i.e., episode- ending) rivers of lava. Evidently, imitating a shortest-path expert is easy, requiring no exploration beyond expert-visited states. Hence, we consider more challenging variants of WC and L.C tasks. For all MINIGRID environments, agents observe a 7 7 7 tensor corresponding to the area just in front of the agent. For our tasks only 3 actions are relevant (move ahead, rotate left, and rotate right). SWITCH. The agent is initialized in a WC or LC environment with the "lights turned off." The agent can use an additional switch action to get unaffected observations, whereas the shortest-path expert can navigate in the dark. Hence, the expert doesn't 't supervise taking the new action. For both WC and LC base tasks, we experiment with two switches: (1) lights stay on after using the switch action (ONCE), or (2) light turn only for a single timestep (FAULTY).
CORRUPT. To evaluate resilience of methods to a corrupted expert. In every episode the expert produces correct actions until it is within No steps of the target, after which it outputs random actions. 2D-LIGHTHOUSE (2D-LH). A harder, 2D variant of the gridworld task introduced in Ex. 2 Agents act by moving in cardinal direction and observe some fixed radius about themselves.
4.2 BASELINES AND ADVISOR-BASED METHODS
Expert supervision comes in two forms: (a) expert policy, (b) dataset of expert demonstrations. We study baselines and ADVISOR in both these forms. For (a), we include IL baselines with different levels of teacher-foreing (tf): tf=0, if annealed from 1-+0, and tf=1. This leads to Behaviour Cloning (BC), Data Aggregation (DAgger, t), and BC respectively. Also, we implement pure RI. (PPO) which learns only on the sparse rewards. Furthermore, we implement popular sequential hybrids such as BC then PPO (BC-PPO)) DAgger then PPO (f PPO), BC + PPO, and a parallel combination of BC PPO(static) This is static variant of adaptive combination ADVISOR (ADV). We introduce hybrids such as DAgger then ADVISOR
(t + ADV), and Bf - ADV. For (b), agents imitate expert demonstrations and hence get no supervision beyond the states in the demonstrations. This leads to BCdomo and its combination with PPO dem + PPO). We introduce a corresponding AD + PPO, applying ADVISOR on expert demonstrations while training PPO on on-policy rollouts (see App. A. for details). Further details of all methods are in Appendix A.6 For fairness, the same model architecture is shared across We defer implementation details to Appendix A.7
all methods (recall Fig. 3c Sec. 3.3
4.3 EVALUATION
Fair Hyperparameter Tuning. Often unintentionally done, extensively tuning the hyperpa- rameters (hps) of a proposed method and not those of the baselines can introduce unfair bias into evaluations. We avoid this by considering two strategies. For PD and all MINIGRID
7

---

Under review as a conference paper at ICLR 2021
View Radius
5
9
9
116
119
116
11
Arg Ep Length
Avg Ep Lengrh
Avg Ep Longth
Ang Tp Length
(a) Behavior Cloning
(b) ADV
(c) PPO
(d) PPO
Figure 5: *Less intelligent" teachers. Learning p'paattial policies using opttiia experts 2D-LH.
tasks, we follow recent best practices (Dodge et al. 2019 Namely, we tune each method by randomly sampling a fixed number of hps and reporting, for each baseline, an estimate of E|Val. reward of best model when allowed a budget of random hps] for < k: S 45. For this we must train 50 models per method, i.e., 700 for each of these nine tasks. More details in Appendix(A.8 For 2D-LH, we tune the hps of a competing method and use these hps for all other methods.
Training. For the eight MINIGRID tasks, train each of the 50 training runs for million steps. For 2D-LH/PD, models saturate much before 10 steps (details are in Appendix (A.9).
Metries. We record avg. rewards, episode lengths, and success rates. In the following, we report subset of these recorded values. Additional plots can be found in Appendix A.11
4.4 RESULTS
Before delving into task-specific analysis, we state three overall trends. First, for tasks where best-performing baselines require querying the expert policy during training, ADV significantly outperforms all methods. These are the more difficult tasks which need exploration, such as PD and SWITCH. Second, for tasks where the best-performing baselines
demonstrations- based, PPO improves over or matches previous methods. Third, a head-on comparison of BC + PPO(static) and ADV shows that our dynamic weighting approach is superior across all tasks.
ADVdemo
PD. This environment was designed to be adversarial to standard imitation-learnin approaches and so it is not surprising, see Fig. 4a that models trained using standard IL techniques (DAgger, BC, BC perform poorly. Qualitatively, they attempt open door 1 with low probability and thus obtain an expected reward near -2/3. Baselines that learn from rewards, e.g., PPO, can learn to avoid doors 2-4 but in practice cannot learn the combination open the first door. This results in an average reward of O. Notice that warm-starting with any form of IL is actively harmful: egg., it takes many hyperparameter evaluations before we consistently find a DAgger- model that reproduces the performance of a plain PPO model. Finally, only our ADV method consistently produces high quality models (avg. reward approaching 1).
LC. The vanilla LC setting reveals that the imitation gap between the expert and the agent obser- vations nominal. Intuitively, the agent's egocentric partial observations are sufficient to learn to replicate the shortest-paie expert actions: it need only follow alleys of safe ground leading to the narrow gaps in otherwise continuous obstacles. This is validated by the high performance of BC (and BC which learn only from expert trajectories and have no hope of bridging an imitation gap. Methods learning from demonstrations and RL (e.g., ADVdemm PPO and de PPPO perform only marginally better. Thus, when the imitation gap is small and BC from demonstrations highly successful, we conclude that the gains from using ADVISOR- methods may be marginal.
LC SWITCH. IL and warm-started methods receive no supervision to explore the switch action and thus learning in the dark resulting in poor policies, see Fig 4c Also, early episode termination when agents encounter lava prevents PPO success due to sparse rewards. ADVISOR leverages it's RL loss to learn to 'wwitch' on lights after which it successfully imitates the expert.
WC CORRUPT.
In Fig. 4d we investigate ADVISOR's ability to learn to ignore corrupted expert. While this is not what ADVISOR was designed for, it is interesting to see that ADV-based methods are able to accomplish this task and do significantly better than the best performing competitor (BC+PPP)). This suggests that ADVISOR is robust to expert failure.
"See also reprodueibility checklist in EMNLP'22 20 CfP: httrss///000..mmpp.orr///////
8

---

Under review as a conference paper at ICLR 2021
2D-LH. Here vary the privilege of expert and study learning from "less intelligent" teachers. Particularly, for each method, we train an f" p'-partial policy using an fl-optimal expert (except for PPO which uses no expert supervision) 25 times. Each policy is then evaluated 200 random episodes and the average episode length (lower heing better) is recorded. For all odd i with S i < 11, 1 S j S 15, and j 2 i we show boxplots of the 25 training runs. Grey vertical lines show optimal average episode lengths for f'-partial policies.
For BC we find training of an pa-pttia policy with an fl-expert to result in near optimal policy when = j but even small increases in j result in dramatic decreases in performance. This emphasizes the imitation gap. Surprisingly, while performance tends to drop with increasing j. the largest j. gaps do not consistently correspond to the worst performing models. While this seems to differ from our results Ex. 2 recall that there the policy js was fixed while here it varies through training, resulting in complex learning dynamics.
Additionally, we find that: (i) PPO can perform well but has high variance across runs due to the problem of sparse rewards, and (ii) for this task, both ADVISOR and DAgger can ameliorate the impact of the imitation gap but ADVISOR consistently outperforms in all settings.
4.5 POINTGOAL NAVIGATION IN VISUALLY-RICH 3D ENVIRONMENTS
To highlight possible applications of ADVISOR to complex, visually rich, environments, we consider the PointGoal navigation (PointNav) task within AlHABITAT (Savva et al. 2019). We briefly describe the task, followed by the methods, and discuss results.
Task. In PointNav, a randomly spawned agent must nav- igate to goal specified by a relaive-diisplaemmnnt vector. The observation space is composed of rich egocentric RGB observations (256x (556x266x3) with a limited field of view. The action space is (move_ahead, rotate_ri ght, rotate lefft, a op ). The task was formulated by An derson et al. 2018)) and implemented for the AIHABITAT simulator by Savva et al. (2019). Our reward structure, train/val/test splits, PointNav dataset, and implementa- tior f follow Savva et al.] (2119). To better understand sample-efficiene we report the primary metric adopted to evaluate PointNav, i.e., success weighted by path length (SPL), on the validation set at different points of training. We train on the standard Gibson set of 76 scenes, and report metrics as an average over the val. set consisting of 14 unseen scenes in AIHABITAT. We use a budget of 25 Mn frames, i.e., ~ days of training on 4 NVIDIA TitanX GPUs, and 28 CPUs for each method.
Sara etal ICCV 2019
0.4
0.3
0.2
Only
Only (LPP proc)
0.1
Only RL (PPO w pro))
Only RL 0PP w/ 16 proc.)
10
25
RGB frames (Million)
SPL.
SPL.
SPL
Methods
@5t
@ 10
@ @55
PPO (3 runs)
0.307
0.404
0.468
BC
0.259
0.505
0.649
ADVISOR
0.579
0.685
0.741
Figure 6: PointNav in AIHABITAT.
Methods. Letho et al. (2019) train their RGB agents using PPO. We train pure IL (on-policy behavior cloning) using the optimal shortest-palt path action. The pure IL agent observes a filtered egocentric observation while the expert has access to the full env. state. We also train on on-policy ADVISOR.
Results. Fig. summarizes the results for the above methods. Despite being extensively tuned, we found that the training configuration released with habitat-lab is susceptible to small changes in the number of processes that collect data for PPO optimization. In contrast, for pure IL and ADVISOR baselines we don't tune hyper-parameters, i.e., we use exactly the same as the released configuration. We mark an SPL of 0.46 via a dashed line, which was the result reported by [Savva al. (2019) for an RGB agent on the Gibson dataset. ADVISOR substantially outperforms other methods.
5 CONCLUSION
We introduce the imitation gap as explanation for the empirical observation that imitating "more intelligent" teachers can lead to worse policies. While prior work has, implicitly, attempted to bridge this imitation gap, we introduce principled adaptive weighting technique (ADVISOR), which we test on a suite of ten tasks. Due to the fast rendering speed of MINIGRID, PD and 2D-LH, we could undertake study where we trained over 6 billion steps, to draw statistically significant inferences.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Peter Anderson, Angel Chang. Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents, arXiv preprint arXiv: 1881v:1807.06777 2018.
Michael Bain and Claude Sammut. A framework for behavioural cloning. Machine Intelligence, 1995.
Marc G Bellemare, Georg Ostrovski, Arthur Guez, Philip Thomas, and Rémi Munos. Increasing the action gap: New operators for reinforcement learning. In AAAI, 2016.
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E. Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In Qiang Yang and Michael J. Wooldridge (eds.), Proceedings of the Twenty- International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pp. 3352-3358. AAAI Press, 2015. URL http: /ttp::://ijaa..../ otg.iiaaaaaaao...oooooo.ooooeeee www.oo/g//t.tttttt....5s
Thomas Cederborg, Ishaan Grover, Charles L. Isbell Jr., and Andrea Lockerd Thomaz. Policy shaping with human teachers. In Qiang Yang and Michael J. Wooldridge (eds.), Proceedings of the Twenty-Fourd Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pp. 3366-3372. AAAI Press, 2015. URL http: /ttp///aaa../////// org/ Abstract/155477
Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daume, and John Langford. Learning to search better than your teacher. ICMI., 2015.
Jessica Chemali and Alessandro Lazaric. Direct policy iteration with demonstrations. In Qiang Yang and Michael J. Wooldridge (eds.), Proceedings of the Twenty-Fourlh International Joint Conference Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pp. 3380-3386. AAAI Press, 2015. URL http:: http:://ffaaioooog/ www.liorg//bbbaaaaee Abs.tbattttatt///////aa
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühh. Learning by cheating. In Conference on Robot Learning, 2020.
Maxime Bhoiser Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. In ICLR, 2018a.
Maxime Chevalier-Boisvert Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. /tttps://gite
com/ http///axxiiccc/e//mmmmm gym- minigrid, 2018b.
Felipe Codevilla, Matthias Müller, Antonio M. López, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brishane, Australia, May 2 21-25, 2018, pp. 1-9. IEEE, 2018. doi: 10.1109/ICR.2011 8.4444447 /tLpttts:://oi.cooo http://o..ooooooooo wtt.:////.......1...111111 1109/ICRA, 2018 8460487
A. Das, S. Datta, G. Gkioxari, Lee, D. Parikh, and D. Batra, Embodied Question Answering. In CVPR, 2018a.
A. Das, G. Gkioxari, S. Lee, D. Parikh, and D. Batra. Neural Modular Control for Embodied Question Answering. In CoRL, 2018b.
Abhishek Das, Satwik Kottur, José M.F. Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual dialog agents with deep reinforcement learning. In ICCV, 2017.
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In EMNLP, 2019.
Tanmay Gangwani and Jian Peng. State-only imitation with transition dynamics mismatch. In ICLR, 2020.
10

---

Under review as a conference paper at ICLR 2021
Tanmay Gangwani, Joel Lehman, Qiang Liu, and Jian Peng. Learning belief representations for imitation learning in pomdps. In Amir Globerson and Ricardo Silva (eds.). Proceedings of the Thirty-Fifth Conference Uncertainty in Artificial Intelligence, UAI 2019, Tel Avh; Israel, July 22- 25, 2019, pp. 383. AUAI Press, 2019. URLLhttp:///uuee www.aaaaiii.rraa..o.....a 201 w/wpoceddinggggeeeeeoo ngs/ 2apers/383.pom
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. In ICL.R, 2017a.
S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik. Cognitive Mapping and Planning for Visual Navigation. In CVPR, 2017b.
Jeffrey Hawke, Richard Shen, Corina Gurau, Siddharth Sharma, Daniele Reda, Nikolay Nikolov, Przemyslaw Mazur, Sean Micklethwaite, Nicolas Griffiths, Amar Shah, and Alex Kendall. Urban driving with conditional imitation learning. In 2020 IEEE International Conference on Robotics and Automation, ICRA 2020, Paris, France, May 31 August 31, 2020, pp. 251-257. IEEE, 2020. doi: 2020.. 9197408
URL ht / http:/oo.ooo..oooooo 10. 1 11911114441/191994444444
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei- Vei-Ying Ma. Dual learning for machine translation. In NeurIPS, 2016.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang. SM Eslami, et Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: 7707.02286, 2017.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In AAAI, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NeurlPS, 2016.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICLR, 2017.
Nan Jiang. On value functions and the agent-envinonment boundary. arXiv preprint 19X5v:1905.133: 2019.
Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Chao Yang, Bin Fang, and Huaping Liu. Reinforcement learning from imperfect demonstrations under soft expert guidance. The Thirty- Artificial Intelligence, AAAI 2020, The Thirty-Seconte Innovative Applications of Artificial Intelligence Conference. IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, FAAI 2020, New York, NY, USA, February 7-12, 2020, pp- 5109-5116 16. AAAI Press, 2020. URL
Fourth AAAI Conference
o ootg////////xxx.....///o/////... php/
BAA//articee/view/555
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations, In 2018.
2018.
D. Kingma and J. Ba. method for stochastic optimization. In CVPR, 2017.
Jens Kober and Jan R Peters. Policy search for motor primitives in robotics. In NeurlPS, 2009.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: an interactive 3d environment for visual AI. arXiv preprint /
2019.
Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudik, Yisong Yue, and Hal Daumé. Hierarchical imitation and reinforcement learning. In 2018.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. JMLR, 2016.
11

---

Under review as a conference paper at ICLR 2021
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
Ryan Lowe, Abhinav Gupta, Jakob N. Foerster, Douwe Kiela, and Joelle Pineau. On the interaction between supervision and self-play in emergent communication. In ICLR, 2020.
A Rupam Mahmood, Hado P van Hasselt, and Richard S Sutton. Weighted importance sampling for off-policy learning with linear function approximation. In NeurlPS, 2014.
P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, al. Learning navigate in complex environments. In ICLR, 2017.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattic, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforce- ment learning. Nature, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ICML, 2016.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Overcom- ing exploration in reinforcement learning with demonstrations. In ICRA, 2018
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learming. Foundations and Trends in Robotics, 2018.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In JCML, 2017.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based baasse character skills. ACM Trans. Graph., 2018.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 2008
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 1991.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In RSS, 2018.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In AISTATS, 2010.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, 2011.
Claude Sammut, Scott Hurst, Dana Kedzier, and Donald Michie. Learning fly. In Machine Learning Proceedings, 1992.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: Platform for Embodied AI Research. In ICCV, 2019.
T. Schaul, J. Quan, Antonoglou, and D. Silver. Prioritized Experience Replay. In ICLR, 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In /CML,, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 1506: 02438, 2015b.
12

---

Under review as a conference paper at ICLR 2021
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1007.06347, 2017.
Tianlin Tim Shi, Andrej Karpathy, Linxi Jim Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In ICML, 2017.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Mottaghi, Luke Zettlemoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. CVPR, 2020.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanetot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 2016.
Kaushik Subramanian, Charles L Isbell Jr, and Andrea Thomaz. Exploration from demonstration for interactive reinforcement learning. In AAMAS, 2016.
Wen Sun, J. Andrew Bagnell, and Byron Boots. Truncated horizon policy search:
Combining rein- forcement learning imitation learning. 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL http::://opeeeeeisiiieee http://..e.e.....oeooeeoee/eoeo http://tttoommmdddmtltlaaaee
Mark van der Laan and Susan Gruber. One-step targeted minimum loss-based estimation based on universal least favorable one-dimensional submodels. The international journal of biostatistics, 12 (1):351-378, 2016.
A.W. van der Vaart. Asymptotic Statistics. Press, 2000. ISBN 9780521784504. URL. hhttps://boo/ .eraggooolee... e. com/books?id
Asymptotic Statistics. Cambridge University
UELQEM5R (WqC
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q learning. AAAI, 2016.
Matej Vecerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Manfred Otto Heess, Thomas Rothörl, Thomas Lampe, and Martin A. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. ArXiv, abs/1707.08817, 2017.
Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang. Video captioning via hierarchical reinforcement learning. In CVPR, 2018
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. In ICLR, 2017.
Larry Wasserman. All of Nonparametric Statistics (Springer Texts in Statistics). Springer-Verlag. Verlag,
Berlin, Heidelberg, 2006. ISBN 0387251456.
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, and Nicolas Heess. Reinforcement and imitation learning for diverse visuomotor skills. In Proceedings of Robotics: Science and Systems, 2018.
13

---

Under review as a conference paper at ICLR 2021
APPENDIX: BRIDGING THE IMITATION GAP BY ADAPTIVE INSUBORDINATION
The appendix includes theoretical extensions of ideas presented in the main paper and details of empirical analysis. We structure the appendix into the following subsections:
A.
A formal treatment of Ex. 2 on ID-Lighthouse.
Proof of Proposition
T
Distance measures beyond (x) = (T)( utilized ADVISOR F
Future strategies for improving statistical efficiency of dan (s) estimator and a prospec- tive approach towards it.
Descriptions of all the tasks that evaluate baselines on, including values for grid size, obstacles, corruption distance etc. We also include details about observation space for each of these tasks.
A.6
Additional details about nature of learning, expert supervision and hyperparameters searched for each baseline introduced Sec.
Sec.4.2 4.2
Details about the underlying model architecture for all baselines across different tasks.
A
Methodologies adopted for ensuring fair hyperparameter tuning of previous baselines when comparing ADVISOR to them.
Training implementation including maximum steps per episode, reward structure and com- puting infrastructure adopted for this work. We clearly summarize all structural and training hyperparameters for better reproducibility.
Additional plots for all tasks to supplement Fig. 4 We include walk- talk-through of Fig. i.e. expected maximum validation performance plots.
Implementation details for the PointNav task on the AlHabitat platform.
A ADDITIONAL INFORMATION
A.1 FORMAL TREATMENT OF EXAMPLE
Let N 2 and consider 1-dimensional grid-world with states S (-N,N) {0, .o. T x
(-N...., .NT.. Here E (NN, N} are possible goal positions, elements t E fo...., TT corre- spond to the episode's 's current timestep, and (p:) E (-N...... ...... trajectories of length T. Taking action E A= (left, right) (-1.1} in state (g.t, (p.)T= results in the deterministic transition to state (g. t 1, (Pi...., Pr. clip(p: + a, N,, 0)). An episode start state is chosen uniformly random from the set [(+N, (0, .., 0)) and the goal of the agent is to reach some state (g, 1, (p.){_)) with p. in the fewest steps possible. We now consider collection of filtration functions f", that allow the agent to spaces up i steps left/right
..........J.. .N7 correspond to possible agent
of its current position but otherwise has perfect memory of its actions. See Figs, 2c 2d for examples of f-- and -a-pariial observations. For 0 < i N we define I' so that
((lo ((fo...., (e), (p Po. Po.
....---PP-- PP Pt- and
(5)
E = (1 (ppp [p:+ [ |k
i)) for
(6)
Here l is tuple of length 2 i + and corresponds to the agent's view at timestep j while Pk-1 pk uniquely identifies the action taken by the agent at timestep k. Let n be the optimal policy given full state information so that (0 t, (p<){=) = (4g==-N), Igg=N() and let μ be a uniform distribution over states in S. It is straightforward show that an agent following policy Tht will
take random actions until it is within a distance of i from one of the corners { (-N,N) after which it will head directly to the goal, see the policies highlighted in Figs. 2c 2d The intuition for this
result is straightforward: until the agent observes one of the corners it cannot know if the goal is to the right or left and, conditional on its observations, each of these events is equally likely under u. Hence in half of these events the expert will instruct the agent to go right and in the other half to go left. The cross entropy loss will thus force T/I be uniform in all such states. Formally.
'We overload main paper's notation d (TT, (8) with d (8)
14

---

Under review as a conference paper at ICLR 2021
Algorithm A.1: On-policy ADVISOR algorithm overview. Some details omitted for clarity.
Input:
Trainable policies (T mux expert policy x<P, rollout length L, environment E.
Output:
Trained policy
begin
Initialize the environment
8 randomly initialized parameters
while Training completion criterion not met do
Take L. steps in the environment using y 0) and record resulting rewards and
observations (restarting & whenever the agent has reached a terminal state)
Evaluate " (-;0) and STP each of the above steps
L the empirical version of the loss from Eq. ( computed using the above rollout
Compute V L using backpropagation
Update 9 using VoL via gradient descent
return x (+;00)
10
(p:)(=)), T (s) (s) if and only if minos S - N will have, for (g,t, maxosgs((Pg) + and, for all other s, we have ( = 1122, Sec. see also Fig. 5 train f partial policies with f'-optimal experts for 2D variant of this example.
A.2 PROOF OF PROPOSITION
We wish to show that the minimizer of 1, Je (S) ④ log F (S)) among all f-partial policies x is the policy π E, (S)) f(S)]. This is straightforward, by the law of iterated expectations and as T y (s) = Ty (f)))) by definition. We obtain
Ep1- (S) @ log (S)) -Ep [Epppp a exp (S) O log T. ( | [(S)]]
=-Ep EEplEple im por (S) O log M ( f(S)]
f
EE (T exp (S) [(S) @ 1y
(E()- @ log (ogry(((S)))
(7)
Now let E S and let o fs). ) well known, by Gibbs' inequality, that (T( @ log (o) is minimized (in (o)) by letting TT #(o) and this minimizer is feasible as have assumed that , contains all f f-partial policies. Hence it follows immediately that Eq. 7 is minimized by letting Tf which proves the claimed proposition.
A.3 OTHER DISTANCE MEASURES
As discussed in Section 3.2 there several different choices may make when choosing measure distance between the expert policy -h and an f p-rrtiill policy T at a state s E S. The measure of distance we use in our experiments, (Ty) = d(n exp (s), f has the (potentially) undesirable property that f(s) = f(s') does imply that du..p (s) di.., (s'). While an in-depth evaluation of the merits of different distance measures is beyond this current work, we suspect that careful choice of such a distance measure may have substantial impact on the speed of training. The following proposition lists collection of possible distance measures with a conceptual illustration given in Fig. A.
Proposition Let and f(s) and for any < < oo define, for any policy and f-partial policy f.
(.)((()))s) ((((((()))) 1f(S)
f(5)
with d (s) equalling the essential supremum of di (TT) under the conditional distribution
Pa(: f(S) = f())). As a special case note that
(-) = E f(S) = f(s)).
15

---

Under review as a conference paper at ICLR 2021
Then, for all B 2 0 and E S (almost surety (s), have that x(s) + mf({(s)) if and onty if (s)) > O.
Proof. This statement follows trivially from the definition of Th and the fact that d(T, x²) 2 0 with d(T, x²) 0 if and only if π n².
The above proposition shows that any d' can be used to consistently detect differences between x and n.. i.e., it can be used to detect the imitation gap. Notice also that for any B we have that oo (1 (s) To (T)) (s') whenever f(s) = f(s').
As an alternative using d, now describe how d (z))) can be estimated in practice during training. Let T be an estimator of z as usual. To estimate d) de (s) we assume we have access to a function approximator Ge : OO - IR parameterized by t/ E V, e-g., neural network. Then we estimate no (x)) (s) with 93 where up is taken to be the minimizer of the loss
sy (1b) En (d((T (
(J(S)) gg 9.
(9)
The following proposition then shows that, assuming that dt d equal xo (m)) and thus 93 may be interpreted as a plug-in estimator of di pore (17)) Proposition For any db E V,
g6 will (m) E {op vb E v),
11 (TT Cuavvapp (b) = E.p(dd (TT - Iy (f(S)))
where c = En(((((=PPS)) n (J(S))
= is constant in up and this implies that if dy (mT(tt E {ow vb E 3 then 96 = dn. v0p (TT(er).
Proof. In the following we let O = =(()). We now have that
E, -p (())) -,)))))))))))))) (0,))
E || (O₂)) - da (TT (S)) + (d) m (T. 9())))
E, (S), 5 (0₂)) oop (TT QUX
(1 E, [ct [(d) N (TT (TT (S) g (0
2.
(S), `
a0x (0₂)) dz tou (T aux (S)) (dd ( ( ( - (0,)))
20x
= =c+ Fp ((d) N, ( mIx (S) - 90 ( (
+ [x[[[((((TT @ (S), f 20x (0,)) d - (m))mr) (S)) (a ( -
(s') for any 8,8' with f(s) = f(s') we have that dy 00 (TTT gv (0,) 0) is constant conditional O, and thus
Now as as dh dup f mux (s) =
E [ m. (0₂)) - d p (TTH) (S)) (d) (o (-))t))))))) (S) (w() O,
Pl]((((= cxp (S), / (0₂)) dh m (x) Oy] F. ( (m) (S) g (0p) |0,]
F [dh (x 2ux (S) = (1 mux (S) Oyl Fp n (xx) (S) (0,) Oyl
O.
Combining the above results and using the law of iterated expectations gives the desired result.
A.4 FUTURE DIRECTIONS IN IMPROVING DISTANCE ESTIMATORS
In this section we highlight possible directions towards improving the estimation of &e (s) for E S. As comprehensive study of these directions is beyond the scope of this work, our aim in this section is intuition over formality. We will focus on d here but similar ideas can be extended to other distance measures, e-g-, those in Sec. A.3
16

---

Under review as a conference paper at ICLR 2021
14
PL 1/00)) 17000-0 (100 m000000
(6
(p)
P2 (t.
L ( ((d)
pa 4 167
a ((0))
(
(a) ((())) (,
(d)
Figure A.1: Concept Illustration. Here we illustrate several of the concepts from our paper. Suppose our action space A contains three elements. Then for any & E S and policy n, the value (()) can be represented as a single point in the 2 2-dimensional probability simplex {(x, v) E R² x 2 0, 2 0, + < } shown as the grey area in (a). Suppose that the fiber f (f) contains the three unique states s² . and s². In (a) we show the hypothetical values of TA when evaluated at these points. Proposition 1 says that x (s) lies in the convex hull of {s*)) visualized as a magenta triangle in (a). Exactly where n (s) lies depends on the probability measure Jr, in (b) we show how a particular instantiation of may result in realization of x (s) (not to scale), (c) shows how da measures the distance between (s") and (s²). Notice that it ignores s² and s In (d), we illustrate how &eesp produces a "smoothed" measure of distance incorporating information about all s².
As discussed in the main paper, estimate &row (m (s) by first estimating n.l with Tht and then forming the "plug-in" estimator areo (TT (TTII))s) (s). For brevity, we will write dia. (xJjes (s) as d. While such plug-in estimators are easy to estimate and conceptually compelling, they need not be statistically efficient. Intuitively, the reason for this behavior is because we are spending too much effort in trying to create a high quality estimate The of nt! when we should be willing to sacrifice some of this quality in service of obtaining a better estimate of dd..p (TT) (s)) Very general work in estimation (TMLE) (van der Laan &
this area has brought about the targeted Gruber 2016) framework. Similar ideas may be fruitful in improving our estimator d.
Another weakness d discussed in the main paper is that is not prospective. In the main paper we assume, for readability, that we have trained the estimator Thjut before we train our main policy. In practice, we train TT. alongside our main policy. Thus the quality of 1h? will improve throughout training. To clarify, suppose that, for i E [0, 1). aux is our estimate of xt after (100 1)% of training has completed. Now suppose that (100 t)% of training has completed and we wish to update our main policy using the ADVISOR loss given in Eq. 2.. In current approach we estimate daop II (s) using ( (=T))) (s) when, ideally, we would prefer to use deow (TT),) from the end
f.t
of training. Of course we will not know the value of dRess (TT 7.1) (s) until the end of training but we can, in principle, use time-series methods to estimate it. To this end, let q., be time-series model with parameters w E (e.g.. IN might be a recurrent neural network) and suppose that we have stored the model checkpoints (T nox
f li/K K < t). We can then train qu to perform forward prediction, for
instance to minimize
KK Σ
(T
(s, (TT (T) (+)))))))
and then use this trained qu to predict the value of deon ( The advantage of this prospective estimator qu is that it can detect that the auxiliary policy will eventually succeed in exactly imitating the expert in given state and thus allow for supervising the main policy with the expert cross entropy loss earlier in training. The downside of such method: it is significantly more complicated to implement and requires running inference using saved model checkpoints.
A.5 ADDITIONAL TASK DETAILS
In Sec. we introduced the different tasks where we compare ADVISOR with various other IL and RL methods. Here, we provide additional details for each of them along with information about observation space associated with each task. For training details for the tasks, please see Sec. A.9
17

---

Under review as a conference paper at ICLR 2021
A.5.1 POISONEDDOORS (PD)
our example from Sec. [ An agent is presented with N = 4 doors d1, ........ d4. Door di is locked, requiring a fixed (0, 1, 2 code to open, but always results in a reward of when opened. For some randomly chosen j {2,3, 4), opening door d, results a reward of and for & {1,j}, opening door d; results in reward of 22 The agent must first choose door after which, if it has chosen door 1, it must enter the combination (receiving reward of 0 if it enters the incorrect combination) and, otherwise, the agent immediately receives its reward. See Fig.
This environment is reproduction
A
Goal
Corner
Obsery
A.5.2 2D- 2D-LIGHTHOUSE (2D-LH)
Boundary A Agent
aations
2D variant of the exemplar grid-world task introduced in Ex. Figure A.2:
2D.
aimed to empirically verify our analysis of the imitation gap. A reward awaits randomly chosen corner square grid of size LIGITTHOUSE
2.V + and the agent can only see the local region, square of size 2i + 1 about itself (an f'-partial observation). Additionally, all f' allow the agent access to it's previous action. As explained in Ex. 2 we experiment with optimizing f" f'-policies when given supervision from f f'-optimal experts (i.e.,
experts that are optimal when restricted to fd-partial observations). See Fig. A.2 for an illustration.
A.5.3 WALLCROSSING (WC)
Initialized on the top-left comner the agent must navigate to the bottom-right goal location. There exists at least one path from start to end, navigating through obstacles. Refer to Fig. 3a where, for illustration, we show simpler grid. Our environment is of size 25 x 25 with 10 walll lals ('S25, N10' as per the notation of (Chevalier-Boisver across the grid. The expert shortest path agent with access to the entire environment's connectivity graph and is implemented via the networks python library.
al. 20T8b)), which are placed vertically or horizontally
A.5.4 LAVACROSSING (LC)
Similar to WALLCROSSING in structure and expert, except that obstacles lava instead of wa walll 1s. Unlike walls (into which the agent can bump without consequence) here the episode terminates the agent steps lava. See Fig, 3b This LC environment has size 25 x 25 with 10 lava rivers ('S25, N10').
A.5.5 WC/LC SWITCH
In this task the agent faces agents for this task have a 'switch' action. Using this switch action, the agents can switch-on the lights of otherwise darkened environment which is implemented WC, even in the dark, an agent can reach the target by taking random actions with non-negligible probability. Achieving this LCC is nearly impossible as random actions will, with high probability, result in stepping into lava and thereby immediately end the episode.
challenging filtration function. In addition navigational actions,
observation tensor of all zeros. In
We experiment with variants of this 'switch' ONCE and FAULTY. In the ONCE SWITCH variant, once the the 'switch' action is taken, the lights remain on for the remainder of the episode. This is implemented as the unaffected observation tensor being available to the agent. contrast, in the FAULTY SWITCH variant, taking the 'switch' action will only turn the lights on for single timestep. This is implemented as observations being available for one timestep followed by zero tensors (unless the 'switch' action is executed again).
The expert for these tasks is the same as for WC and LC. Namely, the expert always takes actions along the shortest path from the agents current position to the goal and is unaffected by whether the light on or off. For the expert-policy-bert basee methods this translates to the learner agent getting perfect (navigational) supervision while struggling in the dark, with no cue for trying the switch action. For the axpert-demonstrattion based methods this translates to the demonstrations being populated with bla observations paired with perfect actions: such actions are, of course,
18

---

Under review as a conference paper at ICLR 2021
difficult to imitate. As FAULTY is more difficult than ONCE (and LC more difficult than WC) we set grid sizes to reduce the difference in difficulty between tasks. In particular, we choose to set WC ONCE SWITCH (S25, N10) grid and the LC ONCE SWITCH on (S15, N7) grid. Moreover, WC FAULTY SWITCH is set with a (S15, N7) grid and L.C FAULTY SWITCH with a (S9, N4) grid.
A.5.6 WC/LC CORRUPT
In the SWITCH task, we study agents with observations affected by a challenging filtration function. In this task we experiment with corrupting the expert's actions. The expert policy flips over to random policy when the expert is Ne steps away from the goal. For the sspert-policy-basel method this translates to the expert outputting uniformly random actions once it is within Nc steps from the target. For the expeettddemonnttatioon---beee based methods this translates to the demonstrations consisting of some valid (observation, expert action) tuples, while the tuples close to the target have the expert action sampled from a uniform distribution over the action space. WC CORRUPT is (S25, N10) grid with Ne = 15, while the LC CORRUPT significantly harder, hence is (S15, N7) grid with Nc 10.
A.5.7 OBSERVATION SPACE
Within our 2D-LH environment we wish to train our agent in the context of Proposition T Sso that the agent may learn any f-partial policy. As the 2D-LH environment is quite simple, we are able to uniquely encode the state observed by an agent using a 4 52 = 6400 dimensional {0, 1 v-valeddd vector such that any f-partial policy can be represented as linear function applied this observation (followed by a soft-max) / at every timestep the agent observes an element of {0, 1, 2, 3} with 0 denoting that no door has yet been chosen, 1 denoting that the agent has chosen door di but has not begun entering the code, 2 indicating that the agent has chosen door d1 and has started entering the code, and representing the final terminal state after door has been opened or combination incorrectly entered. The MINIGRID environments (Chevalier- Chevallir-Booisvert et al. 2018b) enable agents with an egocentric "visual" observation which, in practice, is an integer tensor of shape 7 x 7 3. where the channels contain integer labels corresponding to the cell's type, color, and state. Kindly see (Chevalier- Boisver et al. 2018b a) for details. For the above tasks, the cell types belong to the set of (empty, lava, wall, goal).
Within the PD environment the agent's 's observed state is very simple:
A.6 ADDITIONAL BASELINE DETAILS
In Tab. A.I we include details about the baselines considered in this work, including purely IL (1 - 3, 9). purely RL (4), a sequential combination of them (5 - 7), static combinations of them (8, 10), and our dynamic combinations (11 14). Moreover, we study methods which learn from both expert policy (expert action available for any state) and expert demonstrations (offline dataset of pre-collected trajectories). The hyperparameters (hps) we consider for optimization in study have been chosen as those which, in preliminary experiments, had a substantial impact on model performance. This includes the learning rate (Ir), portion of the training steps devoted to the first stage in methods with two stages (stage-spli),, and the temperature parameter in the weight function [a)l Implicitly, the random environment seed can also be seen as a hyperparameter. We sample hyperparameters uniformly at random. In particular, we sample Ir from [10 44 0.5)) on log-scale, stage-split from [0.1, 0.9), and a from (5.0, 20.0).
A.7 ARCHITECTURE DETAILS
2D-LH model. As discussed in Sec. A.5.7 we have designed the observation given to our agent so that a simple linear layer followed by a soft-max function is sufficient to capture any f-partial policy. As such, our main and auxiliary actor models for this task are simply linear functions mapping
6 As the softmax function prevents us from learning a truly deterministic policy we can only learn a policy arbitrarily close to such policies. In our setting, this distinction is irrelevant.
7 See Sec. 3.22 for definition of the weight function for ADVISOR.
While implemented with supervision from expert policy, due to the teacher forcing being set to 1.0, this method can never explore heyond states (and supervision) in expert demonstrations.
19

---

Under review as a conference paper at ICLR 2021
Table A.1: Baseline details. IL/RL: Nature of learning, Expert supervision: the type of expert supervision leveraged by each method, Hps. searched: hps. that were randomly searched over, fairly done with the same budget (see Sec. A.8 for details).
#
Method
IL/RL
Expert supervision
Hps. searched
BC
II.
Policy
Ir
t
IL
Policy
Ir, stage-split
BC
IL
Policy
Ir
4
PPO
RL
Policy
Ir
BC + PPO
IL RL
Policy
Ir, stage-split
6
+ PPO
-L-
Policy
Ir, stage-split
7 BC + PPO
IL-RL
Policy
Ir, stage-split
BC PPO
RL
Policy
Ir
BC
II.
Demonstrations
Ir
10
BC + PPO
RL
Demonstrations
Ir
11
ADV
IL+RL
Policy
Ir,
12
+ ADV
RL
Policy
Ir, a, stage-split
13
BC + ADV
IL+RL RL
Policy
Ir, a, stage-split
14
BC ADV
+ RL
Demonstrations
Ir, a
the input 6400-dimensional observation to a 4-dimensional output vector followed by a soft-max non-linearity. The critic is computed similarly but with 1-dimensional output and no non-linearity.
PD model. Our PD model has three sequential components. The first embedding layer maps a given observation, a value in {0. 2, 2. 3}, to an 128- 38--dmensiioal embedding. This 128- 128-dimensional vector then fed into a 1-layer LSTM (with 128-dimensional hidden state) to produce an 128-output representation h. We then compute by a soft-max non-linearity. The auxiliary actor is produced similarly but with separate parameters in its linear layer. Finally the critic's s value is generated by applying 128 x linear layer to h.
main actor policy by applying 28 x 7 linear layer followed
MINIGRID model. Here we detail each component of the model architecture illustrated in Fig, 3c The encoder ('Enc.") converts observation tensors (integer tensor of shape 7 x x3) to a corresponding embedding tensor via three embedding sets (of length 8) corresponding to type, color, and state the object. The observation tensor, which represents the fights-out' condition, has unique (i.e., different from the ones listed by (Chevalier-Boiste type, color state from having more than connotation. The output of the encoder of size 7 x 7 x 24. This tensor is flattened and fed into (single-layered) LSTM with a 128-dimensional hidden space. The output of the LSTM is fed to the main actor, auxiliary actor, and the critic. All of these are single linear layers with output size of |A|, |A| and 1, respectively (main and auxiliary actors are followed by soft-max linaarittssssss.
et al., 20185)) type, color and state. This prevents any
A.8 FAIR HYPERPARAMETER TUNING
As discussed in the main paper, we consider two approaches for ensuring that comparisons to baselines are fair. In particular, we hope avoid introducing misleading bias in our results by extensively tuning the hyperparameters (hps) of our ADVISOR methodology while leaving other methods relatively un-tuned.
Tune by Tuning Competing Method. The goal of our experiments with the 2D-LH environment are, principally, to highlight that increasing the imitation gap can have a substantial detrimental impact the quality of policies learned by training IL. Because of this, we wish to give IL the greatest opportunity to succeed and thus we are not, as in our other experiments, attempting to understand its expected performance when we must search for good hyperparameters. To this end, we perform the following procedure for every i, E f1,3, 5 5..., 15} with < <.
2D-LH:
For every learning rate À {100 values evenly spaced in [10-4, on lcg-scale) we train a fi- partial policy to imitate flootimaal expert using BC. For each such trained policy, we roll out trajectories from the policy across 200 randomly sampled episodes (in the 2D-LH there is
20

---

Under review as a conference paper at ICLR 2021
distinction between training, validation, and test episodes there only four unique initial world settings). For each rollout, we compute the average cross entropy between the learned policy and the expert's policy at every step. A "best" learning rate X is then chosen by selecting the learning rate resulting in the smallest cross entropy (after having smoothed the results with a locally-linear regression model (Wasserman 2006)).
A final learning rate is then chosen as the average of the A and this learning rate is then used when training all methods to produce the plots in Fig. 5 As some baselines require additional hyperparameter choices, these other hyperparameters were chosen heuristically (post-hoc experiments suggest that results for the other methods are fairly robust to these other hyperparameters).
All Other Tasks: Random Hyperparameter Evaluations. As described in the main paper, we follow the best practices suggested by Dodge al.] (2019). In particular, for all tasks (except for 2D-LH) we train each of our baselines when sampling that method's hyperparameters, see Table[A.] and recall Sec.[A.6, at random 50 times. Our plots, e.g. Fig. I then report an estimate of the expected (validation set) performance of each of our methods when choosing the best performing model from a fixed number of random hyperparameter evaluations. Unlike (Dodge et al. 2019), we compute this estimate using a U-statistic (van der Vaart (2000, Chapter 12) which is unbiased. Shaded regions encapsulate the 25-to-75th quantiles of the bootstrap distribution of this statistic.
A.9 TRAINING IMPLEMENTATION
A summary of the training hyperparameters and their values is included in Tab. A.2 Kindly see (Schulman et al., 2017) for details on PPO and (Schulman et al. 20156) for details on generalized advantage estimation (GAE).
Max. steps per episode. The maximum number of steps allowed the 2D-LH task is 1000. Within the PD task, an agent can never take more than 11 steps in a single episode (1 action to select the door and then, at most, 10 more actions to input the combination if d, was selected) and thus we do not need to set maximum number of allowed steps. The maximum steps allowed for an episode of WC/LC is set by (Chevalier-Boisver et al. 2018b/p a) to 4S where is the grid size. We share the same limits for the challenging variants SWITCH and CORRUPT. Details of task variants, their grid size, and number of obstacles are included in Sec. A.5
Reward structure. Within the LD task, the agent receives one of three possible rewards after every step: when the agent finds the goal it receives a reward of 0.99, if it otherwise has reached the maximum number of steps (1000) it receives -1 reward, and otherwise, if neither of the prior cases hold, it obtains a reward of -.001. See Sec, A for a description of rewards for the PD task. For WC/LC, (Chevalier-Boisver) et al.) 2018b a) configure the environment to give reward unless the goal reached. If the goal is reached, the reward is 1 structure for our SWITCH and CORRUPT variants as well.
episode length
We adopt the same reward
maximurn sleps"
Computing infrastructure. As mentioned in Sec. 4.3 for all tasks (except LH) we train 50 models (with randomly sampled hps) for each baseline. This amounts to 650 models per task or 5850 models in total. For each task, we utilize g4dn. 1 22 1la a agg instance GPUs and 48 CPUs. We run through a queue of 650 models using 48 processes. For tasks set in the MINIGRID environments, models each require se 0.9 GB GPU memory and all training completes in 18 to 24 hours. For the PD task, model memory footprints are smaller and training all 650 models is significantly faster (< 8 hours).
AWS consisting of 4 NVIDIA T4
A.10 THE DDMDEMO PPO METHOD
As described in the main paper, the ADydemo + PPO method attempts to bring the benefits of our ADVISOR methodology to the setting where expert demonstrations are available but an expert policy (i.e., an expert that can be evaluated at arbitrary states) is not. Attempting to compute the ADVISOR loss (recall Eq. on off-policy demonstrations is complicated however, as our RL loss assumes access to on-policy demonstrations. In theory, importance sampling methods, see, e.g., (Mahmood et al. 2014). can be used to "reinterpret" expert demonstrations as though they But such methods are known to be somewhat unstable, non-trivial to implement, and may require information about the expert policy that we do not have access to. For these reasons, we choose to
on on-policy.
21

---

Under review as a conference paper at ICLR 2021
Table A.2: Structural and training hyperparameters.
Hyperparanter
Value
Structural
Cell type embedding length
Cell color embedding length
8
Cell state embedding length
LSTM layers
1
LSTM hidden size
128
# Layers in critic
1
# Layers in actor
PPO
Clip parameter (c) Schulman et al.
2017
0.1
Decay on c
Linear(1,0)
# Processes to sample steps
20
Rollout timesteps Minibatch size
100
1000
Epochs
4
Value loss coefficient
0.5
Discount factor (7)
0.99
GAE parameter (A)
1.0
Training
Optimizer
Adam Kingma & Ba 2017)
($1, (22) for Adam
(0.9, 0.999)
Learning rate
searched
Gradient clip norm
0.5
Training steps (WC/LC & variants)
1 10
Training steps (2D-LH PD)
3 10a
simple solution: when computing the ADVISOR loss expert demonstrations ignore the RL loss. Thus ADydemo PPO works by looping between two phases:
Collect an (on (on-policy) rollout using the agent's policy, compute the PPO loss for this rollout and perform gradient descent on this loss to update the parameters.
Sample rollout from the expert demonstrations and, using this rollout, compute the demonstration- ADVISOR loss
[ ADV-domo (0) Edemos (W() (S),
(10)
and perform gradient descent on this loss to update the parameters.
POIsON
A.11 ADDITIONAL PLOTS
1.00
11. ADV
E[Reward d
0.75
As mentioned in Sec. 4.3 we record three metrics for our tasks. Reward is the metric that best jointly captures success and effective path planning (see Sec. 2..9 reward structure). In Fig. A..] we illustrate the key aspects of figure showing expected maximum validation performance. the main paper, we included some reward plots in Fig. 4 Specifically, Fig.A.44
acttings are ensugh to mach optimal performance
0.00
Expected mowudd for bost of
A.4e and A.4d have already been included in the main paper (as Fig. 4c 4b 4dd] The remain-
0.25
0.50
ing variants for WC/LC, FAULTY/ONCE SWITCH, and CORRUPT are presented Fig. A.4
training ruus
Success rate shows similar trend, following from the definition of rewards, i.e., agents which reach the target more often, mostly end up with higher rewards. In Fig. A.5 plot success rate for WC/LC, FAULTY/ONCE, SWITCH, and CORRUPT tasks.
Figure Fig.
A.3:
Walk-through
22

---

Under review as a conference paper at ICLR 2021
WALLCRRSSING (WC)
WC FAULTY SWITCH
WC ONCE SWITCH
WC CORRUPT
5,10 :4 (ADY
100
0.7
11. ADV
11. ADV
16.
0.5
0.0
0.6
0.5
B8 Reward
0.5-
0.4
014
0.1
0.3
0.5
0.3
012
012
0.1
11.1
013
0.0
0.00
20
training runs
# training
training runs
training
LRVACIOSSING (LC)
LC FAULTY SWITCH
LC ONCE SWITCH
LC CORRUPT
100
1201
11. ADV
11. ADV
ADV PRO
0.6
0.8
0.5
Rewaard
1.5-
0.4
06
0.
03
0.4
0.4
02
0.2
0.2
0.2
0.1
20
40
20
20
20
40
training
# training runs
# training runs
# training runs
Previous methods
ADVISOR (ours)
IL only:
RL, IL-RL, IL
Based on demonstrations:
11. ADV
1. BC
PPO
9. BC
12. 7 ADV
DAgger (f)
5. BC-> PPO
10. BC PPO
BC + ADV
3.
BC
+ PPO
14. ADV PPO
7. BC + PPO
BC+PPO (statie)
Figure A.4: Reward for baselines MINIGRID tasks. We include all variants of tasks con-
sidered in this work. Similar to Fig. 4 we plot estimates of the expected maximum validation set reward of all baselines (including method), when allowing for increasingly many (random) hyperparameter evaluations (larger [Rewardd] with fewer evals. is better).
23

---

Under review as a conference paper at ICLR 2021
WALL (WC)
WC FAULTY SWITCH
WC ONCE SWITCH
WC CORRUPT
3,10 (ADYY
190
1.0
ADV
1.
TI. ADV
0.5
0.8
0.5
Success
0.6
1.5
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
20
training
# training
training runs
traiming
LAVA CROSSING (LC)
LC FAULTY SWITCH
LC ONCE SWITCH
LC CORRUPT
10.14 (ADD P00H
08
11. ADV
10
II. ADV
10
ADV PPO
0.8
08
B Siccess
0.
0.6
044
03
0.4
1122
02
0.2-
0.0
0.0
0.0
201
40
20
training
# training runs
# training runs
# training runs
Previous methods
ADVISOR (ours)
IL only:
RL, IL-RL, RL
Based demonstrations:
ADV
1. BC
4. PPO
BC
12. i ADV
DAgger
5. BC- PPO
10. BC
PPO
BC ADV
3. BC
6. +- PPO
14. ADV PPO
7. BC PPO
8. BC+PPO (static)
Figure A.5: E[Success Rate] for baselines on MINIGRID tasks. We include all variants of tasks considered in this work. This is the expected maximum validation set success rate of all baselines (including method), when allowing for increasingly many (random) hyperparameter evaluations (larger E, Success Rate] with fewer evals. is better).
24

---

Under review as a conference paper at ICLR 2021
A.12 IMPLEMENTATION DETAILS OF POINTNAV TASK
We follow the dataset splits released for the AlHabitat 2019 Challenge at CVPR 2019. The reward structure is identical to Savva al. (201)) Concretely:
Tgoal + Teloser + V
agent 'stops' next to goal otherwise
(11)
Feloser + v
"goal is the terminal reward of reaching the goal, Telose is the decrease in the geodesic distance to the goal after the step taken time t, and y is the negative reward (time penalty) to encourage optimal paths. ggaa is 10.0 and is -0.01, following the implementation of Saava et al. (2019) released as part of the hab tat-lab.
To create Fig. 6 we evaluate checkpoints after every 1024k frames of experience. This is plotted as the thin line. The thick line and shading depicts the rolling mean (with a window size of 2) and corresponding standard deviation. We stopped training the PPO with 16 processes as it showed no signs of training. For the table, we state the best performance of checkpoints 5, 10, 25 Mn frames of experience.
25


Reviews:
[{"review_id": "tJ3jOE3xO4j", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\n\nPaper aims at attacking the \"imitation gap\" in the canonical LfD problem, where the expert and learner have different observation spaces, leading to poor performance if merely adopting imitation learning without self-exploration. A novel learning-based \nweight proposing function is introduced to mitigate the issue by dynamically weighing the objective of imitation learning (specifically, behavior cloning) and reinforcement learning. The weight is computed based on whether the expert policy disagrees with the best imitation policy on a certain state. Experiments on several artificial tasks demonstrate the effectiveness of the proposed method over pure RL, pure IL, and some canonical RL+IL methodologies.\n\n[Strength]\n\n+) Personally, I like the research problem presented here. Compared to its counterparts on imitation learning with imperfect demonstrations, the authors did an excellent job of making their \"imperfectness\" concrete and also meaningful to potential real-world applications. I won't say it is universally applied but with the intuitive examples, the incentives and the resulting solutions are way more comprehensive than those focusing on injecting noise or perturbation into the demonstrations. \n\n+) The paper is overall clear and well-written. I can't find any major technical issue, while some of the notation could be confusing; Nonetheless, I feel I can get the main idea after reading the main text plus the appendix.\n\n+) The experiments sufficient enough to validate the proposed method. A combination of an artificial but contrived task (PD) and a challenging mini-grid task seems sufficient enough for me.\n\n[Weakness]\n\nThe main concern I have with this submission lies in some missing factors and their potentials to the rationale of the main approach presented here.\n\n-) First thing first, it seems that eq.2 (supposed to be the key loss of the proposed method) implicitly assumes that pi_f is ideal in terms of an imitated policy, so as to the only factor that accounts for the gap between pi_f and pi^exp is the misalignment of observation space. However, I find this assumption could be unrealistic as there are several factors that could affect the imitation performances as well, to name a few: the number of demonstrations (major), different imitation learning algorithms (major), and even the training strategy (epochs, etc, but minor though). The authors did not investigate these factors in their experiments and therefore, it is still unclear whether the loss in eq.2 makes sense as few demonstrations and underfitting will also possibly lead to the disagreement between these two policies. The authors are encouraged to conduct some ablation study with diff. number of demonstrations and also imitation learning methods other than BC to verify how these factors could affect their approach.\n\n-) In the appendix, the authors emphasize that they \"spending too much effort in trying to create a high-quality estimate of pi^aux to pi_f\", while if my understanding is correct, they achieve this by co-train pi_aux with an imitation-only object along with the training of other models. I'm wondering why not just train pi_aux first then query it in the training of the rest? Is there any specific reason to sacrifice the estimation accuracy with co-training? Will the proposed approach enjoy better performance with this pre-train strategy?\n\n-) According to eq.2, the computation of the weight requires a query on state s to both pi_f and pi^exp, while the authors introduce a demonstration-only variant in their experiments. As only the action is marginalized out, it could be tricky to do such computation merely with samples of (s, a). The authors are expected to provide more details on their implementation.\n\n[Suggestions&Questions]\n\n(1) Add an ablation study on how the number of demonstrations and imitation learning algorithm could affect the performances.\n\n(2) Add a baseline that only pre-train pi^aux rather than co-train with other models.\n\n(3) How can eq.2 work with a demonstration-only expert? Please clarify.\n\n(4) Some citations need to be added:\n\nReinforcement learning from demonstration through shaping, in IJCAI, 2015\n\nDirect policy iteration with demonstrations, in IJCAI, 2015\n\nPolicy shaping with human teachers, in IJCAI, 2015\n\nTruncated horizon policy search: Combining reinforcement learning and imitation learning, in ICLR, 2018\n\nReinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance, in AAAI, 2020\n\n[Post-rebuttal]\n\nI have read through all the other reviews and the rebuttal. Would like to thank the authors for their efforts in improving this submission. I  do believe some of my concerns have been addressed while the main issue on the lack of some necessary evaluations (e.g. num. of demonstrations) remains. Thus I may not be able to escalate my justification to even higher.", "extra_scores": {}, "extra_text": {}}, {"review_id": "f2M1MR9XEKE", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1. this paper studies an interesting problem, \"imitation gap\" in imitation learning. \n2. the paper is easy to follow. Especially, the example part is easy to understand.\n3. the intuition for this idea is well-explained.\n\n\nCons:\n1. from my perspective, the basic idea is dynamically using imitation learning and reinforcement learning for agent learning by a weighting function. It is a straightforward idea but lacks some novelty. \n\n2. the main contribution of this paper is proposing an advisor and integrating it with the imitation and reinforcement learning process. However, only averaging loss to leverage imitation and exploration is a comparatively little contribution.\n\n3. in experiment PD, adding advisor from the beginning seems not fair enough, since at the beginning the reward is much higher than baselines. Maybe the experiments need a comparison before the advisor is added and after the advisor added.\n\nIn summary, in this paper, the motivation is clear and easy to understand, and the problem is worthy to study. But the contribution of this paper is a little limited. A better solution is needed.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z8575NVVuj", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper identifies a problem in imitation learning when an expert has access to privileged information that is not available to the learner. When a decision has to be made based on the privileged information, the learner tends to choose average or uniformly random actions of the expert due to the lack of important information, which is called the \"imitation gap\". Therefore, in such cases, learning from the expert can actually harm the training of the learner.\nThis paper proposes to tackle this imitation gap by following the expert only when the learner can reproduce the expert's behaviors only with partial observations. Otherwise, the learner ignores the expert's guidance and relies on reward-based RL. The proposed method effectively balances between learning from the expert and learning from reward with the divergence between action distributions between expert and an auxiliary policy, which is trained sorely from the expert. The exhaustive experiments prove that the proposed method outperforms baselines in environments with a large imitation gap.\n\n### Strengths\n- This paper identifies the \"imitation gap\" that could be critical for many imitation learning applications with intuitive examples.\n- The proposed method is simple but effective in resolving the imitation gap by learning from the expert only when the learner does not require access to privileged information.\n- The comparative experiments are thoroughly conducted and well presented.\n- The proposed method has many potential applications, such as sim2real transfer, where a policy trained in simulation has access to state information, and thus the imitation gap can be problematic.\n\n### Weaknesses\n- As the proposed method involves multiple learnable components, providing the algorithm could clarify how the method works. Currently, it is not clear to understand how the auxiliary policy is trained together with the imitation learning policy.\n- The tasks tested in the paper are limited to simple 2D examples, which are tailored to the proposed method. Including more general and complex tasks could improve the work. It would be great to see how it can be generally applicable to continuous action space, e.g., ant navigation or manipulation tasks with partial observations or cluttered environments.\n- ADV fails to achieve good performance (stuck around 0.2) on the LC task. In theory, ADV should learn as much as BC-demo+PPO and ADV-demo+PPO can. More explanation would help to understand the low performance of ADV compared to other methods.\n- The observation and action spaces can be briefly described in the main paper. For example, the action space for PoisonedDoor is not presented in the paper.\n\n### Questions and additional feedback\n- Cite a relevant paper \"Zhu et al. Reinforcement and Imitation Learning for Diverse Visuomotor Control, RSS 2018\", which combines GAIL and RL, and demonstrates its effectiveness on diverse manipulation tasks.\n- What is the reference for \"teacher-forcing\"?\n- Attach the appendix at the end of the main paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PEQRqx5536z", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Short summary:\n\nThe paper presents an algorithm that allows an agent to leverage demonstrations from an expert with potential privileged  information compared to the agent. The main idea consists in measuring the distance or divergence between the imitation policy learned on the expert demonstrations and the expert policy and use this information to weight (convex combination) the importance of the imitation term and the reinforcement learning term in the combined loss function. \n\nIf there is indeed a discrepancy between the learned imitation policy and the expert policy, this could be linked to the fact that the expert and the agent do not share the same belief state.\n\n General comments:\n\nThe paper presents an important problem that could arise when naively applying imitation learning methods in partially observable MDPs where expert and apprentice do not share the same amount of information about the underlying state.  Basically, if the apprentice cannot distinguish states that are actually different for the expert, the imitation policy will be an average distribution of the expert policy over those states. This can be quite bad because the apprentice takes an average action instead of a precise one as it does not understand the underlying state as well as the expert. This is the imitation gap phenomenon.\nHowever, it is also important to remind the reader that most imitation learning algorithms have been designed under the MDP hypothesis and therefore are not made to answer the imitation gap problems. It is only careless application of those algorithms in inappropriate settings such as POMDPs with asymmetric privileged information between expert and apprentice where this problem can arise. Therefore, I think the authors should mention some previous works that consisted in mitigating the problem of imitation gap such as in the field of conditional imitation learning:\n- End-to-end Driving via Conditional Imitation Learning (https://arxiv.org/abs/1710.02410)\n- Urban Driving with Conditional Imitation Learning (https://arxiv.org/abs/1912.00177)\nwhere the goal is to add privileged information to the apprentice (in the form of navigational instructions) in order to have a better understanding of the underlying state. In addition the authors may also refer to works in representation learning for Imitation learning such as:\n-  Learning Belief Representations for Imitation Learning in POMDPs (https://arxiv.org/abs/1906.09510)\nthat  try to provide better underlying state for the apprentice. Even if those works are not directly related to deal with imitation gap, they are partial solutions to it. Unlike the method proposed by the authors that consists in reducing the importance of demonstrations where there is an imitation gap, they directly try to improve the amount of information of the state provided to the apprentice.\n\nFinally, concerning the method proposed by the authors, I do think that it is an interesting algorithm that weights demonstrations depending on the imitation gap. However this should be coupled with a method that try to enrich the state of the apprentice in order to not lose too much demonstrations. Using an LSTM is a first step towards that direction but more could be done (see papers provided in the previous section).\n\nDetailed remarks:\n\nI have several comments concerning specific sentences in the paper:\n- \"To overcome this imitation gap, prior work often uses stage-wise training\": I do not agree with that at all. I think that previous work used stage-wise training mainly to improve upon the expert demonstrations. Indeed in some of the previous works,  expert and apprentice had directly access to the full state and therefore no imitation gap was possible. Still uncertainty on the expert policy could remain because demonstrations could come from different experts for instance.\n\n-\"As we show empirically, ADVISOR combines the benefits of IL and RL while avoiding the pitfalls of either method alone.\" I would also add that most of the IL methods were not designed to deal with the imitation gap as they make the MDP hypothesis.\n\n-\"These advances have been further improved upon through policy gradient methods\", this has not really been shown or only on a very small set of control tasks. I think this should be reformulated because it sounds like policy gradient methods are better than deep-Q learning methods.\n\n-\"To the best of our knowledge, this hasn’t been studied before.\" weighting demonstrations depending on the imitation gap may not have been studied but methods that try to fill the gap have. It will be good that add this nuance. \n\n- Concerning the imitation gap section, I would directly introduce the POMDP setting and explain simply the imitation gap there. Then I would go on formalising it. The example 2 description is too long and could be more concise and have more impact. I feel also the same with the algorithmic section that could be better organise.\n\nExperiments:\n\nThe experiments are conducted in grid-worlds. They are interesting to show the intuition behind the algorithm. However, it will be easier to grant acceptance if the experiments were on visually rich 3D environnements or in robotics.\n\nRating: I think I could increase my rating if the authors take into account my remarks. I also believe that it is important to not only discard demonstrations where the information gap is important but to try to close the gap by better representation learning or memories. In addition a careful use and collection of expert demonstrations could also mitigate this problem.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers acknowledged that the problem addressed in this paper is interesting and is not solved by the existing literature. They appreciated that the setup was well defined and the paper was clearly written. Yet they kept several concerns after the rebuttal. Especially, they expected the comparison to be done with algorithms using both demonstrations and rewards and the current empirical evaluation was not judged as fair. Also, the simple baseline consisting of adding an LSTM to BC to integrate past observations has not been considered either. This baseline is still missing to assess the quality of the proposed method. 
Decision: Reject