Bridging the Imitation Gap by Adaptive Insubordination
Authors: Luca Weihs
Unnat Jain
Iou-Jen Liu
Jordi Salvador
Svetlana Lazebnik
Aniruddha Kembhavi
Alex Schwing
Venue: NeurIPS 2021 Poster (2021)
Keywords: Imitation Learning
Reinforcement Learning
Actor-Critic
POMDPs
Privileged Experts
Visual Navigation

Abstract: In practice, imitation learning is preferred over pure reinforcement learning whenever it is possible to design a teaching agent to provide expert supervision. However, we show that when the teaching agent makes decisions with access to privileged information that is unavailable to the student, this information is marginalized during imitation learning, resulting in an "imitation gap" and, potentially, poor results. Prior work bridges this gap via a progression from imitation learning to reinforcement learning. While often successful, gradual progression fails for tasks that require frequent switches between exploration and memorization. To better address these tasks and alleviate the imitation gap we propose 'Adaptive Insubordination' (ADVISOR). ADVISOR dynamically weights imitation and reward-based reinforcement learning losses during training, enabling on-the-fly switching between imitation and exploration. On a suite of challenging tasks set within gridworlds, multi-agent particle environments, and high-fidelity 3D simulators, we show that on-the-fly switching with ADVISOR outperforms pure imitation, pure reinforcement learning, as well as their sequential and parallel combinations.

Bridging the Imitation Gap by Adaptive Insubordination
Luca Weihs* 11, Unnat Jain 222, Iou-Jen Liu?, Jordi Salvador', Svetlana Lazebnik², Aniruddha Kembhavi', Alexander Schwing Allen Institute for AI, University of Illinois at Urbana-Champaign [lucaw, jordis, anik)@allenai.c {uj2, iliu3, slazebmi, aschwing)@illiiniie .du
Abstract
In practice, imitation learning is preferred over pure reinforcement learning when- ever it is possible to design teaching agent to provide expert supervision. However, we show that when the teaching agent makes decisions with access to privileged information that is unavailable to the student, this information is marginalized dur- ing imitation learning, resulting in an "imitation gap" and, potentially, poor results. Prior work bridges this gap via progression from imitation learning to reinforce- ment learning. While often successful, gradual progression fails for tasks that require frequent switches between exploration and memorization. To better address these tasks and alleviate the imitation gap we propose 'Adaptive Insubordination' (ADVISOR). ADVISOR dynamically weights imitation and reward-based rein- forcement learning losses during training, enabling on-the-fly switching between imitation and exploration. On suite of challenging tasks within gridworlds, multi-agent particle environments, and high-fidelity 3D simulators, we show that on-the-fly switching with ADVISOR outperforms pure imitation, pure reinforce- ment learning, as well as their sequential and parallel combinations.
Introduction
Imitation learning (IL) can be remarkably successful settings where reinforcement learning (RL) struggles. For instance, IL has been shown succeed in complex tasks with sparse rewards & 47 44]. and when the observations are gigh--imensional, e.g., in visual 3D environments [1 To succeed, IL provides the agent with consistent expert supervision at every timestep. making it less reliant on the agent randomly attaining success. To obtain this expert supervision, it is often convenient to use "privileged information," i.e., information that is unavailable the student at inference time. This privileged information takes many forms in practice. For instance, in navigational tasks, experts are frequently designed using shortest path algorithms which access the environment's connectivity graph [e.g., [9. Other forms of privilege include semantic maps [e.g,, [0. [3, the ability to see into "the future" via rollouts (6). and ground-truth world layouts [7]. The following example shows how this type of privileged information can result in IL dramatically failing.
Example (Poisoned Doors). Suppose an agent is presented with N 2 3 doors di,..., dx. As illustrated in Fig. D (for N = 4), opening d1 requires entering an unknown fixed code of length M. Successful code entry results a reward of 1, otherwise the reward is O. Since the code unknown to the agent, it would need to learn the code by trial and error. All other doors can be opened without a code. For some randomly chosen 2 j < N (sampled each episode), the reward behind d, is 2 but for all E (2. . N) (i} the reward behind d, is -2. Without knowing j. the optimal policy is to always enter the correct code to open dj obtaining an expected reward of 1. In contrast, if the expert
denotes equal contribution by LW and UJ: work done, in part, as an intern at Allen Institute for Al
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

is given the privileged knowledge of the door dj with reward 2, it will always choose to open this door immediately. It is easy see that an agent without knowledge of j attempting to imitate such an expert will learn to open a door among .2....; dy uniformly at random obtaining expected return of -2. (N - 3)/(N - 1). In this setting, training with reward- RI. after a 'warm start' with II. is strictly worse than starting without it: the agent needs to unlearn its policy and then, by chance, stumble into entering the correct code for door d1, a practical impossibility when M is large.
To characterize this imitation failure, we show that training a student to imitate teacher who uses privileged information results in the student learning a policy which marginalizes out this privileged in- formation. This result in sub-optimal, even uniformly random, student policy over a large collection of states. We call the discrep- ancy between the teacher's and student's policy the imitation gap. To bridge the imitation gap, we introduce Adaptive Insubordination (ADVISOR). ADVISOR adaptively weights imitation and RL losses. Specifically, throughout training we use an auxiliary actor which judges whether the current observation better treated using an IL or a RL loss. For this, the auxiliary actor attempts to reproduce the teacher's action using the observations of the student at every step. Intuitively, the weight corresponding to the IL loss is large when the auxiliary actor can reproduce the teacher's s action with high confidence.
2.
3.
4.
?
?
-2
-2
2
Consistent reward behind locked door
High-value door chosen random
Figure 1: POISONEDDOORS.
We study the benefits of ADVISOR on thirteen tasks, including "POISONEDDOORS" from Ex. 2D "lighthouse" gridworld, suite of tasks set within the MINIGRID environment & 9 Cooperative Navigation with limited range (COOPNAV) in the multi-agent particle environment (MPE) 43 38 and two navigational tasks set in 3D, high visual fidelity, simulators of weal-world living environments (POINTNAV in AIHABITAT [54] and OBJECTNAV in ROBOTHOR [3 IE!). Our results show that, the imitation gap's 's size directly impacts agent performance when using modern learning methods, ADVISOR is performant (outperforming IL and RL baselines), robust, and sample efficient,
L
ADVISOR can succeed even when expert supervision is partially corrupted, and
ADVISOR can be easily integrated in existing pipelines spanning diverse observations (grids and pixels), actions spaces (discrete and continuous), and algorithms (PPO and MADDPG).
2
Related Work
A series of methods [e.g.. 41
have made off- off-policy deep Q-learning stable for complex environments like Atari Games. Several nigh-performance (on-policy) policy-gradient methods for deep-RL have also been proposed [56 42 34 68 61]. For instance, Trust Region Policy Optimization (TRPO) 56 improves sample-efficiency by safely integrating larger gradient steps. Proximal Policy Optimization (PPO) 5 employs a clipped variant of TRPO's surrogate objective and is widely adopted in the deep RL community. We use PPO as a bascline in our experiments.
65 5
55
As environments get more complex, navigating the search space with only deep RL and simple heuristic exploration (such as c-greedy) is increasingly difficult. Therefore, methods that imitate expert (i.e., teacher) supervision were introduced. A popular approach to imitation learning (IL) is Behaviour Cloning (BC), i.e., use of a supervised classification loss between the policy of the student and expert agents 5 p.. However, BC suffers from compounding errors. Namely, single mistake of the student may lead to settings that have never been observed in training (51). To address this, Data Aggregation (DAgger) 52 trains sequence of student policies by querying the expert at states beyond those that would be reached by following only expert actions. IL is further enhanced by, e.g., hierarchies [33,, improving over the expert [5] 27 bypassing any intermediate reward function inference 24 and/or learning from experts that differ from the student 18 26 [6] Importantly, a sequential combination of IL and RL, i.e., pre-training model on expert data before letting the agent interact with the environment, performs remarkably well. This strategy has been applied in a wide
range of applications- the game of Go [61, robotic and motor skills 49 20 visually realistic environments
48 50 navigation in
19 and web & language based tasks
67
More recent methods mix expert demonstrations with the agent's own rollouts instead of using sequential combination of II. followed by RI. Chemali and Lazaric (6] perform policy iteration from expert and on-policy demonstrations, DQID 2 initializes the replay buffer with expert episodes

---

and adds rollouts of (a pretrained) agent. They weight experiences based on the previous temporal and use a supervised loss to learn from the expert. For continuous action spaces, DDPGfD 66 analogously incorporates II. into DDPG 35 POAD (28 improves by adding
difference errors [55
a demonstration-gun -uided exploration term, L.e. the densen-Shannop divergence between the expert's and the learner's s policy (estimated using occupancy measures). THOR uses suboptimal experts to reshape rewards and then searches over a finite planning horizon [62]. Zhu et al. [72] show that a combination of GAIL [44] and RL can be highly effective for difficult manipulation tasks.
Critically, the above methods have, implicitly or explicitly, been designed under certain assumptions (e.g., the agent operates in an MDP) which imply the expert and student observe the same state. Different from the above methods, we investigate the difference of privilege between the expert policy and the learned policy. Contrary to sequential, static, or rule-based combination of supervised loss or divergence, we train an auxiliary actor to adaptively weight II. and RI. losses. To the best of our knowledge, this hasn't been studied before. In concurrent work, Warrington et al. [99] address the imitation gap by jointly training their teacher and student to adapt the teacher to the student. For our applications of interest, this work is not applicable our expert teachers are fixed.
Our approach attempts reduce the imitation gap directly, assuming the information available the learning agent is fixed. An indirect approach to reduce this gap is to enrich the information available to the agent or improve the agent's memory of past experience. Several works have considered this direction in the context of autonomous driving [TO] [20] and continuous control [17]. We expect that these methods can be beneficially combined with the method that we discuss next.
3 ADVISOR
We first introduce notation to define the imitation gap and illustrate how arises due to 'policy averaging.' Using an 'auxiliary policy' construct, we then propose ADVISOR to bridge this gap. Finally, we show how to estimate the auxiliary policy in practice using deep networks. In what follows we will use the terms teacher and expert interchangeably, Our use of "teacher" is meant to emphasize that these policies are (I) designed for providing supervision for a student and (2) need not be optimal among all policies.
3.1 Imitation gap
We want an agent to complete task T in environment E. The environment has states 8 E S and the agent executes an action a E at every discrete timestep 2 0. For simplicity and w.l.o.g. assume both A and S are finite. For example, let & be a ID-griworld in which the agent is tasked with navigating to a location by executing actions to move left or right, as shown in Fig. 2a Here and below we assume states s E S encapsulate historical information so that s includes the full trajectory of the agent up to time 2 O. The objective to find a policy π, i.e., a mapping from states to distributions over actions, which maximizes an evaluation criterion. Often this policy search restricted to set of feasible policies ITfeas, for instance Iffens. may be the set (T(:; E RD
where (-; 0) is deep neural network with D-dimensional parameters O. In classical (deep) RL 41 42 the evaluation criterion is usually the expected y-discounted future return.
We focus on the setting of partially-observed Markov decision processes (POMDPs) where an agent makes decisions without access the full state information. We model this restricted access by defining a filtration function f S + Oy and limiting the space of feasible policies to those policies funruationnfcction
11 for which the value of 7 (s) depends on s only through f(s), i.e., so that f(s) f(x') implies T(s) = 300==sss), (ss).. We call any satisfying this condition an f-restricted policy and the set of feasible f-restricted policies IT In gridworld example, / might restrict 8 to only include information local the agent's current position as shown in Figs. 2c 2d If f-restricted policy optimal among all other f-restricted policies, we say is f-optimal. We call o E O partiaalobservation and for any f-restricted policy Ty we write / to mean x (s) f(s)
o. It frequently the case that, during training, we have access to a teacher policy which is able to successfully complete the task T. This teacher policy may have access to the whole environment state and thus may be optimal among all policies. Alternatively, the teacher policy may, like the student, only make decisions given partial information (e.g., human who sees exactly the same inputs as the student). For flexibility we will teach define the teacher policy plan, denoting it is an f peach restricted policy for some filtration function
3

---

Envirromeent & start states
Hypothetionl episode
guartial alos.
8 partial ols..
+/
1.
12
11
n.
Bankin
1.0
A
Monn right
A
00
1.0
15
0.7
03
16
0.. 05
Mose rigle
Grai
Boundory Aguni
Actions A fleft, right)
a. Goll rostee
RL
(a)
(b)
(c)
(d)
Figure 2: Effect of partial observability in I-dimensional gridworld environment. (a) The two start states and actions space for ID- ID-Lighthouse with N = 4. (b) A trajectory of the agent following a hypothetical random policy. At every trajectory step we display output probabilities as per the shortest-pan path expert (I7(sch) for each state. (c/d) Using the same trajectory from (b) we highlight the sitrial-obsevvations available to the agent (shaded gray) under different filtration function f*, f2. Notice that, under f the agent does not the goal within its first four steps. The policies a "f " y2. Tll learned by imitating z/ench, show that imitation results in sub-optimal policies, i.e. fis g o xeeach
freach For simplicity, we will assume that seach freach -ptiial. Subsequently, we will drop the fusch
subscript freach unless we wish to explicitly discuss multiple teachers simultaneously.
In IL 45
Ty is trained to mimic x rlach by minimizing the (expected) cross-entropy between Ty and π over a set of sampled states E S:
teach
min E,,.EE(TT teach (S)) xy E
(1)
where C Ty (S) = - @ log क. (S), @ denotes the usual dot-product, and $ is a random variable taking values S with probability measure μ : S + [0, 1]. Often u(s)) is chosen to equal the frequency with which an exploration policy (e.g., random actions or π Ileach) visits state s in randomly initialized episode. When it exists, we denote the policy minimizing Eq. x niah
When μ and π leacch are unambiguous, we write 7 = o =
What happens when there is difference of privilege (or filtration functions) between the teacher and the student? Intuitively, if the information that a teacher uses to make a decision is unavailable to the student then the student has little hope of being able mimic the teacher's decisions. As we show in our next example, even when optimizing perfectly, depending on the choice of f and french IL may result in w!l being uniformly random over large collection of states. We call the phenomenon that IL # n teach the imitation gap.
Example 2 (ID-Lighthouse). We illustrate the imitation gap using a gridworld spanning (-N,..., ,). The two start states correspond to the goal being at either -N or N, while the agent is always initialized at 0 (see Fig. 2ap. Clearly, with full state information, n. techh maps states to an 'always left' or "always right' probability distribution, depending on whether the goal is on the left or right, respectively. Suppose now that the agent's 's visibility is constrained to radius of i (Fig. 2c shows i 1). i.e., an f'-restricted observation is accessible. An agent following an optimal policy with visibility of radius i will begin to move deterministically towards any corner, w.l.o.g. assume right. When the agent sees the rightmost edge (from position N -i), it will either continue to move right if the goal is visible or, if it's not, move left until it reaches the goal (at -N). Now we may ask: what is the best resttriite policy that can be learnt by imitating x zteach (i.e., what is x/): Tragically, the cross-entrops loss causes THz to be uniform in a large number of states. In particular, an agent following policy T will execute left (and right) actions with probability 0.5, until is within a distance of from one of the corners. Subsequently, will head directly the goal. See the policies highlighted in Figs. 2e 2d The intuition for this result is straightforward: until the agent observes one of the corners it cannot know the goal is to the right or left and, conditional on its observations, each of these events is equally likely under (assumed uniform). Hence for half of these events the teacher will instruct the agent to go right. For the other half the instruction to go left. See App. A. for a rigorous treatment of this example. In Sec. 4 and Fig. G we train f'-restricted policies with fs-optimal teachers for 2D variant of this example. We empirically verify that student learns a better policy when imitating teachers whose filtration function is closest to their own.

---

The above example shows: when a student attempts to imitate an expert that is privileged with information not available the student, the student learns a version of zleach in which this privileged information is marginalized out. We formalize this intuition in the following proposition.
Proposition 1 (Policy Averaging).
In the setting of Section 3.1 suppose that Theas contains all f-restricted policies. Then, for any S with = f(s), we have that of (0)( = By [T (S) f(S) = oj.
Given our definitions, the proof of this proposition is quite straightforward, see Appendix
The imitation gap provides theoretical justification for the common practical observation that an agent trained via IL can often be significantly improved by continuing to train the agent using pure RL PPO) B8 [3. Obviously training first with IL and then via pure RL is ad hoc and potentially sub-optimal as discussed in Ex. and empirically shown in Sec. 4 To alleviate this problem, the student should imitate the teacher's policy only in settings where the teacher's policy can, in principle, be exactly reproduced by the student. Otherwise the student should learn via 'standard' RI.. To achieve this we introduce ADVISOR.
3.2 Adaptive Insubordination (ADVISOR) with Policy Gradients
To close the imitation gap, ADVISOR adaptively weights reward- and imitation losses. Intu- itively, it supervises student by asking it to imitate teacher's policy only in those states s E S for which the imitation gap is small. For all other states, it trains the student using reward-based RL. To simplify notation, we denote the reward-based RL loss via [0,, S)] for some loss function L This loss formulation is general and spans all policy gradient methods, including A2C and PPO. The imitation loss is the standard cross-entropy loss E.p. [C (each x (5; Concretely, the ADVISOR loss is:
CADY (0) Ep(w(S) CE((T (S), (1 (1-11111)) w()) L(,
Our goal is find a weight function : 8 + [0, 1] where w(s) Rs 1 when the imitation gap small and w(s) ss 0 otherwise. For this we need an estimator of the distance between 7 reach and Tt at state s and mapping from this distance to weights in [0, 1].
We now define distance estimate do (nt,,, (s) between policy and f-restricted policy πy at a state s. We can use any common non-negative distance (or divergence) d between probability distributions A, e.g., in our experiments
use the KL-divergence. While there are many possible strategies for using d to estimate (I, Ty) (s), perhaps the simplest of these strategies is to define d'T,, (s) = d((r(s), (s)). Note that this quantity does not attempt to use any information about the fiber (f(ss) which may be useful in producing more holistic measures of distances Appendix A.3 considers how those distances can be used in lieu of d', Next, using the above, we need to estimate the quantity (nr (s).
Unfortunately it is, in general, impossible to compute d (nrtceh (s) exactly as it is intractable to compute the optimal minimizer T.. Instead we leverage an estimator of T. which we term mijis, and which we will define in the next section.
Given x we obtain the estimator (m (T)x) of dd n...). Additionally, we make use of the monotonically decreasing function ma 1Rz0 + [0, 1]. where a 0. We define our weight function (s)) for E S as:
w(s) = m. (d (1((eeeect a (s)) with (.) c
(3)
FFor readability, we implicitly make three key simplifications. First, computing the expectation [.... generally intractable, hence we cannot directly minimize losses such as (0, S)J. Instead, we approximate the expectation using rollouts from μ and optimize the empirical loss. Second, recent RL methods adjust the measure y over states as optimization progresses while we assume it to be statie for simplicity. Our final simplification regards the degree to which any loss can he, and is, optimized. In general, losses are often optimized by gradient descent and generally no guarantees are given that the global optimum can be found. Extending our presentation to encompass these issues is straightforward but notationally dense.
.] is
$ "Measures using such information include (ar(e d(((ss'), xy (s)) or a corresponding expectation instead of the maximization, i.e., (S)) f(S) = o].
5

---

Ausi aerieno to past
Aaask car areaca to soel oigest
Oigniin: bakk leshaario und ueed collasee
Aprit
peil
Tocks
Doopa Jon:
Bize egeala
Lark cle
2D-LR
LAVACRosivG
CooPNAV
PoINTN
ORJECTNAY
Simulakors
MiniGrid
MPE
AlHabital
A12-THOR
Optiinal Vaiiiiity
Glchal
iesiaad with visilility emago
Gisbal ducteriogalli
Global sourtisk-palh
Expents
artion. undfieciad light
loyont)
lageott)
Strdy the iniiation ladiorssss
F'st simmlation enaidee extensive stadly
Demonstrate ADVISOR
Stody ADVISDR's
Demonstrate
Objective
be applied contlssuss mull--agent
withli
ADVISOR succoeds
performance
Figure Representative tasks from experiments. 2D-LH: Harder 2D variant of the gridworld task introduced in Ex. 2 LAVACROSSING: one of our 8 tasks in the MINIGRID environment requiring safe navigation. We test up-to 15 15 grids with 10 lava rivers. COOPNAV: multi-agent cooperative task set in multi-agent particle environments. POINTNAV: An agent embodied in the AIHABITAT environment must navigate using egocentric visual observations to a goal position specified by a GPS coordinate. OBJECTNAV: An agent in ROBOTHOR must navigate an object of given category.
3.3 The Auxiliary Policy nh Estimating o in Practice
In this section describe how can, during training, obtain auxiliary policy x which estimates T... Given this auxiliary policy we estimate d (T(ttechh ()) using the plug-in estimator ds "))" While plug-in estimators are intuitive and simple to de- fine, they need not be statistically efficient. In Appendix A.4 we consider possible strate- gies for improving the statistical efficiency of our plug-in estimator via prospective estimation.
In Fig. we provide an overview of how we compute the estimator π via deep nets. As common practice
New partial observation
Env
Action
Annolle
8 the policy net (-; () is composed via a, r2 with = (v, A), where a, is the actor head (possibly complemented in actor-critic models by a critic head v.) and FΛ is called the representation net- work. Generally 0g is lightweight, for instance a linear layer or a shallow MLP followed by soft-max function, while yy is deep, and possi- bly recurrent neural, net. We add another actor head a, to existing network which shares the underlying representation yx, i.e., of ag o rx We experiment with the actors sharing their representation and estimating T via two separate networks, i.e., 0' (') In practice we train ( 0) and (:; jointly using stochastic gradient descent, summarized in Alg, A.I
22
Pa
Actoo
(e; Ø)
ATD ISOIt Ioen
Eacoder
LSTM
Chilie By
Valuee
Value
f(8) Partial
aus (x;
Optionally
share backhooe
Figure 3: Model overview. An auxiliary actor is added and trained only using IL. The 'main' actor policy is trained using the ADVISOR loss.
4 Experiments
We rigorously compare ADVISOR to IL methods, RL. methods, and popularly-adopter (but often ad hoc) IL & RL combinations. In particular, evaluate 15 learning methods. We do this over thirteen tasks realizations of Ex. [ & Ex. eight tasks varying complexity within the fast, versatile MINIGRID environment
8 J. Cooperative Navigation (COOPNAV) with reduced visible range in the multi-agent particle environment (MPF) 43 37), PointGoal navigation (POINTNAV) using the
Gibson dataset in AIHABITAT
711 54 and ObjectGoal Navigation (OrJECTNAV) in ROBOTHOR
!4!
Furthermore, to probe robustness, we train 50 hyperparameter variants for each of the 15
'The ROBOTHOR environment sub-environment of 12-THOR 31

---

LAVACROSSING
WALLCROSSING
Training routines 1
Base Ver.
Corrupe Exp.
Faally Swisch
Once Switch
Base Ver.
Corrupt Exp.
Faulty Switch
Once Switch
RL
0
0.01
0
0.00
0.07
0.12
D,05
0188
0.02
0.96
0.85
0.11
IL & RL
-0.17
0.94
0.74
0.97
0.18
0.17
0.1
Demo, Besed
-0.09
0.96
0.2
0.02
0
0.97
0.07
0.18
0.11
ADV. Based (ours)
0.96
0.94
0.77
0.8
0.97
0.31
0.38
0.45
Table 1: Expected rewards for the POISONED task and MINIGRID tasks. For each of
our 15 training routines we report the expected maximum validation set performance (when given budget of 10 random hyperparameter evaluations) after training for =300k steps POISONEDDOORS and RJ IMn steps in our 8 MINIGRID tasks. The maximum reward is I for the MINIGRID tasks.
learning methods for our MINIGRID tasks. We find ADVISOR-based methods outperform or match performance of all baselines.
All code to reproduce our experiments will be made public under the Apache 2.0 license The environments used are public for academic and commercial use under the Apache 2.0 (MINIGRID and ROBOTHOR) and MIT licence (MPE and AlHABITAT).
4.1 Tasks
Detailed descriptions of our tasks (and teachers) are deferred to Appendix A.5 high-level overview of 5 representative tasks.
See Fig. for
4.2
Baselines and ADVISOR-basee Methods
We briefly introduce baselines and variants of our ADVISOR method. Further details all methods
in Appendix A.7 For fairness, the same model architecture is shared across all methods (recall
Fig.
Sec.
3.3)
We defer implementation details to Appendix [A.8]
RL only. Proximal Policy Optimization [S8] serves the pure RL baseline for all our tasks with
discrete action space. For the continuous and multi-agent COOPNAV task, we follow prior work and
adopt MADDPG 37 [36]
IL only. baselines where supervision comes from an expert policy with different levels of feacher-forcing (tf), i.e., tf=0, tf annealed from 1-00, and tf=1. This leads to Behaviour Cloning
(BC), Data Aggregation (DAgger or t), and BC respectively
53
IL RL. Basclines that use a mix IL and RL losses, either in sequence in parallel. These are popularly adopted in the literature to warm-start agent policies. Sequential combinations include BC then PPO (BC-PPO), DAgger then PPO (t + PPO), and Bf - PPO. The parallel combination of BC + (st(staiic) is static analog of our adaptive combination of and RL losses.
Demonstration-Dased These agents imitate expert demonstrations and hence get no supervision beyond the states in the demonstrations. We implement HC its combination with PPO (BCdemo + PPO), and Generative Adversarial Imitation Learning (GAIL) 24.
ADVISOR-based (ours). Our Adaptive Insubordination methodology can learn from expert policy and can be given a warm-start via BC or DAgger. This leads to ADVISOR (ADV), 1f + ADV, and t > ADV) baselines. Similarly, ADydemo + PPO employs Adaptive Insubordination to learn from expert demonstrations while training with PPO on on-policy rollouts.
4.3 Evaluation
Fair Hyperparameter Tuning, Often unintentionally done, extensively tuning the hyperparameters (hps) of a proposed method and not those of the baselines can introduce unfair bias into evaluations. We avoid this by considering two strategies. For PD and all MINIGRID tasks, we follow recent best practices 15). Namely, we tune each method by randomly sampling a fixed number of hps and
See httpss//unnat.. github. io/advisor/ for up-to-date link this code.
7

---

POSSNNEDDORS
LAVA CROSSEVG (LC)
LC SwITCH ONCE
LC CORRUPT EXPERT
Previous methods
only
Exppecee Bewward
PPO
onlys
IRCT
DApger
4. Bg
TL-RI. I--RI:
5. BC+PPO (sbalic)
6. BC-4 PPO
[A] Triming sicpe
(b) Tninen Mope (tillion))
[6] Tratning stopp (aciliena)
(4) Transe stope (nalliioe))
-
8. B PPO
Desomtatika-aaa
RobnntBewardddh
9. B
Be PPO
ADVISOR Based (oum)
13 f ADW
BC ADV
(1)
(9)
(1/
AnV PPO
Figure 5: Evaluation following [15). Plots for 15 training routines in four selected tasks (additional plots in appendix). For clarity, we highlight the best performing training routine within five categories, e.g., RL only, IL only etc. (details in Sec. 4.2) with all other plots shaded lighter. (a)-(d) As described in Sec. 4.3 we plot RobustRewardee @ 10 at multiple points during training. (e)-(f) Plots of RobustReward@@ N for values of N E {l..... 45 Recall that RobustReward is the expected validation reward of best-found model when allowed N random hyperparameter evaluations.
Tasks +
PointGoal Navigation
ObjectGoal Navigation
Cooperative Navigation Reward @ 100%
Success
Success
Training routines
@ @10%
@ 100%
@10%
@ 100%
@ 10%
@ 100%
@10%
@ 100%
@ 10%
RL only
30.9
54.7
$4.7
79.0
6.7
13.1
11.1
31.6
-661.8
-456.0
IL only
30.1
68.7
35.5
76.7
3.8
8.8
13.6
-460.3
-416.7
II. RL static
48.9
71.5
56.7
78.2
6.5
11.3
11.7
19.8
-475.5
-424.6
ADVISOR (ours)
57.7
77.1
67.3
88.2
11.9
14.1
22.7
29.9
419.9
405.6
Table Quantitative results for high-fidelitg visual environments and continuous control. Val-
idation set performance after 10% and 100% of training has completed for four training routines on the POINTNAV, OBJECTNAV, and COOPNAv tasks (specifics of these routines can be found in the Appendix). For POINTNAV and OBjECTNAV we include the common success weighted path length (SPL) metric [I] addition to the success rate.
reporting, for each baseline, an estimate of RobustRewarrd = [Vall. reward of best model from & random hyperparam. evaluations) (4)
for < k v 45. For this we must train 50 models per method, i.e., 750 for each of these nine tasks. In order to show learning curves over training steps we also report RobustReward 10 at points during training. More details in Appendix A.9 For 2D-LH, we tune the hps of a competing method and use these hps for all other methods.
Training. For the eight MINIGRID tasks, we train each of the 50 training runs for I million steps. For 2D-LH/PD, models saturate much before 3- 10 steps. POINTNAV, OBJECTNAV. and COOPNAV are trained for standard budgets of 50Mn, 100Mn, and 1 1.5MM steps. Details are in Appendix A.10 Metries. We record standard metrics for each task. This includes avg. rewards (PD, MINIGRID tasks, and OBJECTNAV), and avg. episode lengths (2D-LH). Following visual navigation works [ [54 [4].
we report success rates and success- weightedd path length (SPL) for POINTNAV and OBJECTNAV. In
the following, we report a subset of the above and defer additional plots to Appendix A.11
4.4 Results
In the following, include takeaways based the results in Fig. 5 Fig. 6
Tab. and Tab.
Smaller imitation gap = better performance. central claim of our paper that the imitation gap is not merely a theoretical concern: the degree to which the teacher is privi- leged over the student has significant impact on the student's performance. To study this em- pirically, we vary the degree to which teachers are privileged over its students in 2D-LH

---

task. In particular, we use behavior cloning to train an f f'-restricted policy (i.e., an agent that can see grid locations away) using an fo-optimal teacher 25 times. Each policy is then evaluated on 200 random episodes and the average episode length (lower being better) recorded. For select 1. pairs we show boxplots of the 25 average episode lengths in Fig. 5 See our appendix for similar plots when using other training routines (e.g., ADVISOR).
Grey vertical lines show optimal average episode lengths for fi-restricted policies. We find that training an f'-restricted policy with an fl-expert results in near optimal policy when
View Rudins of Tescher (f)
Botser performauces
View Radius of Student Agont 160
3, 15
i j but even small increases in j dramati- cally decrease performance. While performance tends to drop with increasing j. the largest i, gaps do not consistently correspond to the worst performing models. While this seems to differ from our results in Ex. 2 recall that there the policy was fixed while here it varies through training, resulting in complex learning dynam- ics. Surprisingly we also find that, even when there is no imitation gap (e.g., the ADVISOR can outperform BC, see A.6.A.6
j = 5.7....1 hastation Cup
Smaller
7,9....11
9, 13,15
114
13,13,15
200
400
600
800
Avg. Ep. Length for BC
Figure 6: The size of the imitation gap directly impacts performance (in 2D-LH).
case),
ADVISOR outperforms, even in complex visual environments. Across all of our tasks, ADVISOR-base methods perform as well or better than competing methods. In particular, see Tab. [ for our results the POISONED (PD) and MINIGRID tasks and Tab. 2 for our results on the POINTNAV, OBJECTNAV, and COOPNAV tasks. 2D-LH results are deferred to the Appendix.
While the strong performance of ADVISOR is likely expected on our PD, MINIGRID, and 2D-LH tasks (indeed we designed a subset these with the explicit purpose of studying the imitation gap), it is nonetheless surprising to see that in the PD and LC ONCE SWITCHI tasks, all non-ADVISOR methods completely fail. Moreover, it is extremely promising to see that ADVISOR can provide substantial benefits in variety of standard tasks, namely OBJECTNAV, POINTNAV. and COOPNAv with limited visible range. Note that OBJECTÑAV and POINTNAV are set in 3D high-fidelity visual environments while COOPNAV requires multi-agent collaboration in a continuous space.
ADVISOR is sample efficient. To understand the sample efficiency of ADVISOR, we plot valida- tion set performance over training of select tasks (see Figures 5a to 3d and, in Table 2 we show performance of our models after 10% of training has elapsed for the OBJECTNAV, POINTNAV, and COOPNAV tasks. Note that in Table 2 ADVISOR trained models frequently reach better performance after 10% of training than other methods manage to reach by the end of training.
ADVISOR is robust. Rigorously studying sensitivity to hyperparameter choice requires retraining every method under consideration tens to hundreds of times. This computational task can make evaluating our methods on certain tasks infeasible (training single POINTNAV or OBjECTNAV model can easily require GPU-week of computation). Because of these computational constraints, we limit our study of robustness to the PD and MINIGRID tasks. In Figures 5e to 5h (additional results in Appendix) we plot, for each of the 15 evaluated methods, how the expected performance of each method behaves as we increase the budget of random hyperparameter evaluations. In general, relatively few hyperparameter evaluations are required for ADVISOR before high performance model is expected to be found.
Expert demonstrations can be critical to success. While it is frequently assumed that on on-policy expert supervision is better than learning from off-policy demonstrations, we found several instances in our MINIGRID experiments where demonstration based methods outperformed competing methods. See, for example, Figures 55 and 5t In such cases our demonstration--asee ADVISOR variant (see Appendix A.7 for details) performed very well.
ADVISOR helps even when the expert is corrupted. In LC CORRUPT EXPERT and WC CORRUPT EXPERT, where the expert is designed to be corrupted (outputting random actions as supervision) when the agent gets sufficiently close to the goal. While ADVISOR was not designed with the possibility of corrupted experts in mind, Figures 5d and 5h can succeed despite this corruption.
(see also Table show that ADVISOR

---

5 Conclusion
We propose the imitation gap as one explanation for the empirical observation that imitating "more intelligent" teachers can lead to worse policies. While prior work has, implicitly, attempted to bridge this gap, we introduce a principled adaptive weighting technique (ADVISOR), which we test on suite of thirteen tasks. Due to the fast rendering speed of MINIGRID, PD, and 2D-LH, could undertake a study where we trained over 6 billion steps, to draw statistically significant inferences.
Limitations and Societal Impact
While have attempted robustly evaluate our proposed ADVISOR methodology, we have primarily focused our experiments on navigational tasks where shortest path experts can be quickly computed. Further work needed to validate that ADVISOR can be successful in other domains, e.g., imitation in interactive robotic tasks or natural language applications.
While the potential for direct negative societal impact of this work is small, it is worth noting that, in enabling agents to learn more effectively from expert supervision, this work makes imitation learning more attractive option RL researchers. If expert supervision is obtained from humans, RL agents trained with such data will inevitably reproduce any (potentially harmful) biases of these humans.
Acknowledgements
This material is based upon work supported in part by the National Science Foundation under Grants No. 1563727, 1718221, 1637479, 165205, 1703166, 2008387, 2045586, 21 2106825, MRI #1 #1725729, NIFA award 2020-67021-327999 Samsung, 3M. Sloan Fellowship, NVIDIA Artificial Intelligence Lab, Allen Institute for AI, Amazon, AWS Research Awards, and Siebel Scholars Award. We thank Nan Jiang and Tanmay Gangwani for feedback on this work.
References
[1] Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, Koltun, Kosecka, Malik, R. Mottaghi, M. Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv: nen:::007.05555 2018.
[2] M. Bain and C. Sammut. A framework for behavioural cloning. In Machine Intelligence, 1995.
[3] M. G. Bellemare, G. Ostrovski, A. Guez, P. S. Thomas, and R. Munos. Increasing the action gap: New operators for reinforcement learning. In AAAI, 2016.
[4] Brys, A. Harutyunyan, H. B. Suay, S. Chernova, M. F. Taylor, and Nowé. Reinforcement learning from demonstration through shaping. In Q. Yang and M. J. Wooldridge, editors, IJCAI, 2015.
[5] K.-W. Chang, A. Krishnamurthy, A. Agarwal, H. Daume, and Langford. Learning to search better than your teacher. In ICML. 2015.
[6] Chemali and A. Lazaric. Direct policy iteration with demonstrations. In Q. Yang and M. J. Wooldridge, editors, IJCAI, 2015.
[7] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühh.. Learning by cheating. In Conference Robot Learning, 2020.
[8] M. Chevalier-Boisvert,, D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and Y. Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. In ICLR, 2018.
[9] M. Chevalicr-Boisver,, L. Willems, and S. Pal. Minimalistic gridworld environment for openai gym. https://gitte httt::://ggiuuub.ccm//xeeeeee www.///aiimeb////oooooooooo glggggygmmmmnign 2018.
[10] Codevilla, M. Müller, A. M. López, V. Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In ICRA, 2018.
10

---

[11] A. Das, S. Kottur, J. M. Moura, S. Lee, and D. Batra. Learning cooperative visual dialog agents with deep reinforcement learning. In ICCV, 2017.
[12] A. Das, S. Datta, G. Gkioxari, Lee, D. Parikh, and D. Batra. Embodied Question Answering. In CVPR, 2018.
[13] A. Das, G. Gkioxari, S. Lee, D. Parikh, and D. Batra. Neural Modular Control for Embodied Question Answering. In CoRL, 2018.
[14] M. Deitke, W. Han, A. Herrasti, A. Kembhavi, E. Kolve, R. Mottaghi, J. Salvador, D. Schwenk, E. VanderBilt, M. Wallingford, L. Weihs, M. Yatskar, and A. Farhadi. RoboTHOR: An Open Comulation-to-Red Embodied AI Platform. In CVPR, 2020.
[15] J. Dodge, S. Gururangan, D. Card, R. Schwartz, and N. A. Smith. Show your work: Improved reporting of experimental results. In EMNLP, 2019.
[16] T. Gangwani and J. Peng. State-only imitation with transition dynamics mismatch. In ICLR, 2020.
[17] T. Gangwani, J. Lehman, Q. Liu, and J. Peng. Learning belief representations for imitation learning in pomdps. In A. Globerson and R. Silva, editors, UAI, 2019.
[18] A. Gupta, C. Devin, Y. Liu, P. Abbeel, and S. Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. In ICLR, 2017.
[19] S. Gupta, J. Davidson, Levine, R. Sukthankar, and Malik. Cognitive Mapping and Planning for Visual Navigation. In CVPR, 2017.
[20] Hawke, R. Shen, C. Gurau, S. Sharma, D. Reda, N. Nikolov, P. Mazur, S. Micklethwaite. N. Griffiths, A. Shah, and A. Kendall. Urban driving with conditional imitation learning. In ICRA, 2020.
[21] D. He, Y. Xia, T. Qin, Wang. N. Yu, T.-Y, Liu, and W.-Y. Ma. Dual learning for machine translation. In NeurlPS, 2016.
[22] N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint 1rXiv:170..02288 2017.
[23] T. Hester, M. Vecerík, O. Pietquin, M. Lanetot, T. Schaul, B. Piot, D. Horgan, Quan, A. Sendonaris, I. Osband, et al. Deep q-learning from demonstrations. In AAAI, 2018.
[24] J. Ho and S. Ermon. Generative adversarial imitation learning, In NeurIPS, 2016.
[25] M. Jaderberg, V. Mnih, W. M. Czamecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICLR, 2017.
[26] N. Jiang. On value functions and the agent-environment boundary. arXiv preprint arXiv: 133411 2019.
[27] M. Jing, X. Ma. W. Huang, F. Sun, C. Yang, B. Fang, and H. Liu. Reinforcement learning from imperfect demonstrations under soft expert guidance. In AAAI, 2020.
[28] B. Kang, 7. Jie, and J. Feng. Policy optimization with demonstrations. In ICMI, 2018.
[29] D. Kingma and J. Ba. A method for stochastic optimization. In CVPR, 2017.
[30] J. Kober and J. R. Peters. Policy search for motor primitives in robotics. In NeurIPS, 2009.
[31] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. A12-THOR: an interactive 3d environment for visual AI. arXiv preprint arXiv: 17Xiv:1712.004474 2019.
[32] Kostrikov. Pytorch implemeetionnnn of reinforcement learning algorithms.
com hamp://kosrrikeorrpotrrddd----ee acktr gaill 2018

---

[33] H. Le. N. Jiang, A. Agarwal, M. Dudik, Y. Yue, and H. Daumé. Hierarchical imitation and reinforcement learning. In ICML, 2018.
[34] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. JMLR, 2016.
[35] T. P. Lillierap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
[36] I.-J. Liu, R. Yeh, and A. G. Schwing. PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning. In CORL, 2019.
[37] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch. Multi-Agent Actor-Critic for Mixed Croperative-oompetiie Environments. In NewrIPS, 2017.
[38] R. Lowe, A. Gupta, J. N. Foerster, D. Kiela, and J. Pineau. On the interaction between supervision and self-play in emergent communication. In ICLR, 2020.
[39] A. R. Mahmood, H. P. van Hasselt, and R. S. Sutton. Weighted importance sampling for off oof-po learning with linear function approximation. In NeurlPS, 2014.
[40]
Mirowski, Pascanu, Viola, H. Soyer, A. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, et al. Learning to navigate in complex environments. In ICLR, 2017.
[41] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
[42] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap. T. Harley, Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ICML, 2016.
[43] I. Mordatch and P. Abbeel. Emergence of Grounded Compositional Language in Multi-Agent Populations. In AAAI, 2018.
[44] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In ICRA, 2018.
[45] T. Osa, J. Pajarinen, G. Neumann, A. Bagnell, P. Abbeel, and Peters. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 2018.
[46] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In ICML, 2017.
[47] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: ssample-guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 2018
[48] J. Peters and S. Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 2008.
[49] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 1991.
[50] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In RSS, 2018.
[51] S. Ross and D. Bagnell. Efficient reductions for imitation learning. In AISTATS, 2010.
[52] S. Ross, G. Gordon, and D. Bagnell. reduction imitation learning and structured prediction no-regret online learning. In AISTATS, 2011.
[53] C. Sammut, S. Hurst, D. Kedzier, and D. Michie. Learning to fly. In Machine Learning Proceedings, 1992.
12

---

[54] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: Platform for Embodied AI Research. In ICCV, 2019.
[55] T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized Experience Replay, In ICLR, 2016.
[56] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In ICML, 2015.
[57] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.0244: 2015.
[58] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint S.XKv:1707.0664777 2017.
[59] T. T. Shi, A. Karpathy, L. J. Fan, J. Hernandez, and P. Liang. World of bits: An open-domain platform for web-based agents. In ICML, 2017.
[60] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. CVPR, 2020.
[61] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonogiou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalch- brenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 2016.
[62] W. Sun, J. A. Bagnell, and Boots. Truncated horizon policy search: Combining reinforcement learning & imitation learning. In ICLR, 2018
[63] M. van der Laan and S. Gruber. One-step targeted minimum boss-based estimation based on universal least favorable one-dimensional submodels. The international journal of biostatistics, 12(1):351-378, 2016.
[64] A. van der Vaart. Asymptotic Statistics. Asymptotic Statistics. Cambridge University Press, 2000. ISBN 9780521784504 URL https:ttt:://oookt http://boooooooglllelloo e. wwt.commm/oo/oe.e Bbooks?id=UEuuENCC Wgg
[65] H. van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In AAAI, 2016.
[66] M. Vecerik, Hester, J. Scholz, Wang, O. Pietquin, B. Piot, Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv: 101.117777.88111 2017.
[67] X. Wang, W. Chen, J. Wu, Y.-F. Wang, and W. Yang Wang. Video captioning via hierarchical reinforcement learning. In CVPR, 2018
[68] Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, and N. de Freitas. Sample efficient actor-critic with experience replay. In ICIR, 2017.
[69] A. Warrington, J. W. Lavington, A. Scibior, M. Schmidt, and E. Wood. Robust asymmetric learning in pomdps. CoRR, 22 2020. URL http::ttt://aaat http:/vxvvvv....aaaaaea atgg/bbss/012 15566
Springer-Verlag. [70] L. Wasserman. All of Nonparametric Staristics (Springer Texts Statistics). Springer- Verlag, Berlin, Heidelberg. 2006. ISBN 0387251456.
[71] F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese. Gibson env: Real-world perception for embodied agents. In CVPR, 2018
[72] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, and N. Heess. Reinforcement and imitation learning for diverse visuomotor skills. In Proceedings of Robotics: Science and Systems, 2018
13


Reviews:
[{"review_id": "yxnMjRLZJB9", "reviewer": "Reviewer_JVEf", "summary": "This work introduces ADVISOR, a simple yet effective approach to adaptively combining RL with supervision from imitation learning. The approach is motivated by the issue of *information gaps,* which describes the case where the policy generating the supervision has more information about the state than does the policy being trained. The authors motivate their approach as a way to address this issue and contribute a thorough set of empirical analyses to demonstrate its effectiveness. In addition, this work provides a concrete demonstration that the information gap is an obstacle.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I consider this a strong paper. It clearly motivates the problem as well as the solution it proposes (which itself is attractively simple), making use of concrete demonstration throughout. It is one of the rare papers that seems to get *better* as you read it.\n\nI lack the specific expertise to fully evaluate the originality of the work, but, as best I can tell, it should be considered novel. At the very least, the combination of contributions makes the aggregate work in the paper original.\n\nThe ideas in the paper are presented quite clearly. The authors use examples well and provide clear formalizations where useful. The paper does not rely on the appendix, but it contains plenty of clarifying information nevertheless. The main results figure (Fig 5) is a bit hard to parse.\n\nA major strength of the paper is its experimental rigor and use of illustrative \"toy\" settings. The adherence to best practices makes the results more meaningful and impactful. The analysis presented in Figure 6 is particularly nice and helps to establish the significance of the information gap.\n\nOne issue with the paper is that it seems to be particularly focused on the settings where it would be possible to query the teacher policy at every state. Often, I would think, that is not the case. There is at least one experiment using ADVISOR with a fixed set of teacher demonstrations, so the method can handle this case. My sense is that it would strengthen the potential impact of the paper if deeper analysis were presented of how ADVISOR fares in these demonstration settings.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "7CTTyWbpkdb", "reviewer": "Reviewer_Rksz", "summary": "The paper approaches the problem of imitation gap. In situations where the teacher (an expert) has access to privileged information, the student may not have such comfort. Therefore, blindly imitating the teacher may lead to inefficient, unwanted or incorrect behavior of the student. The paper provides examples of such situations. Actually, sometimes the student may not even be able to imitate the teacher in some states as not all the information is available. \n\nThis leads to an imitation gap. Performing classic imitation learning in states where the imitation gap is significant can be pointless. The authors propose to try existing RL methods in such situations.\n\nSpecifically, the imitation loss and the RL loss are weighted by a normalized coefficient. One of the main contributions of the paper is how to dynamically determine the value of the coefficient. The idea is to in some sense 'measure' or 'quantify' the imitation gap in a given state. The authors follow the intuition that for states where the imitation gap is substantial, learning well the teacher's distribution over actions may not be achievable. It means the divergence between the teacher's distribution and the imitation distribution of actions will be relatively large. This quantity is used as an estimation of imitation gap.\n\nTo compute that quantity an approximation to the imitation policy is introduced. The divergence between the teacher and this approximation is used to estimate the imitation gap.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very clear to read and understand. The flow is logical and smooth. The problem is well stated and supported by a clear example. \n\nThe approach is precisely described. The notations is consistent and easy to comprehend. \n\nThe method is practical in the sense that it takes under consideration that computing the exact imitation policy is not possible. Instead, it offers a simple and efficient way of approximating it. On the other hand it is flexible as not-critical elements of the algorithm are abstracted away. For example, there are multiple possible distance functions between the teacher policy and the imitation policy (or its approximation).\n\nThe paper introduces a novel approach to estimating the coefficient between imitation learning and reinforcement learning for states that have a significant imitation gap.\n\nThe results are on the example problems and on more challenging environments. The proposed approach performs well in the basic and more complex circumstances.\n\nMy rating is based on the proposed approach but also on the significance of the problem itself.\n\nOf course it would be great to see the method evaluated on even more complex environments and challenges. Nevertheless, that probably will be the topic of future work. At this point the presented experiments illustrate the problem and the presented approach is able to make significant progress. \n\nOne comment is that it would be beneficial to include more analysis of the intuitions behind the imitation gap and the suggested way of quantifying it. For example, where does the quantification method perform well and what are the cases where it is imperfect? Why? What should be considered to improve over the proposed algorithm? What are the edge cases?\n\n\n\n\n\n   \n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper has a concrete chapter addressing limitations and social impact. It is clear that authors had these in their mind and addressed potential concerns.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "0YOTrtSD0SH", "reviewer": "Reviewer_Hfmt", "summary": "This paper points out the existence of \"imitation gap\" when a teacher and student policy has different observations. Due to this imitation gap, the student policy cannot determine which action leads to a higher reward, and thus ends up outputting a uniformly random action or average actions of the teacher policy.\n\nThe proposed method, ADVISOR, tackles such imitation gap by balancing between learning from reward and expert. When the limited observation is enough to predict the optimal action, the student policy imitates the demonstrations; otherwise, it learns from reward. As the weight function, ADVISOR uses the divergence between the teacher policy and the auxiliary policy, which solely learns from the demonstrations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: This paper identifies a new concept of \"imitation gap\", which can easily happen and degrade imitation learning performance, and proposes a simple method to get around the imitation gap.\n\n**Clarity**: The paper is clearly written with intuitive examples.\n\n**Quality**: The exhaustive experiments on diverse environments demonstrate the existence of the imitation gap and ADVISOR's superior performance under the imitation gap. \n\n**Significance**: Overall, the paper proves the importance of dealing with \"imitation gap\" and the proposed method effectively learns toy tasks under the imitation gap. This tells us that when we design an expert policy or collect expert demonstrations, we must be aware of the effect of such imitation gaps and try to minimize the gaps. However, it is not very clear how important this imitation gap is in complex environments, such as PointGoal and ObjectGoal navigation tasks, as the performance gap is not significant.\n\n\nHere are a few questions regarding the method and experiments.\n\n- The only imitation gap considered in this paper is about the observational difference between the teacher and student. However, the gap between the teacher and student could be often aroused from different dynamics, embodiments, and environment stochasticity. Can this proposed method handle such gaps as well, especially under stochastic environments?\n- The experiments are mostly focused on navigation. A few experiments on common continuous control benchmarks (locomotion or manipulation) could make the paper stronger.\n- The key of ADVISOR is to ignore the expert policy on states which are not reproducible in the student's observation space. This highly relies on the accurate auxiliary policy. To learn a more accurate auxiliary policy, use of a replay buffer can be more appropriate. What is the rationale of using on-policy data, which can make the auxiliary policy biased toward recent trajectories?\n- The weight function used in ADVISOR is somewhat complicated and requires two additional hyperparameters to be examined. Is there any benefit or explanation of the use of this complex function compared to simple linear, quadratic, or exponential ones?\n\n### Minor comments and questions\n\n- In some places, the white spaces between lines and figures are too compact to read comfortably.\n- In Table 2, \"e\" is added to the second column.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was a strong consensus among reviewers that this paper should be accepted. The paper proposes a method to address the imitation gap - the setting where the expert has access to privileged information making imitation impossible for the student. The paper shows both the efficacy of the method and the importance of the imitation gap for performance. They only point of improvement would be to look at more diverse environment. Currently the paper is focused on visual navigation.
Decision: Accept (Poster)