Bridging the Imitation Gap by Adaptive Insubordination

1 Introduction . Imitation learning ( IL ) can be remarkably successful in settings where reinforcement learning ( RL ) struggles . For instance , IL has been shown to succeed in complex tasks with sparse rewards [ 8 , 47 , 44 ] , and when the observations are high-dimensional , e.g. , in visual 3D environments [ 31 , 54 ] . To succeed , IL provides the agent with consistent expert supervision at every timestep , making it less reliant on the agent randomly attaining success . To obtain this expert supervision , it is often convenient to use “ privileged information , ” i.e. , information that is unavailable to the student at inference time . This privileged information takes many forms in practice . For instance , in navigational tasks , experts are frequently designed using shortest path algorithms which access the environment ’ s connectivity graph [ e.g. , 19 ] . Other forms of privilege include semantic maps [ e.g. , 60 , 13 ] , the ability to see into “ the future ” via rollouts [ 61 ] , and ground-truth world layouts [ 7 ] . The following example shows how this type of privileged information can result in IL dramatically failing . Example 1 ( Poisoned Doors ) . Suppose an agent is presented with N 3 doors d1 , . . . , dN . As illustrated in Fig . 1 ( for N = 4 ) , opening d1 requires entering an unknown fixed code of length M . Successful code entry results in a reward of 1 , otherwise the reward is 0 . Since the code is unknown to the agent , it would need to learn the code by trial and error . All other doors can be opened without a code . For some randomly chosen 2 j N ( sampled each episode ) , the reward behind dj is 2 but for all i 2 { 2 , . . . , N } \ { j } the reward behind di is 2 . Without knowing j , the optimal policy is to always enter the correct code to open d1 obtaining an expected reward of 1 . In contrast , if the expert ⇤denotes equal contribution by LW and UJ ; †work done , in part , as an intern at Allen Institute for AI 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is given the privileged knowledge of the door dj with reward 2 , it will always choose to open this door immediately . It is easy to see that an agent without knowledge of j attempting to imitate such an expert will learn to open a door among d2 , . . . , dN uniformly at random obtaining an expected return of 2 · ( N 3 ) / ( N 1 ) . In this setting , training with reward-based RL after a ‘ warm start ’ with IL is strictly worse than starting without it : the agent needs to unlearn its policy and then , by chance , stumble into entering the correct code for door d1 , a practical impossibility when M is large . To characterize this imitation failure , we show that training a student to imitate a teacher who uses privileged information results in the student learning a policy which marginalizes out this privileged information . This can result in a sub-optimal , even uniformly random , student policy over a large collection of states . We call the discrepancy between the teacher ’ s and student ’ s policy the imitation gap . To bridge the imitation gap , we introduce Adaptive Insubordination ( ADVISOR ) . ADVISOR adaptively weights imitation and RL losses . Specifically , throughout training we use an auxiliary actor which judges whether the current observation is better treated using an IL or a RL loss . For this , the auxiliary actor attempts to reproduce the teacher ’ s action using the observations of the student at every step . Intuitively , the weight corresponding to the IL loss is large when the auxiliary actor can reproduce the teacher ’ s action with high confidence . We study the benefits of ADVISOR on thirteen tasks , including ‘ POISONEDDOORS ’ from Ex . 1 , a 2D “ lighthouse ” gridworld , a suite of tasks set within the MINIGRID environment [ 8 , 9 ] , Cooperative Navigation with limited range ( COOPNAV ) in the multi-agent particle environment ( MPE ) [ 43 , 38 ] , and two navigational tasks set in 3D , high visual fidelity , simulators of real-world living environments ( POINTNAV in AIHABITAT [ 54 ] and OBJECTNAV in ROBOTHOR [ 31 , 14 ] ) . Our results show that , • the imitation gap ’ s size directly impacts agent performance when using modern learning methods , • ADVISOR is performant ( outperforming IL and RL baselines ) , robust , and sample efficient , • ADVISOR can succeed even when expert supervision is partially corrupted , and • ADVISOR can be easily integrated in existing pipelines spanning diverse observations ( grids and pixels ) , actions spaces ( discrete and continuous ) , and algorithms ( PPO and MADDPG ) . 2 Related Work . A series of methods [ e.g. , 41 , 65 , 3 , 55 ] have made off-policy deep Q-learning stable for complex environments like Atari Games . Several high-performance ( on-policy ) policy-gradient methods for deep-RL have also been proposed [ 56 , 42 , 34 , 68 , 61 ] . For instance , Trust Region Policy Optimization ( TRPO ) [ 56 ] improves sample-efficiency by safely integrating larger gradient steps . Proximal Policy Optimization ( PPO ) [ 58 ] employs a clipped variant of TRPO ’ s surrogate objective and is widely adopted in the deep RL community . We use PPO as a baseline in our experiments . As environments get more complex , navigating the search space with only deep RL and simple heuristic exploration ( such as ✏-greedy ) is increasingly difficult . Therefore , methods that imitate expert ( i.e. , teacher ) supervision were introduced . A popular approach to imitation learning ( IL ) is Behaviour Cloning ( BC ) , i.e. , use of a supervised classification loss between the policy of the student and expert agents [ 53 , 2 ] . However , BC suffers from compounding errors . Namely , a single mistake of the student may lead to settings that have never been observed in training [ 51 ] . To address this , Data Aggregation ( DAgger ) [ 52 ] trains a sequence of student policies by querying the expert at states beyond those that would be reached by following only expert actions . IL is further enhanced by , e.g. , hierarchies [ 33 ] , improving over the expert [ 5 , 4 , 27 ] , bypassing any intermediate reward function inference [ 24 ] , and/or learning from experts that differ from the student [ 18 , 26 , 16 ] . Importantly , a sequential combination of IL and RL , i.e. , pre-training a model on expert data before letting the agent interact with the environment , performs remarkably well . This strategy has been applied in a wide range of applications – the game of Go [ 61 ] , robotic and motor skills [ 49 , 30 , 48 , 50 ] , navigation in visually realistic environments [ 19 , 12 ] , and web & language based tasks [ 21 , 11 , 59 , 67 ] . More recent methods mix expert demonstrations with the agent ’ s own rollouts instead of using a sequential combination of IL followed by RL . Chemali and Lazaric [ 6 ] perform policy iteration from expert and on-policy demonstrations . DQfD [ 23 ] initializes the replay buffer with expert episodes and adds rollouts of ( a pretrained ) agent . They weight experiences based on the previous temporal difference errors [ 55 ] and use a supervised loss to learn from the expert . For continuous action spaces , DDPGfD [ 66 ] analogously incorporates IL into DDPG [ 35 ] . POfD [ 28 ] improves by adding a demonstration-guided exploration term , i.e. , the Jensen-Shannon divergence between the expert ’ s and the learner ’ s policy ( estimated using occupancy measures ) . THOR uses suboptimal experts to reshape rewards and then searches over a finite planning horizon [ 62 ] . Zhu et al . [ 72 ] show that a combination of GAIL [ 24 ] and RL can be highly effective for difficult manipulation tasks . Critically , the above methods have , implicitly or explicitly , been designed under certain assumptions ( e.g. , the agent operates in an MDP ) which imply the expert and student observe the same state . Different from the above methods , we investigate the difference of privilege between the expert policy and the learned policy . Contrary to a sequential , static , or rule-based combination of supervised loss or divergence , we train an auxiliary actor to adaptively weight IL and RL losses . To the best of our knowledge , this hasn ’ t been studied before . In concurrent work , Warrington et al . [ 69 ] address the imitation gap by jointly training their teacher and student to adapt the teacher to the student . For our applications of interest , this work is not applicable as our expert teachers are fixed . Our approach attempts to reduce the imitation gap directly , assuming the information available to the learning agent is fixed . An indirect approach to reduce this gap is to enrich the information available to the agent or to improve the agent ’ s memory of past experience . Several works have considered this direction in the context of autonomous driving [ 10 , 20 ] and continuous control [ 17 ] . We expect that these methods can be beneficially combined with the method that we discuss next . 3 ADVISOR . We first introduce notation to define the imitation gap and illustrate how it arises due to ‘ policy averaging. ’ Using an ‘ auxiliary policy ’ construct , we then propose ADVISOR to bridge this gap . Finally , we show how to estimate the auxiliary policy in practice using deep networks . In what follows we will use the terms teacher and expert interchangeably . Our use of “ teacher ” is meant to emphasize that these policies are ( 1 ) designed for providing supervision for a student and ( 2 ) need not be optimal among all policies . 3.1 Imitation gap . We want an agent to complete task T in environment E . The environment has states s 2 S and the agent executes an action a 2 A at every discrete timestep t 0 . For simplicity and w.l.o.g . assume both A and S are finite . For example , let E be a 1D-gridworld in which the agent is tasked with navigating to a location by executing actions to move left or right , as shown in Fig . 2a . Here and below we assume states s 2 S encapsulate historical information so that s includes the full trajectory of the agent up to time t 0 . The objective is to find a policy ⇡ , i.e. , a mapping from states to distributions over actions , which maximizes an evaluation criterion . Often this policy search is restricted to a set of feasible policies ⇧feas. , for instance ⇧feas . may be the set { ⇡ ( · ; ✓ ) : ✓ 2 RD } where ⇡ ( · ; ✓ ) is a deep neural network with D-dimensional parameters ✓ . In classical ( deep ) RL [ 41 , 42 ] , the evaluation criterion is usually the expected -discounted future return . We focus on the setting of partially-observed Markov decision processes ( POMDPs ) where an agent makes decisions without access to the full state information . We model this restricted access by defining a filtration function f : S ! Of and limiting the space of feasible policies to those policies ⇧feas.f for which the value of ⇡ ( s ) depends on s only through f ( s ) , i.e. , so that f ( s ) = f ( s 0 ) implies ⇡ ( s ) = ⇡ ( s0 ) . We call any ⇡ satisfying this condition an f -restricted policy and the set of feasible f -restricted policies ⇧feas.f . In a gridworld example , f might restrict s to only include information local to the agent ’ s current position as shown in Figs . 2c , 2d . If a f -restricted policy is optimal among all other f -restricted policies , we say it is f -optimal . We call o 2 Of a partial-observation and for any f -restricted policy ⇡f we write ⇡f ( o ) to mean ⇡f ( s ) if f ( s ) = o . It is frequently the case that , during training , we have access to a teacher policy which is able to successfully complete the task T . This teacher policy may have access to the whole environment state and thus may be optimal among all policies . Alternatively , the teacher policy may , like the student , only make decisions given partial information ( e.g. , a human who sees exactly the same inputs as the student ) . For flexibility we will define the teacher policy as ⇡teachf teach , denoting it is an f teach-restricted policy for some filtration function A A Goal Boundary Agent 1 . 2 . A A A A 0 . Random start 1 . Move right · · · An . Goal reached Hypothetical episode 2 . Move right 3 . Move left 0.0 1.0 0.0 1.0 0.0 1.0 0.0 1.0 ( a ) ( b ) ( c ) ( d ) Environment E start states Actions A = { left , right } = { L , R } L R A A A A· · · A f 2-partial obs . 0.5 0.5 0.5 0.5 0.0 1.0 0.0 1.0 L R ⇡ IL f2 A A A A A· · · f 1-partial obs . 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 L R ⇡ IL f1⇡ teach < latexit sha1_base64= '' Vl/iaP1zJc4vIB4jjoXAxI2rjqE= '' > AAACEXicbVDLSsNAFJ3UV62vWpdugkVwVRJRdFl047KCfUAby2R60w6dZMLMjbSEfIV7t/oL7sStX+Af+BlOHwvbeuDC4Zx7OZfjx4JrdJxvK7e2vrG5ld8u7Ozu7R8UD0sNLRPFoM6kkKrlUw2CR1BHjgJasQIa+gKa/vB24jefQGkuowccx+CFtB/xgDOKRuoWS52YP6YdhBGmCJQNsqxbLDsVZwp7lbhzUiZz1LrFn05PsiSECJmgWrddJ0YvpQo5E5AVOomGmLIh7UPb0IiGoL10+ntmnxqlZwdSmYnQnqp/L1Iaaj0OfbMZUhzoZW8i/ue1EwyuvZRHcYIQsVlQkAgbpT0pwu5xBQzF2BDKFDe/2mxAFWVo6lpI8aUcIvV1VjDVuMtFrJLGecW9qFzeX5SrN/OS8uSYnJAz4pIrUiV3pEbqhJEReSGv5M16tt6tD+tztpqz5jdHZAHW1y8vtp5g < /latexit > Figure 2 : Effect of partial observability in a 1-dimensional gridworld environment . ( a ) The two start states and actions space for 1D-Lighthouse with N = 4 . ( b ) A trajectory of the agent following a hypothetical random policy . At every trajectory step we display output probabilities as per the shortest-path expert ( ⇡teach ) for each state . ( c/d ) Using the same trajectory from ( b ) we highlight the partial-observations available to the agent ( shaded gray ) under different filtration function f1 , f2 . Notice that , under f1 , the agent does not see the goal within its first four steps . The policies ⇡ILf1 , ⇡ IL f2 , learned by imitating ⇡teach , show that imitation results in sub-optimal policies , i.e. , ⇡ILf1 , ⇡ IL f2 6= ⇡ teach . f teach . For simplicity , we will assume that ⇡teachf teach is f teach-optimal . Subsequently , we will drop the subscript f teach unless we wish to explicitly discuss multiple teachers simultaneously . In IL [ 45 , 52 ] , ⇡f is trained to mimic ⇡teach by minimizing the ( expected ) cross-entropy between ⇡f and ⇡teach over a set of sampled states s 2 S : min ⇡f2⇧feas.f Eµ [ CE ( ⇡teach , ⇡f ) ( S ) ] , ( 1 ) where CE ( ⇡teach , ⇡f ) ( S ) = ⇡teach ( S ) log ⇡f ( S ) , denotes the usual dot-product , and S is a random variable taking values s 2 S with probability measure µ : S ! [ 0 , 1 ] . Often µ ( s ) is chosen to equal the frequency with which an exploration policy ( e.g. , random actions or ⇡teach ) visits state s in a randomly initialized episode . When it exists , we denote the policy minimizing Eq . ( 1 ) as ⇡µ , ⇡ teach f . When µ and ⇡teach are unambiguous , we write ⇡ILf = ⇡ µ , ⇡teach f . What happens when there is a difference of privilege ( or filtration functions ) between the teacher and the student ? Intuitively , if the information that a teacher uses to make a decision is unavailable to the student then the student has little hope of being able to mimic the teacher ’ s decisions . As we show in our next example , even when optimizing perfectly , depending on the choice of f and f teach , IL may result in ⇡ILf being uniformly random over a large collection of states . We call the phenomenon that ⇡ IL f 6= ⇡ teach the imitation gap . Example 2 ( 1D-Lighthouse ) . We illustrate the imitation gap using a gridworld spanning { N , . . . , N } . The two start states correspond to the goal being at either N or N , while the agent is always initialized at 0 ( see Fig . 2a ) . Clearly , with full state information , ⇡teach maps states to an ‘ always left ’ or ‘ always right ’ probability distribution , depending on whether the goal is on the left or right , respectively . Suppose now that the agent ’ s visibility is constrained to a radius of i ( Fig . 2c shows i = 1 ) , i.e. , an f i-restricted observation is accessible . An agent following an optimal policy with a visibility of radius i will begin to move deterministically towards any corner , w.l.o.g . assume right . When the agent sees the rightmost edge ( from position N i ) , it will either continue to move right if the goal is visible or , if it ’ s not , move left until it reaches the goal ( at N ) . Now we may ask : what is the best f i-restricted policy that can be learnt by imitating ⇡teach ( i.e. , what is ⇡ILfi ) ? Tragically , the cross-entropy loss causes ⇡ IL fi to be uniform in a large number of states . In particular , an agent following policy ⇡ILfi will execute left ( and right ) actions with probability 0.5 , until it is within a distance of i from one of the corners . Subsequently , it will head directly to the goal . See the policies highlighted in Figs . 2c , 2d . The intuition for this result is straightforward : until the agent observes one of the corners it can not know if the goal is to the right or left and , conditional on its observations , each of these events is equally likely under µ ( assumed uniform ) . Hence for half of these events the teacher will instruct the agent to go right . For the other half the instruction is to go left . See App . A.1 for a rigorous treatment of this example . In Sec . 4 and Fig . 6 , we train f i-restricted policies with f j-optimal teachers for a 2D variant of this example . We empirically verify that a student learns a better policy when imitating teachers whose filtration function is closest to their own . The above example shows : when a student attempts to imitate an expert that is privileged with information not available to the student , the student learns a version of ⇡teach in which this privileged information is marginalized out . We formalize this intuition in the following proposition . Proposition 1 ( Policy Averaging ) .

Summary: The paper approaches the problem of imitation gap. In situations where the teacher (an expert) has access to privileged information, the student may not have such comfort. Therefore, blindly imitating the teacher may lead to inefficient, unwanted or incorrect behavior of the student. The paper provides examples of such situations. Actually, sometimes the student may not even be able to imitate the teacher in some states as not all the information is available.   This leads to an imitation gap. Performing classic imitation learning in states where the imitation gap is significant can be pointless. The authors propose to try existing RL methods in such situations.  Specifically, the imitation loss and the RL loss are weighted by a normalized coefficient. One of the main contributions of the paper is how to dynamically determine the value of the coefficient. The idea is to in some sense 'measure' or 'quantify' the imitation gap in a given state. The authors follow the intuition that for states where the imitation gap is substantial, learning well the teacher's distribution over actions may not be achievable. It means the divergence between the teacher's distribution and the imitation distribution of actions will be relatively large. This quantity is used as an estimation of imitation gap.  To compute that quantity an approximation to the imitation policy is introduced. The divergence between the teacher and this approximation is used to estimate the imitation gap.