Bridging the Imitation Gap by Adaptive Insubordination

1 Introduction . Imitation learning ( IL ) can be remarkably successful in settings where reinforcement learning ( RL ) struggles . For instance , IL has been shown to succeed in complex tasks with sparse rewards [ 8 , 47 , 44 ] , and when the observations are high-dimensional , e.g. , in visual 3D environments [ 31 , 54 ] . To succeed , IL provides the agent with consistent expert supervision at every timestep , making it less reliant on the agent randomly attaining success . To obtain this expert supervision , it is often convenient to use “ privileged information , ” i.e. , information that is unavailable to the student at inference time . This privileged information takes many forms in practice . For instance , in navigational tasks , experts are frequently designed using shortest path algorithms which access the environment ’ s connectivity graph [ e.g. , 19 ] . Other forms of privilege include semantic maps [ e.g. , 60 , 13 ] , the ability to see into “ the future ” via rollouts [ 61 ] , and ground-truth world layouts [ 7 ] . The following example shows how this type of privileged information can result in IL dramatically failing . Example 1 ( Poisoned Doors ) . Suppose an agent is presented with N 3 doors d1 , . . . , dN . As illustrated in Fig . 1 ( for N = 4 ) , opening d1 requires entering an unknown fixed code of length M . Successful code entry results in a reward of 1 , otherwise the reward is 0 . Since the code is unknown to the agent , it would need to learn the code by trial and error . All other doors can be opened without a code . For some randomly chosen 2 j N ( sampled each episode ) , the reward behind dj is 2 but for all i 2 { 2 , . . . , N } \ { j } the reward behind di is 2 . Without knowing j , the optimal policy is to always enter the correct code to open d1 obtaining an expected reward of 1 . In contrast , if the expert ⇤denotes equal contribution by LW and UJ ; †work done , in part , as an intern at Allen Institute for AI 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is given the privileged knowledge of the door dj with reward 2 , it will always choose to open this door immediately . It is easy to see that an agent without knowledge of j attempting to imitate such an expert will learn to open a door among d2 , . . . , dN uniformly at random obtaining an expected return of 2 · ( N 3 ) / ( N 1 ) . In this setting , training with reward-based RL after a ‘ warm start ’ with IL is strictly worse than starting without it : the agent needs to unlearn its policy and then , by chance , stumble into entering the correct code for door d1 , a practical impossibility when M is large . To characterize this imitation failure , we show that training a student to imitate a teacher who uses privileged information results in the student learning a policy which marginalizes out this privileged information . This can result in a sub-optimal , even uniformly random , student policy over a large collection of states . We call the discrepancy between the teacher ’ s and student ’ s policy the imitation gap . To bridge the imitation gap , we introduce Adaptive Insubordination ( ADVISOR ) . ADVISOR adaptively weights imitation and RL losses . Specifically , throughout training we use an auxiliary actor which judges whether the current observation is better treated using an IL or a RL loss . For this , the auxiliary actor attempts to reproduce the teacher ’ s action using the observations of the student at every step . Intuitively , the weight corresponding to the IL loss is large when the auxiliary actor can reproduce the teacher ’ s action with high confidence . We study the benefits of ADVISOR on thirteen tasks , including ‘ POISONEDDOORS ’ from Ex . 1 , a 2D “ lighthouse ” gridworld , a suite of tasks set within the MINIGRID environment [ 8 , 9 ] , Cooperative Navigation with limited range ( COOPNAV ) in the multi-agent particle environment ( MPE ) [ 43 , 38 ] , and two navigational tasks set in 3D , high visual fidelity , simulators of real-world living environments ( POINTNAV in AIHABITAT [ 54 ] and OBJECTNAV in ROBOTHOR [ 31 , 14 ] ) . Our results show that , • the imitation gap ’ s size directly impacts agent performance when using modern learning methods , • ADVISOR is performant ( outperforming IL and RL baselines ) , robust , and sample efficient , • ADVISOR can succeed even when expert supervision is partially corrupted , and • ADVISOR can be easily integrated in existing pipelines spanning diverse observations ( grids and pixels ) , actions spaces ( discrete and continuous ) , and algorithms ( PPO and MADDPG ) . 2 Related Work . A series of methods [ e.g. , 41 , 65 , 3 , 55 ] have made off-policy deep Q-learning stable for complex environments like Atari Games . Several high-performance ( on-policy ) policy-gradient methods for deep-RL have also been proposed [ 56 , 42 , 34 , 68 , 61 ] . For instance , Trust Region Policy Optimization ( TRPO ) [ 56 ] improves sample-efficiency by safely integrating larger gradient steps . Proximal Policy Optimization ( PPO ) [ 58 ] employs a clipped variant of TRPO ’ s surrogate objective and is widely adopted in the deep RL community . We use PPO as a baseline in our experiments . As environments get more complex , navigating the search space with only deep RL and simple heuristic exploration ( such as ✏-greedy ) is increasingly difficult . Therefore , methods that imitate expert ( i.e. , teacher ) supervision were introduced . A popular approach to imitation learning ( IL ) is Behaviour Cloning ( BC ) , i.e. , use of a supervised classification loss between the policy of the student and expert agents [ 53 , 2 ] . However , BC suffers from compounding errors . Namely , a single mistake of the student may lead to settings that have never been observed in training [ 51 ] . To address this , Data Aggregation ( DAgger ) [ 52 ] trains a sequence of student policies by querying the expert at states beyond those that would be reached by following only expert actions . IL is further enhanced by , e.g. , hierarchies [ 33 ] , improving over the expert [ 5 , 4 , 27 ] , bypassing any intermediate reward function inference [ 24 ] , and/or learning from experts that differ from the student [ 18 , 26 , 16 ] . Importantly , a sequential combination of IL and RL , i.e. , pre-training a model on expert data before letting the agent interact with the environment , performs remarkably well . This strategy has been applied in a wide range of applications – the game of Go [ 61 ] , robotic and motor skills [ 49 , 30 , 48 , 50 ] , navigation in visually realistic environments [ 19 , 12 ] , and web & language based tasks [ 21 , 11 , 59 , 67 ] . More recent methods mix expert demonstrations with the agent ’ s own rollouts instead of using a sequential combination of IL followed by RL . Chemali and Lazaric [ 6 ] perform policy iteration from expert and on-policy demonstrations . DQfD [ 23 ] initializes the replay buffer with expert episodes and adds rollouts of ( a pretrained ) agent . They weight experiences based on the previous temporal difference errors [ 55 ] and use a supervised loss to learn from the expert . For continuous action spaces , DDPGfD [ 66 ] analogously incorporates IL into DDPG [ 35 ] . POfD [ 28 ] improves by adding a demonstration-guided exploration term , i.e. , the Jensen-Shannon divergence between the expert ’ s and the learner ’ s policy ( estimated using occupancy measures ) . THOR uses suboptimal experts to reshape rewards and then searches over a finite planning horizon [ 62 ] . Zhu et al . [ 72 ] show that a combination of GAIL [ 24 ] and RL can be highly effective for difficult manipulation tasks . Critically , the above methods have , implicitly or explicitly , been designed under certain assumptions ( e.g. , the agent operates in an MDP ) which imply the expert and student observe the same state . Different from the above methods , we investigate the difference of privilege between the expert policy and the learned policy . Contrary to a sequential , static , or rule-based combination of supervised loss or divergence , we train an auxiliary actor to adaptively weight IL and RL losses . To the best of our knowledge , this hasn ’ t been studied before . In concurrent work , Warrington et al . [ 69 ] address the imitation gap by jointly training their teacher and student to adapt the teacher to the student . For our applications of interest , this work is not applicable as our expert teachers are fixed . Our approach attempts to reduce the imitation gap directly , assuming the information available to the learning agent is fixed . An indirect approach to reduce this gap is to enrich the information available to the agent or to improve the agent ’ s memory of past experience . Several works have considered this direction in the context of autonomous driving [ 10 , 20 ] and continuous control [ 17 ] . We expect that these methods can be beneficially combined with the method that we discuss next . 3 ADVISOR . We first introduce notation to define the imitation gap and illustrate how it arises due to ‘ policy averaging. ’ Using an ‘ auxiliary policy ’ construct , we then propose ADVISOR to bridge this gap . Finally , we show how to estimate the auxiliary policy in practice using deep networks . In what follows we will use the terms teacher and expert interchangeably . Our use of “ teacher ” is meant to emphasize that these policies are ( 1 ) designed for providing supervision for a student and ( 2 ) need not be optimal among all policies . 3.1 Imitation gap . We want an agent to complete task T in environment E . The environment has states s 2 S and the agent executes an action a 2 A at every discrete timestep t 0 . For simplicity and w.l.o.g . assume both A and S are finite . For example , let E be a 1D-gridworld in which the agent is tasked with navigating to a location by executing actions to move left or right , as shown in Fig . 2a . Here and below we assume states s 2 S encapsulate historical information so that s includes the full trajectory of the agent up to time t 0 . The objective is to find a policy ⇡ , i.e. , a mapping from states to distributions over actions , which maximizes an evaluation criterion . Often this policy search is restricted to a set of feasible policies ⇧feas. , for instance ⇧feas . may be the set { ⇡ ( · ; ✓ ) : ✓ 2 RD } where ⇡ ( · ; ✓ ) is a deep neural network with D-dimensional parameters ✓ . In classical ( deep ) RL [ 41 , 42 ] , the evaluation criterion is usually the expected -discounted future return . We focus on the setting of partially-observed Markov decision processes ( POMDPs ) where an agent makes decisions without access to the full state information . We model this restricted access by defining a filtration function f : S ! Of and limiting the space of feasible policies to those policies ⇧feas.f for which the value of ⇡ ( s ) depends on s only through f ( s ) , i.e. , so that f ( s ) = f ( s 0 ) implies ⇡ ( s ) = ⇡ ( s0 ) . We call any ⇡ satisfying this condition an f -restricted policy and the set of feasible f -restricted policies ⇧feas.f . In a gridworld example , f might restrict s to only include information local to the agent ’ s current position as shown in Figs . 2c , 2d . If a f -restricted policy is optimal among all other f -restricted policies , we say it is f -optimal . We call o 2 Of a partial-observation and for any f -restricted policy ⇡f we write ⇡f ( o ) to mean ⇡f ( s ) if f ( s ) = o . It is frequently the case that , during training , we have access to a teacher policy which is able to successfully complete the task T . This teacher policy may have access to the whole environment state and thus may be optimal among all policies . Alternatively , the teacher policy may , like the student , only make decisions given partial information ( e.g. , a human who sees exactly the same inputs as the student ) . For flexibility we will define the teacher policy as ⇡teachf teach , denoting it is an f teach-restricted policy for some filtration function A A Goal Boundary Agent 1 . 2 . A A A A 0 . Random start 1 . Move right · · · An . Goal reached Hypothetical episode 2 . Move right 3 . Move left 0.0 1.0 0.0 1.0 0.0 1.0 0.0 1.0 ( a ) ( b ) ( c ) ( d ) Environment E start states Actions A = { left , right } = { L , R } L R A A A A· · · A f 2-partial obs . 0.5 0.5 0.5 0.5 0.0 1.0 0.0 1.0 L R ⇡ IL f2 A A A A A· · · f 1-partial obs . 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 L R ⇡ IL f1⇡ teach < latexit sha1_base64= '' Vl/iaP1zJc4vIB4jjoXAxI2rjqE= '' > AAACEXicbVDLSsNAFJ3UV62vWpdugkVwVRJRdFl047KCfUAby2R60w6dZMLMjbSEfIV7t/oL7sStX+Af+BlOHwvbeuDC4Zx7OZfjx4JrdJxvK7e2vrG5ld8u7Ozu7R8UD0sNLRPFoM6kkKrlUw2CR1BHjgJasQIa+gKa/vB24jefQGkuowccx+CFtB/xgDOKRuoWS52YP6YdhBGmCJQNsqxbLDsVZwp7lbhzUiZz1LrFn05PsiSECJmgWrddJ0YvpQo5E5AVOomGmLIh7UPb0IiGoL10+ntmnxqlZwdSmYnQnqp/L1Iaaj0OfbMZUhzoZW8i/ue1EwyuvZRHcYIQsVlQkAgbpT0pwu5xBQzF2BDKFDe/2mxAFWVo6lpI8aUcIvV1VjDVuMtFrJLGecW9qFzeX5SrN/OS8uSYnJAz4pIrUiV3pEbqhJEReSGv5M16tt6tD+tztpqz5jdHZAHW1y8vtp5g < /latexit > Figure 2 : Effect of partial observability in a 1-dimensional gridworld environment . ( a ) The two start states and actions space for 1D-Lighthouse with N = 4 . ( b ) A trajectory of the agent following a hypothetical random policy . At every trajectory step we display output probabilities as per the shortest-path expert ( ⇡teach ) for each state . ( c/d ) Using the same trajectory from ( b ) we highlight the partial-observations available to the agent ( shaded gray ) under different filtration function f1 , f2 . Notice that , under f1 , the agent does not see the goal within its first four steps . The policies ⇡ILf1 , ⇡ IL f2 , learned by imitating ⇡teach , show that imitation results in sub-optimal policies , i.e. , ⇡ILf1 , ⇡ IL f2 6= ⇡ teach . f teach . For simplicity , we will assume that ⇡teachf teach is f teach-optimal . Subsequently , we will drop the subscript f teach unless we wish to explicitly discuss multiple teachers simultaneously . In IL [ 45 , 52 ] , ⇡f is trained to mimic ⇡teach by minimizing the ( expected ) cross-entropy between ⇡f and ⇡teach over a set of sampled states s 2 S : min ⇡f2⇧feas.f Eµ [ CE ( ⇡teach , ⇡f ) ( S ) ] , ( 1 ) where CE ( ⇡teach , ⇡f ) ( S ) = ⇡teach ( S ) log ⇡f ( S ) , denotes the usual dot-product , and S is a random variable taking values s 2 S with probability measure µ : S ! [ 0 , 1 ] . Often µ ( s ) is chosen to equal the frequency with which an exploration policy ( e.g. , random actions or ⇡teach ) visits state s in a randomly initialized episode . When it exists , we denote the policy minimizing Eq . ( 1 ) as ⇡µ , ⇡ teach f . When µ and ⇡teach are unambiguous , we write ⇡ILf = ⇡ µ , ⇡teach f . What happens when there is a difference of privilege ( or filtration functions ) between the teacher and the student ? Intuitively , if the information that a teacher uses to make a decision is unavailable to the student then the student has little hope of being able to mimic the teacher ’ s decisions . As we show in our next example , even when optimizing perfectly , depending on the choice of f and f teach , IL may result in ⇡ILf being uniformly random over a large collection of states . We call the phenomenon that ⇡ IL f 6= ⇡ teach the imitation gap . Example 2 ( 1D-Lighthouse ) . We illustrate the imitation gap using a gridworld spanning { N , . . . , N } . The two start states correspond to the goal being at either N or N , while the agent is always initialized at 0 ( see Fig . 2a ) . Clearly , with full state information , ⇡teach maps states to an ‘ always left ’ or ‘ always right ’ probability distribution , depending on whether the goal is on the left or right , respectively . Suppose now that the agent ’ s visibility is constrained to a radius of i ( Fig . 2c shows i = 1 ) , i.e. , an f i-restricted observation is accessible . An agent following an optimal policy with a visibility of radius i will begin to move deterministically towards any corner , w.l.o.g . assume right . When the agent sees the rightmost edge ( from position N i ) , it will either continue to move right if the goal is visible or , if it ’ s not , move left until it reaches the goal ( at N ) . Now we may ask : what is the best f i-restricted policy that can be learnt by imitating ⇡teach ( i.e. , what is ⇡ILfi ) ? Tragically , the cross-entropy loss causes ⇡ IL fi to be uniform in a large number of states . In particular , an agent following policy ⇡ILfi will execute left ( and right ) actions with probability 0.5 , until it is within a distance of i from one of the corners . Subsequently , it will head directly to the goal . See the policies highlighted in Figs . 2c , 2d . The intuition for this result is straightforward : until the agent observes one of the corners it can not know if the goal is to the right or left and , conditional on its observations , each of these events is equally likely under µ ( assumed uniform ) . Hence for half of these events the teacher will instruct the agent to go right . For the other half the instruction is to go left . See App . A.1 for a rigorous treatment of this example . In Sec . 4 and Fig . 6 , we train f i-restricted policies with f j-optimal teachers for a 2D variant of this example . We empirically verify that a student learns a better policy when imitating teachers whose filtration function is closest to their own . The above example shows : when a student attempts to imitate an expert that is privileged with information not available to the student , the student learns a version of ⇡teach in which this privileged information is marginalized out . We formalize this intuition in the following proposition . Proposition 1 ( Policy Averaging ) .

Summary: This work introduces ADVISOR, a simple yet effective approach to adaptively combining RL with supervision from imitation learning. The approach is motivated by the issue of *information gaps,* which describes the case where the policy generating the supervision has more information about the state than does the policy being trained. The authors motivate their approach as a way to address this issue and contribute a thorough set of empirical analyses to demonstrate its effectiveness. In addition, this work provides a concrete demonstration that the information gap is an obstacle.