Teachable Reinforcement Learning via Advice Distillation

1 Introduction . Reinforcement learning ( RL ) offers a promising paradigm for building agents that can learn complex behaviors from autonomous interaction and minimal human effort . In practice , however , significant human effort is required to design and compute the reward functions that enable successful RL [ 49 ] : the reward functions underlying some of RL ’ s most prominent success stories involve significant domain expertise and elaborate instrumentation of the agent and environment [ 37 , 38 , 44 , 28 , 15 ] . Even with this complexity , a reward is ultimately no more than a scalar indicator of how good a particular state is relative to others . Rewards provide limited information about how to perform tasks , and reward-driven RL agents must perform significant exploration and experimentation within an environment to learn effectively . A number of alternative paradigms for interactively learning policies have emerged as alternatives , such as imitation learning [ 40 , 20 , 50 ] , DAgger [ 43 ] , and preference learning [ 10 , 6 ] . But these existing methods are either impractically low bandwidth ( extracting little information from each human intervention ) [ 25 , 30 , 10 ] or require costly data collection [ 44 , 23 ] . It has proven challenging to develop training methods that are simultaneously expressive and efficient enough to rapidly train agents to acquire novel skills . Human learners , by contrast , leverage numerous , rich forms of supervision : joint attention [ 34 ] , physical corrections [ 5 ] and natural language instruction [ 9 ] . For human teachers , this kind of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . coaching is often no more costly to provide than scalar measures of success , but significantly more informative for learners . In this way , human learners use high-bandwidth , low-effort communication as a means to flexibly acquire new concepts or skills [ 46 , 33 ] . Importantly , the interpretation of some of these feedback signals ( like language ) , is itself learned , but can be bootstrapped from other forms of communication : for example , the function of gesture and attention can be learned from intrinsic rewards [ 39 ] ; these in turn play a key role in language learning [ 31 ] . This paper proposes a framework for training automated agents using similarly rich interactive supervision . For instance , given an agent learning a policy to navigate and manipulate objects in a simulated multi-room object manipulation problem ( e.g. , Fig 3 left ) , we train agents using not just reward signals but advice about what actions to take ( “ move left ” ) , what waypoints to move towards ( “ move towards ( 1,2 ) ” ) , and what sub-goals to accomplish ( “ pick up the yellow ball ” ) , offering human supervisors a toolkit of rich feedback forms that direct and modify agent behavior . To do so , we introduce a new formulation of interactive learning , the Coaching-Augmented Markov Decision Process ( CAMDP ) , which formalizes the problem of learning from a privileged supervisory signal provided via an observation channel . We then describe an algorithmic framework for learning in CAMDPs via alternating advice grounding and distillation phases . During the grounding phase , agents learn associations between teacher-provided advice and high-value actions in the environment ; during distillation , agents collect trajectories with grounded models and interactive advice , then transfer information from these trajectories to fully autonomous policies that operate without advice . This formulation allows supervisors to guide agent behavior interactively , while enabling agents to internalize this guidance to continue performing tasks autonomously once the supervisor is no longer present . Moreover , this procedure can be extended to enable bootstrapping of grounded models that use increasingly sparse and abstract advice types , leveraging some types of feedback to ground others . Experiments show that models trained via coaching can learn new tasks more efficiently and with 20x less human supervision than naïve methods for RL across puzzle-solving [ 8 ] , navigation [ 14 ] , and locomotion domains [ 8 ] . In summary , this paper describes : ( 1 ) a general framework ( CAMDPs ) for human-in-the-loop RL with rich interactive advice ; ( 2 ) an algorithm for learning in CAMDPs with a single form of advice ; ( 3 ) an extension of this algorithm that enables bootstrapped learning of multiple advice types ; and finally ( 4 ) a set of empirical evaluations on discrete and continuous control problems in the BabyAI [ 8 ] and D4RL [ 14 ] environments . It thus offers a groundwork for moving beyond reward signals in interactive learning , and instead training agents with the full range of human communicative modalities . 2 Coaching Augmented Markov Decision Processes . To develop our procedure for learning from rich feedback , we begin by formalizing the environments and tasks for which feedback is provided . This formalization builds on the framework of multi-task RL and Markov decision processes ( MDP ) , augmenting them with advice provided by a coach in the loop through an arbitrary prescriptive channel of communication . Conider the grid-world environment depicted in Fig 3 left [ 8 ] . Tasks in this environment specify particular specific desired goal states ; e.g . “ place the yellow ball in the green box and the blue key in the green box ” or “ open all doors in the blue room. ” In multi-task RL , a learner ’ s objective is produce a policy p ( at |st , t ) that maximizes reward in expectation over tasks t . More formally , a multi-task MDP is defined by a 7-tuple M ⌘ ( S , A , T , R , r ( s0 ) , g , p ( t ) ) , where S denotes the state space , A denotes the action space , p : S ⇥A ⇥S 7 ! R 0 denotes the transition dynamics , r : S ⇥A ⇥ t 7 ! R denotes the reward function , r : S 7 ! R 0 denotes the initial state distribution , g 2 [ 0,1 ] denotes the discount factor and p ( t ) denotes the distribution over tasks . The objective in a multi-task MDP is to learn a policy pq that maximizes the expected sum of discounted returns in expectation over tasks : maxq JE ( pq , p ( t ) ) = Eat⇠pq ( ·|st , t ) t⇠p ( t ) [ Â•t=0 g t r ( st , at , t ) ] . Why might additional supervision beyond the reward signal be useful for solving this optimization problem ? Suppose the agent in Fig 3 is in the ( low-value ) state shown in the figure , but could reach a high-value state by going “ right and up ” towards the blue key . This fact is difficult to communicate through a scalar reward , which can not convey information about alternative actions . A side channel for providing this type of rich information at training-time would be greatly beneficial . We model this as follows : a coaching-augmented MDP ( CAMDP ) consists of an ordinary multitask MDP augmented with a set of coaching functions C = { C 1 , C 2 , · · · , C i } , where each C j provides a different form of feedback to the agent . Like a reward function , each coaching function models a form of supervision provided externally to the agent ( by a coach ) ; these functions may produce informative outputs densely ( at each timestep ) or only infrequently . Unlike rewards , which give agents feedback on the desirability of states and actions they have already experienced , this coaching provides information about what the agent should do next . 1 As shown in Figure 3 , advice can take many forms , for instance action advice ( c0 ) , waypoints ( c1 ) , language sub-goals ( c2 ) , or any other local information relevant to task completion.2 Coaching in a CAMDP is useful if it provides an agent local guidance on how to proceed toward a goal that is inferrable from the agent ’ s current observation , when the mapping from observations and goals to actions has not yet been learned . As in standard reinforcement learning in an multi-task MDP , the goal in a CAMDP is to learn a policy pq ( · | st , t ) that chooses an action based on Markovian state st and high level task information t without interacting with c j . However , we allow learning algorithms to use the coaching signal c j to learn this policy more efficiently at training time ( although this is unavailable during deployment ) . For instance , the agent in Fig 3 can leverage hints “ go left ” or “ move towards the blue key ” to guide its exploration process but it eventually must learn how to perform the task without any coaching required . Section 3 decribes an algorithm for acquiring this independent , multi-task policy pq ( · | st , t ) from coaching feedback , and Section 4 presents an empirical evaluation of this algorithm . 3 Leveraging Advice via Distillation . 3.1 Preliminaries . The challenge of learning in a CAMDP is twofold : first , agents must learn to ground coaching signals in concrete behavior ; second , agents must learn from these coaching signals to independently solve the task of interest in the absence of any human supervision . To accomplish this , we divide agent training into three phases : ( 1 ) a grounding phase , ( 2 ) an improvement phase and ( 3 ) an evaluation phase . In the grounding phase , agents learn how to interpret coaching . The result of the grounding phase is a surrogate policy q ( at | st , t , c ) that can effectively condition on coaching when it is provided in the training loop . As we will discuss in Section 3.2 , this phase can also make use of a bootstrapping process in which more complex forms of feedback are learned using signals from simpler ones . During the improvement phase , agents use the ability to interpret advice to learn new skills . Specifically , the learner is presented with a novel task ttest that was not provided during the grounding phase , and must learn to perform this task using only a small amount of interaction in which advice c is provided by a human supervisor who is present in the loop . This advice , combined with the 1While the design of optimal coaching strategies and explicit modeling of coaches are important research topics [ 16 ] , this paper assumes that the coach is fixed and not explicitly modeled . Our empirical evaluation use both scripted coaches and human-in-the-loop feedback . 2When only a single form of advice is available to the agent , we omit the superscript for clarity . learned surrogate policy q ( at |st , t , c ) , can be used to efficiently acquire an advice-independent policy p ( at |st , t ) , which can perform tasks without requiring any coaching . Finally , in the evaluation phase , agent performance is evaluated on the task ttest by executing the advice-independent , multi-task policy p ( at |st , ttest ) in the environment . 3.2 Grounding Phase : Learning to Interpret Advice . The goal of the grounding phase is to learn a mapping from advice to contextually appropriate actions , so that advice can be used for quickly learning new tasks . In this phase , we run RL on a distribution of training tasks p ( t ) . As the purpose of these training environments is purely to ground coaching , sometimes called “ advice ” , the tasks may be much simpler than test-time tasks . During this phase , the agent uses access to a reward function r ( s , a , c ) , as well as the advice c ( s , a ) to learn a surrogate policy qf ( a|s , t , c ) . The reward function r ( s , a , c ) is provided by the coach during the grounding phase only and rewards the agent for correctly following the provided coaching , not just for accomplishing the task . Since coaching instructions ( e.g . cardinal directions ) are much easier to follow than completing a full task , grounding can be learned quickly . The process of grounding is no different than standard multi-task RL , incorporating advice c ( s , a ) as another component of the observation space . This formulation makes minimal assumptions about the form of the coaching c. During this grounding process , the agent ’ s optimization objective is : max f J ( q ) = E t⇠p ( t ) at⇠qf ( at |st , t , c ) Â t r ( st , at , c ) , ( 1 ) Bootstrapping Multi-Level Advice The previous section described how to train an agent to interpret a single form of advice c. In practice , a coach might find it useful to use multiple forms of advice—for instance high-level language sub-goals for easy stages of the task and low-level action advice for more challenging parts of the task . While high-level advice can be very informative for guiding the learning of new tasks in the improvement phase , it can often be quite difficult to ground quickly pure RL . Instead of relying on RL , we can bootstrap the process of grounding one form of advice ch from a policy q ( a|s , t , cl ) that can interpret a different form of advice cl . In particular , we can use a surrogate policy which already understands ( using the grounding scheme described above ) low-level advice q ( a|s , t , cl ) to bootstrap training of a surrogate policy which understands higher-level advice q ( a|s , t , ch ) . We call this process “ bootstrap distillation ” . Intuitively , we use a supervisor in the loop to guide an advice-conditional policy that can interpret a low-level form of advice qf1 ( a|s , t , cl ) to perform a training task , obtaining trajectories D = { ( s0 , a0 , cl0 , ch0 ) , ( s1 , a1 , cl1 , ch1 ) · · · , ( sH , aH , clH , chH ) } Nj=1 , then distilling the demonstrated behavior via supervised learning into a policy qf2 ( a|s , t , ch ) that can interpret higher-level advice to perform this new task without requiring the low level advice any longer . More specifically , we make use of an input remapping solution , as seen in Levine et al . [ 28 ] , where the policy conditioned on advice cl is used to generate optimal action labels , which are then remapped to observations with a different form of advice ch as input . To bootstrap the understanding of an abstract form of advice ch from a more low level one cl , the agent optimizes the following objective to bootstrap the agent ’ s understanding of one advice type from another : D = { ( s0 , a0 , cl0 , ch0 ) , ( s1 , a1 , cl1 , ch1 ) , · · · , ( sH , aH , clH , chH ) } Nj=1 s0 ⇠ p ( s0 ) , at ⇠ qf1 ( at |st , t , c l ) , st+1 ⇠ p ( st+1|st , at ) max f2 E ( st , at , cht , t ) ⇠D h logqf2 ( at |st , t , c h t ) i With this procedure , we only need to use RL to ground the simplest , fastest-learned advice form , and we can use more efficient bootstrapping to ground the others .

Summary: This paper presents a framework for teaching agents using language advice. The framework consists of three stages: (i) grounding language advice to action, (ii) learning an unconditioned policy that mimics the advice-conditioned policy (iii) evaluating the unconditioned policy. Experiments are conducted in maze navigation environments. Results demonstrate the effectiveness of the framework compared with imitation learning and reinforcement learning.