Teachable Reinforcement Learning via Advice Distillation

1 Introduction . Reinforcement learning ( RL ) offers a promising paradigm for building agents that can learn complex behaviors from autonomous interaction and minimal human effort . In practice , however , significant human effort is required to design and compute the reward functions that enable successful RL [ 49 ] : the reward functions underlying some of RL ’ s most prominent success stories involve significant domain expertise and elaborate instrumentation of the agent and environment [ 37 , 38 , 44 , 28 , 15 ] . Even with this complexity , a reward is ultimately no more than a scalar indicator of how good a particular state is relative to others . Rewards provide limited information about how to perform tasks , and reward-driven RL agents must perform significant exploration and experimentation within an environment to learn effectively . A number of alternative paradigms for interactively learning policies have emerged as alternatives , such as imitation learning [ 40 , 20 , 50 ] , DAgger [ 43 ] , and preference learning [ 10 , 6 ] . But these existing methods are either impractically low bandwidth ( extracting little information from each human intervention ) [ 25 , 30 , 10 ] or require costly data collection [ 44 , 23 ] . It has proven challenging to develop training methods that are simultaneously expressive and efficient enough to rapidly train agents to acquire novel skills . Human learners , by contrast , leverage numerous , rich forms of supervision : joint attention [ 34 ] , physical corrections [ 5 ] and natural language instruction [ 9 ] . For human teachers , this kind of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . coaching is often no more costly to provide than scalar measures of success , but significantly more informative for learners . In this way , human learners use high-bandwidth , low-effort communication as a means to flexibly acquire new concepts or skills [ 46 , 33 ] . Importantly , the interpretation of some of these feedback signals ( like language ) , is itself learned , but can be bootstrapped from other forms of communication : for example , the function of gesture and attention can be learned from intrinsic rewards [ 39 ] ; these in turn play a key role in language learning [ 31 ] . This paper proposes a framework for training automated agents using similarly rich interactive supervision . For instance , given an agent learning a policy to navigate and manipulate objects in a simulated multi-room object manipulation problem ( e.g. , Fig 3 left ) , we train agents using not just reward signals but advice about what actions to take ( “ move left ” ) , what waypoints to move towards ( “ move towards ( 1,2 ) ” ) , and what sub-goals to accomplish ( “ pick up the yellow ball ” ) , offering human supervisors a toolkit of rich feedback forms that direct and modify agent behavior . To do so , we introduce a new formulation of interactive learning , the Coaching-Augmented Markov Decision Process ( CAMDP ) , which formalizes the problem of learning from a privileged supervisory signal provided via an observation channel . We then describe an algorithmic framework for learning in CAMDPs via alternating advice grounding and distillation phases . During the grounding phase , agents learn associations between teacher-provided advice and high-value actions in the environment ; during distillation , agents collect trajectories with grounded models and interactive advice , then transfer information from these trajectories to fully autonomous policies that operate without advice . This formulation allows supervisors to guide agent behavior interactively , while enabling agents to internalize this guidance to continue performing tasks autonomously once the supervisor is no longer present . Moreover , this procedure can be extended to enable bootstrapping of grounded models that use increasingly sparse and abstract advice types , leveraging some types of feedback to ground others . Experiments show that models trained via coaching can learn new tasks more efficiently and with 20x less human supervision than naïve methods for RL across puzzle-solving [ 8 ] , navigation [ 14 ] , and locomotion domains [ 8 ] . In summary , this paper describes : ( 1 ) a general framework ( CAMDPs ) for human-in-the-loop RL with rich interactive advice ; ( 2 ) an algorithm for learning in CAMDPs with a single form of advice ; ( 3 ) an extension of this algorithm that enables bootstrapped learning of multiple advice types ; and finally ( 4 ) a set of empirical evaluations on discrete and continuous control problems in the BabyAI [ 8 ] and D4RL [ 14 ] environments . It thus offers a groundwork for moving beyond reward signals in interactive learning , and instead training agents with the full range of human communicative modalities . 2 Coaching Augmented Markov Decision Processes . To develop our procedure for learning from rich feedback , we begin by formalizing the environments and tasks for which feedback is provided . This formalization builds on the framework of multi-task RL and Markov decision processes ( MDP ) , augmenting them with advice provided by a coach in the loop through an arbitrary prescriptive channel of communication . Conider the grid-world environment depicted in Fig 3 left [ 8 ] . Tasks in this environment specify particular specific desired goal states ; e.g . “ place the yellow ball in the green box and the blue key in the green box ” or “ open all doors in the blue room. ” In multi-task RL , a learner ’ s objective is produce a policy p ( at |st , t ) that maximizes reward in expectation over tasks t . More formally , a multi-task MDP is defined by a 7-tuple M ⌘ ( S , A , T , R , r ( s0 ) , g , p ( t ) ) , where S denotes the state space , A denotes the action space , p : S ⇥A ⇥S 7 ! R 0 denotes the transition dynamics , r : S ⇥A ⇥ t 7 ! R denotes the reward function , r : S 7 ! R 0 denotes the initial state distribution , g 2 [ 0,1 ] denotes the discount factor and p ( t ) denotes the distribution over tasks . The objective in a multi-task MDP is to learn a policy pq that maximizes the expected sum of discounted returns in expectation over tasks : maxq JE ( pq , p ( t ) ) = Eat⇠pq ( ·|st , t ) t⇠p ( t ) [ Â•t=0 g t r ( st , at , t ) ] . Why might additional supervision beyond the reward signal be useful for solving this optimization problem ? Suppose the agent in Fig 3 is in the ( low-value ) state shown in the figure , but could reach a high-value state by going “ right and up ” towards the blue key . This fact is difficult to communicate through a scalar reward , which can not convey information about alternative actions . A side channel for providing this type of rich information at training-time would be greatly beneficial . We model this as follows : a coaching-augmented MDP ( CAMDP ) consists of an ordinary multitask MDP augmented with a set of coaching functions C = { C 1 , C 2 , · · · , C i } , where each C j provides a different form of feedback to the agent . Like a reward function , each coaching function models a form of supervision provided externally to the agent ( by a coach ) ; these functions may produce informative outputs densely ( at each timestep ) or only infrequently . Unlike rewards , which give agents feedback on the desirability of states and actions they have already experienced , this coaching provides information about what the agent should do next . 1 As shown in Figure 3 , advice can take many forms , for instance action advice ( c0 ) , waypoints ( c1 ) , language sub-goals ( c2 ) , or any other local information relevant to task completion.2 Coaching in a CAMDP is useful if it provides an agent local guidance on how to proceed toward a goal that is inferrable from the agent ’ s current observation , when the mapping from observations and goals to actions has not yet been learned . As in standard reinforcement learning in an multi-task MDP , the goal in a CAMDP is to learn a policy pq ( · | st , t ) that chooses an action based on Markovian state st and high level task information t without interacting with c j . However , we allow learning algorithms to use the coaching signal c j to learn this policy more efficiently at training time ( although this is unavailable during deployment ) . For instance , the agent in Fig 3 can leverage hints “ go left ” or “ move towards the blue key ” to guide its exploration process but it eventually must learn how to perform the task without any coaching required . Section 3 decribes an algorithm for acquiring this independent , multi-task policy pq ( · | st , t ) from coaching feedback , and Section 4 presents an empirical evaluation of this algorithm . 3 Leveraging Advice via Distillation . 3.1 Preliminaries . The challenge of learning in a CAMDP is twofold : first , agents must learn to ground coaching signals in concrete behavior ; second , agents must learn from these coaching signals to independently solve the task of interest in the absence of any human supervision . To accomplish this , we divide agent training into three phases : ( 1 ) a grounding phase , ( 2 ) an improvement phase and ( 3 ) an evaluation phase . In the grounding phase , agents learn how to interpret coaching . The result of the grounding phase is a surrogate policy q ( at | st , t , c ) that can effectively condition on coaching when it is provided in the training loop . As we will discuss in Section 3.2 , this phase can also make use of a bootstrapping process in which more complex forms of feedback are learned using signals from simpler ones . During the improvement phase , agents use the ability to interpret advice to learn new skills . Specifically , the learner is presented with a novel task ttest that was not provided during the grounding phase , and must learn to perform this task using only a small amount of interaction in which advice c is provided by a human supervisor who is present in the loop . This advice , combined with the 1While the design of optimal coaching strategies and explicit modeling of coaches are important research topics [ 16 ] , this paper assumes that the coach is fixed and not explicitly modeled . Our empirical evaluation use both scripted coaches and human-in-the-loop feedback . 2When only a single form of advice is available to the agent , we omit the superscript for clarity . learned surrogate policy q ( at |st , t , c ) , can be used to efficiently acquire an advice-independent policy p ( at |st , t ) , which can perform tasks without requiring any coaching . Finally , in the evaluation phase , agent performance is evaluated on the task ttest by executing the advice-independent , multi-task policy p ( at |st , ttest ) in the environment . 3.2 Grounding Phase : Learning to Interpret Advice . The goal of the grounding phase is to learn a mapping from advice to contextually appropriate actions , so that advice can be used for quickly learning new tasks . In this phase , we run RL on a distribution of training tasks p ( t ) . As the purpose of these training environments is purely to ground coaching , sometimes called “ advice ” , the tasks may be much simpler than test-time tasks . During this phase , the agent uses access to a reward function r ( s , a , c ) , as well as the advice c ( s , a ) to learn a surrogate policy qf ( a|s , t , c ) . The reward function r ( s , a , c ) is provided by the coach during the grounding phase only and rewards the agent for correctly following the provided coaching , not just for accomplishing the task . Since coaching instructions ( e.g . cardinal directions ) are much easier to follow than completing a full task , grounding can be learned quickly . The process of grounding is no different than standard multi-task RL , incorporating advice c ( s , a ) as another component of the observation space . This formulation makes minimal assumptions about the form of the coaching c. During this grounding process , the agent ’ s optimization objective is : max f J ( q ) = E t⇠p ( t ) at⇠qf ( at |st , t , c ) Â t r ( st , at , c ) , ( 1 ) Bootstrapping Multi-Level Advice The previous section described how to train an agent to interpret a single form of advice c. In practice , a coach might find it useful to use multiple forms of advice—for instance high-level language sub-goals for easy stages of the task and low-level action advice for more challenging parts of the task . While high-level advice can be very informative for guiding the learning of new tasks in the improvement phase , it can often be quite difficult to ground quickly pure RL . Instead of relying on RL , we can bootstrap the process of grounding one form of advice ch from a policy q ( a|s , t , cl ) that can interpret a different form of advice cl . In particular , we can use a surrogate policy which already understands ( using the grounding scheme described above ) low-level advice q ( a|s , t , cl ) to bootstrap training of a surrogate policy which understands higher-level advice q ( a|s , t , ch ) . We call this process “ bootstrap distillation ” . Intuitively , we use a supervisor in the loop to guide an advice-conditional policy that can interpret a low-level form of advice qf1 ( a|s , t , cl ) to perform a training task , obtaining trajectories D = { ( s0 , a0 , cl0 , ch0 ) , ( s1 , a1 , cl1 , ch1 ) · · · , ( sH , aH , clH , chH ) } Nj=1 , then distilling the demonstrated behavior via supervised learning into a policy qf2 ( a|s , t , ch ) that can interpret higher-level advice to perform this new task without requiring the low level advice any longer . More specifically , we make use of an input remapping solution , as seen in Levine et al . [ 28 ] , where the policy conditioned on advice cl is used to generate optimal action labels , which are then remapped to observations with a different form of advice ch as input . To bootstrap the understanding of an abstract form of advice ch from a more low level one cl , the agent optimizes the following objective to bootstrap the agent ’ s understanding of one advice type from another : D = { ( s0 , a0 , cl0 , ch0 ) , ( s1 , a1 , cl1 , ch1 ) , · · · , ( sH , aH , clH , chH ) } Nj=1 s0 ⇠ p ( s0 ) , at ⇠ qf1 ( at |st , t , c l ) , st+1 ⇠ p ( st+1|st , at ) max f2 E ( st , at , cht , t ) ⇠D h logqf2 ( at |st , t , c h t ) i With this procedure , we only need to use RL to ground the simplest , fastest-learned advice form , and we can use more efficient bootstrapping to ground the others .

Summary: The authors propose a reinforcement learning approach where humans can  provide different forms of advice to complete a task. The proposed approach, called CAMDPs, follows a two-stage process: Grounding and Improvement.   During grounding the system learns how to interpret the advice. The result of this phase is a surrogate policy. Grounding learns a mapping from advice to actions using information from the true reward function. Grounding can be seen as an ordinarily multitask RL, where each task is a piece of advice, and the agent learns using normal RL how to achieve the advice.   Distillation is used to learn a policy using supervised learning and bootstrap distillation uses the previously learned policies for individual advice to learn a policy for a particular task.  During improvement the coach provides advice to solve a new task. A very similar approach is followed as bootstrap distillation, a new task is solved using advice followed by a distillation process.  It is expected for the learned policies to be general enough to be applied in other, although similar, domains.  The proposed system is evaluated in three, relatively similar, domains. As expected, it is experimentally shown that the proposed system learns faster with advice.