Teachable Reinforcement Learning via Advice Distillation
Year: 2022

Abstract:
Training automated agents to complete complex tasks in interactive environments is challenging: reinforcement learning requires careful hand-engineering of reward functions, imitation learning requires specialized infrastructure and access to a human expert, and learning from intermediate forms of supervision (like binary preferences) is time-consuming and extracts little information from each human intervention. Can we overcome these challenges by building agents that learn from rich, interactive feedback instead? We propose a new supervision paradigm for interactive learning based on “teachable” decision-making systems that learn from structured advice provided by an external teacher. We begin by formalizing a class of human-in-the-loop decision making problems in which multiple forms of teacher-provided advice are available to a learner. We then describe a simple learning algorithm for these problems that first learns to interpret advice, then learns from advice to complete tasks even in the absence of human supervision. In puzzle-solving, navigation, and locomotion domains, we show that agents that learn from advice can acquire new skills with significantly less human supervision than standard reinforcement learning algorithms and often less than imitation learning.

heading: 1 Introduction
text: Reinforcement learning (RL) offers a promising paradigm for building agents that can learn complex behaviors from autonomous interaction and minimal human effort. In practice, however, significant human effort is required to design and compute the reward functions that enable successful RL [49]: the reward functions underlying some of RL’s most prominent success stories involve significant domain expertise and elaborate instrumentation of the agent and environment [37, 38, 44, 28, 15]. Even with this complexity, a reward is ultimately no more than a scalar indicator of how good a particular state is relative to others. Rewards provide limited information about how to perform tasks, and reward-driven RL agents must perform significant exploration and experimentation within an environment to learn effectively. A number of alternative paradigms for interactively learning policies have emerged as alternatives, such as imitation learning [40, 20, 50], DAgger [43], and preference learning [10, 6]. But these existing methods are either impractically low bandwidth (extracting little information from each human intervention) [25, 30, 10] or require costly data collection [44, 23]. It has proven challenging to develop training methods that are simultaneously expressive and efficient enough to rapidly train agents to acquire novel skills.
Human learners, by contrast, leverage numerous, rich forms of supervision: joint attention [34], physical corrections [5] and natural language instruction [9]. For human teachers, this kind of
35th Conference on Neural Information Processing Systems (NeurIPS 2021), virtual.
coaching is often no more costly to provide than scalar measures of success, but significantly more informative for learners. In this way, human learners use high-bandwidth, low-effort communication as a means to flexibly acquire new concepts or skills [46, 33]. Importantly, the interpretation of some of these feedback signals (like language), is itself learned, but can be bootstrapped from other forms of communication: for example, the function of gesture and attention can be learned from intrinsic rewards [39]; these in turn play a key role in language learning [31].
This paper proposes a framework for training automated agents using similarly rich interactive supervision. For instance, given an agent learning a policy to navigate and manipulate objects in a simulated multi-room object manipulation problem (e.g., Fig 3 left), we train agents using not just reward signals but advice about what actions to take (“move left”), what waypoints to move towards (“move towards (1,2)”), and what sub-goals to accomplish (“pick up the yellow ball”), offering human supervisors a toolkit of rich feedback forms that direct and modify agent behavior. To do so, we introduce a new formulation of interactive learning, the Coaching-Augmented Markov Decision Process (CAMDP), which formalizes the problem of learning from a privileged supervisory signal provided via an observation channel. We then describe an algorithmic framework for learning in CAMDPs via alternating advice grounding and distillation phases. During the grounding phase, agents learn associations between teacher-provided advice and high-value actions in the environment; during distillation, agents collect trajectories with grounded models and interactive advice, then transfer information from these trajectories to fully autonomous policies that operate without advice. This formulation allows supervisors to guide agent behavior interactively, while enabling agents to internalize this guidance to continue performing tasks autonomously once the supervisor is no longer present. Moreover, this procedure can be extended to enable bootstrapping of grounded models that use increasingly sparse and abstract advice types, leveraging some types of feedback to ground others. Experiments show that models trained via coaching can learn new tasks more efficiently and with 20x less human supervision than naïve methods for RL across puzzle-solving [8], navigation [14], and locomotion domains [8].
In summary, this paper describes: (1) a general framework (CAMDPs) for human-in-the-loop RL with rich interactive advice; (2) an algorithm for learning in CAMDPs with a single form of advice; (3) an extension of this algorithm that enables bootstrapped learning of multiple advice types; and finally (4) a set of empirical evaluations on discrete and continuous control problems in the BabyAI [8] and D4RL [14] environments. It thus offers a groundwork for moving beyond reward signals in interactive learning, and instead training agents with the full range of human communicative modalities.

heading: 2 Coaching Augmented Markov Decision Processes
text: To develop our procedure for learning from rich feedback, we begin by formalizing the environments and tasks for which feedback is provided. This formalization builds on the framework of multi-task RL and Markov decision processes (MDP), augmenting them with advice provided by a coach in the loop through an arbitrary prescriptive channel of communication. Conider the grid-world environment depicted in Fig 3 left [8]. Tasks in this environment specify particular specific desired goal states; e.g. “place the yellow ball in the green box and the blue key in the green box” or “open all doors in
the blue room.” In multi-task RL, a learner’s objective is produce a policy p(at |st ,t) that maximizes reward in expectation over tasks t . More formally, a multi-task MDP is defined by a 7-tuple M ⌘ (S ,A ,T ,R,r(s0),g, p(t)), where S denotes the state space, A denotes the action space, p : S ⇥A ⇥S 7! R 0 denotes the transition dynamics, r : S ⇥A ⇥ t 7! R denotes the reward function, r : S 7! R 0 denotes the initial state distribution, g 2 [0,1] denotes the discount factor and p(t) denotes the distribution over tasks. The objective in a multi-task MDP is to learn a policy pq that maximizes the expected sum of discounted returns in expectation over tasks: maxq JE(pq , p(t)) = Eat⇠pq (·|st ,t)
t⇠p(t) [Â•t=0 g t r(st ,at ,t)].
Why might additional supervision beyond the reward signal be useful for solving this optimization problem? Suppose the agent in Fig 3 is in the (low-value) state shown in the figure, but could reach a high-value state by going “right and up” towards the blue key. This fact is difficult to communicate through a scalar reward, which cannot convey information about alternative actions. A side channel for providing this type of rich information at training-time would be greatly beneficial.
We model this as follows: a coaching-augmented MDP (CAMDP) consists of an ordinary multitask MDP augmented with a set of coaching functions C = {C 1,C 2, · · · ,C i}, where each C j provides a different form of feedback to the agent. Like a reward function, each coaching function models a form of supervision provided externally to the agent (by a coach); these functions may produce informative outputs densely (at each timestep) or only infrequently. Unlike rewards, which give agents feedback on the desirability of states and actions they have already experienced, this coaching provides information about what the agent should do next. 1 As shown in Figure 3, advice can take many forms, for instance action advice (c0), waypoints (c1), language sub-goals (c2), or any other local information relevant to task completion.2 Coaching in a CAMDP is useful if it provides an agent local guidance on how to proceed toward a goal that is inferrable from the agent’s current observation, when the mapping from observations and goals to actions has not yet been learned.
As in standard reinforcement learning in an multi-task MDP, the goal in a CAMDP is to learn a policy pq (· | st ,t) that chooses an action based on Markovian state st and high level task information t without interacting with c j. However, we allow learning algorithms to use the coaching signal c j to learn this policy more efficiently at training time (although this is unavailable during deployment). For instance, the agent in Fig 3 can leverage hints “go left” or “move towards the blue key” to guide its exploration process but it eventually must learn how to perform the task without any coaching required. Section 3 decribes an algorithm for acquiring this independent, multi-task policy pq (· | st ,t) from coaching feedback, and Section 4 presents an empirical evaluation of this algorithm.

heading: 3 Leveraging Advice via Distillation

heading: 3.1 Preliminaries
text: The challenge of learning in a CAMDP is twofold: first, agents must learn to ground coaching signals in concrete behavior; second, agents must learn from these coaching signals to independently solve the task of interest in the absence of any human supervision. To accomplish this, we divide agent training into three phases: (1) a grounding phase, (2) an improvement phase and (3) an evaluation phase.
In the grounding phase, agents learn how to interpret coaching. The result of the grounding phase is a surrogate policy q(at | st ,t,c) that can effectively condition on coaching when it is provided in the training loop. As we will discuss in Section 3.2, this phase can also make use of a bootstrapping process in which more complex forms of feedback are learned using signals from simpler ones.
During the improvement phase, agents use the ability to interpret advice to learn new skills. Specifically, the learner is presented with a novel task ttest that was not provided during the grounding phase, and must learn to perform this task using only a small amount of interaction in which advice c is provided by a human supervisor who is present in the loop. This advice, combined with the
1While the design of optimal coaching strategies and explicit modeling of coaches are important research topics [16], this paper assumes that the coach is fixed and not explicitly modeled. Our empirical evaluation use both scripted coaches and human-in-the-loop feedback.
2When only a single form of advice is available to the agent, we omit the superscript for clarity.
learned surrogate policy q(at |st ,t,c), can be used to efficiently acquire an advice-independent policy p(at |st ,t), which can perform tasks without requiring any coaching. Finally, in the evaluation phase, agent performance is evaluated on the task ttest by executing the advice-independent, multi-task policy p(at |st ,ttest)in the environment.

heading: 3.2 Grounding Phase: Learning to Interpret Advice
text: The goal of the grounding phase is to learn a mapping from advice to contextually appropriate actions, so that advice can be used for quickly learning new tasks. In this phase, we run RL on a distribution of training tasks p(t). As the purpose of these training environments is purely to ground coaching, sometimes called “advice”, the tasks may be much simpler than test-time tasks. During this phase, the agent uses access to a reward function r(s,a,c), as well as the advice c(s,a) to learn a surrogate policy qf (a|s,t,c). The reward function r(s,a,c) is provided by the coach during the grounding phase only and rewards the agent for correctly following the provided coaching, not just for accomplishing the task. Since coaching instructions (e.g. cardinal directions) are much easier to follow than completing a full task, grounding can be learned quickly. The process of grounding is no different than standard multi-task RL, incorporating advice c(s,a) as another component of the observation space. This formulation makes minimal assumptions about the form of the coaching c.
During this grounding process, the agent’s optimization objective is:
max f J(q) = E t⇠p(t) at⇠qf (at |st ,t,c)
 Â
t
r(st ,at ,c) , (1)
Bootstrapping Multi-Level Advice The previous section described how to train an agent to interpret a single form of advice c. In practice, a coach might find it useful to use multiple forms of advice—for instance high-level language sub-goals for easy stages of the task and low-level action advice for more challenging parts of the task. While high-level advice can be very informative for guiding the learning of new tasks in the improvement phase, it can often be quite difficult to ground quickly pure RL. Instead of relying on RL, we can bootstrap the process of grounding one form of advice ch from a policy q(a|s,t,cl) that can interpret a different form of advice cl . In particular, we can use a surrogate policy which already understands (using the grounding scheme described above) low-level advice q(a|s,t,cl) to bootstrap training of a surrogate policy which understands higher-level advice q(a|s,t,ch). We call this process “bootstrap distillation”. Intuitively, we use a supervisor in the loop to guide an advice-conditional policy that can interpret a low-level form of advice qf1(a|s,t,cl) to perform a training task, obtaining trajectories D = {(s0,a0,cl0,ch0),(s1,a1,cl1,ch1) · · · ,(sH ,aH ,clH ,chH)}Nj=1, then distilling the demonstrated behavior via supervised learning into a policy qf2(a|s,t,ch) that can interpret higher-level advice to perform this
new task without requiring the low level advice any longer. More specifically, we make use of an input remapping solution, as seen in Levine et al. [28], where the policy conditioned on advice cl is used to generate optimal action labels, which are then remapped to observations with a different form of advice ch as input. To bootstrap the understanding of an abstract form of advice ch from a more low level one cl , the agent optimizes the following objective to bootstrap the agent’s understanding of one advice type from another:
D ={(s0,a0,cl0,ch0),(s1,a1,cl1,ch1), · · · ,(sH ,aH ,clH ,chH)}Nj=1 s0 ⇠ p(s0),at ⇠ qf1(at |st ,t,c l),st+1 ⇠ p(st+1|st ,at)
max f2
E(st ,at ,cht ,t)⇠D h logqf2(at |st ,t,c h t ) i
With this procedure, we only need to use RL to ground the simplest, fastest-learned advice form, and we can use more efficient bootstrapping to ground the others.

heading: 3.3 Improvement Phase: Learning New Tasks Efficiently with Advice
text: At the end of the grounding phase, we have an advice-following agent qf (a|s,t,c) that can interpret various forms of advice. Ultimately, we want a policy p(a|s,t) which is able to succeed at performing the new test task ttest, without requiring advice at evaluation time. To achieve this, we make use of a similar idea to the one described above for bootstrap distillation. In the improvement phase, we leverage a supervisor in the loop to guide an advice-conditional surrogate policy qf (a|s,t,c) to perform the new task ttest, obtaining trajectories D = {s0,a0,c0,s1,a1,c1, · · · ,sH ,aH ,cH}Nj=1, then distill this behavior into an advice-independent policy pq (a|s,t) via behavioral cloning. The result is a policy trained using coaching, but ultimately able to select tasks even when no coaching is provided. In Fig 3 left, this improvement process would involve a coach in the loop providing action advice or language sub-goals to the agent during learning to coach it towards successfully accomplishing a task, and then distilling this knowledge into a policy that can operate without seeing action advice or sub-goals at execution time. More formally, the agent optimizes the following objective:
D = {s0,a0,c0,s1,a1,c1, · · · ,sH ,aH ,cH}Nj=1 s0 ⇠ p(s0),at ⇠ qf (at |st ,t,ct),st+1 ⇠ p(st+1|st ,at) max
q E(st ,at ,t)⇠D [logpq (at |st ,t)]
This improvement process, which we call advice distillation, is depicted Fig 2. This distillation process is preferable over directly providing demonstrations because the advice provided can be more convenient than providing an entire demonstration (for instance, compare the difficulty of producing a demo by navigating an agent through an entire maze to providing a few sparse waypoints). Interestingly, even if the new tasks being solved ttest are quite different from the training distribution of tasks p(t), since advice c (for instance waypoints) is provided locally and is largely invariant to this distribution shift, the agent’s understanding of advice generalizes well.
Learning with Off-Policy Advice One limitation to the improvement phase procedure described above is that advice must be provided in real time. However, a small modification to the algorithm allows us to train with off-policy advice. During the improvement phase, we roll out an initiallyuntrained advice-independent policy p(a|s,t). After the fact, the coach provides high-level advice c
h at a multiple points along the trajectory. Next, we use the advice-conditional surrogate policy qf (a|s,t,c) to relabel this trajectory with near-optimal actions at each timestep. This lets us use behavioral cloning to update the advice-free agent on this trajectory. While this relabeling process must be performed multiple times during training, it allows a human to coach an agent without providing real-time advice, which can be more convenient. This process can be thought of as the coach performing DAgger [42] at the level of high-level advice (as was done in in [26]) rather than low-level actions. This procedure can be used for both the grounding and improvement phases. Mathematically, the agent optimizes the following objective:
D = {s0,a0,c0,s1,a1,c1, · · · ,sH ,aH ,cH}Nj=1 s0 ⇠ p(s0),at ⇠ p(at |st ,t),st+1 ⇠ p(st+1|st ,at) max
q E (st ,t)⇠D
a ⇤⇠qf (at |st ,t,c)
[logpq (a⇤|st ,t)]

heading: 3.4 Evaluation Phase: Executing tasks Without a Supervisor
text: In the evaluation phase, the agent simply needs to be able to perform the test tasks ttest without requiring a coach in the loop. We run the advice-independent agent learned in the improvement phase, p(a|s,t) on the test task ttest and record the average success rate.

heading: 4 Experimental Evaluation
text: We aim to answer the following questions through our experimental evaluation (1) Can advice be grounded through interaction with the environment via supervisor in the loop RL? (2) Can grounded advice allow agents to learn new tasks more efficiently than standard RL? (3) Can agents bootstrap the grounding of one form of advice from another?

heading: 4.1 Evaluation Domains
text: BabyAI: In the open-source BabyAI [8] grid-world, an agent is given tasks involving navigation, pick and place, door-opening and multi-step manipulation. We provide three types of advice:
1. Action Advice: Direct supervision of the next action to take. 2. OffsetWaypoint Advice: A tuple (x, y, b), where (x, y) is the goal coordinate minus the
agent’s current position, and b tells the agent whether to interact with an object.
3. Subgoal Advice: A language subgoal such as “Open the blue door.”
2-D Maze Navigation (PM): In the 2D navigation environment, the goal is to reach a random target within a procedurally generated maze. We provide the agent different types of advice:
1. Direction Advice: The vector direction the agent should head in. 2. Cardinal Advice: Which of the cardinal directions (N, S, E, W) the agent should head in. 3. Waypoint Advice: The (x,y) position of a coordinate along the agent’s route. 4. OffsetWaypoint Advice: The (x,y) waypoint minus the agent’s current position.
Ant-Maze Navigation (Ant): The open-source ant-maze navigation domain [14] replaces the simple point mass agent with a quadrupedal “ant” robot. The forms of advice are the same as the ones described above for the point navigation domain.
In all domains, we describe advice forms provided each timestep (Action Advice and Direction Advice) as “low-level” advice, and advice provided less frequently as “high-level” advice. We present experiments involving both scripted coaches and real human-in-the-loop advice.

heading: 4.2 Experimental Setup
text: For the environments listed above, we evaluate the ability of the agent to perform grounding efficiently on a set of training tasks, to learn new test tasks quickly via advice distillation, and to leverage one form of advice to bootstrap another. The details of the exact set of training and testing tasks, as well as architecture and algorithmic details, are provided in the appendix.
We evaluate all the environments using the metric of advice efficiency rather than sample efficiency. By advice efficiency, we are evaluating the number of instances of coach-in-the-loop advice that are needed in order to learn a task. In real-world learning tasks, this coach is typically a human, and the cost of training largely comes from the provision of supervision (rather than time the agent spends interacting with the environment). The same is true for other forms of supervision such as behavioral cloning and RL (unless the human spends extensive time instrumenting the environment to allow autonomous rewards and resets). This “advice units” metric more accurately reflects the true quantity we would like to measure: the amount of human time and effort needed to provide a particular course of coaching. For simplicity, we consider every time a supervisor provides any supervision, such as a piece of advice or a scalar reward, to constitute one advice unit. We measure efficiency in terms of how many advice units are needed to learn a task. We emphasize that this metric makes a strong simplifying assumption—that all forms of advice have the same cost—which is certainly not true for real-world supervision. However, it is challenging to design a metric which accurately captures human effort. In Section 4.7 we validate our method by measuring the real human interaction time needed to train agents. We also plot more traditional sample efficiency measures in Appendix D.
We compare our proposed framework to an RL baseline that is provided with a task instruction but no advice. In the improvement phase, we also compare with behavioral cloning from an expert for environments where it is feasible to construct an oracle.

heading: 4.3 Grounding Prescriptive Advice during Training
text: Fig 4 shows the results of the grounding phase, where the agent grounds advice by training an advice-conditional policy through RL. We observe the the agent learns the task more quickly when provided with advice, indicating that the agent is learning to interpret advice to complete tasks. However, we also see that the agent fails to improve much when conditioning on some more abstract forms of advice, such as waypoint advice in the ant environment. This indicates that the advice form has not been grounded properly through RL. In cases like this, we instead must instead ground these advice forms through bootstrapping, as discussed in Section 3.2.

heading: 4.4 Bootstrapping Multi-Level Feedback
text: Once we have successfully grounded the easiest form of advice, in each environment, we efficiently ground the other forms using the bootstrapping procedure from Section 3.2. As we see in Fig 4, bootstrap distillation is able to ground new forms of advice significantly more efficiently than if we start grounding things from scratch with naïve RL. It performs exceptionally well even for advice forms where naïve RL does not succeed at all, while providing additional speed up for environments
where it does. This suggests that advice is not just a tool to solve new tasks, but also a tool for grounding more complex forms of communication for the agent.

heading: 4.5 Learning New Tasks with Grounded Prescriptive Advice
text: Finally, we evaluate whether we can use grounded advice to guide the agent through new tasks. In most cases, we directly used adviceconditional policies learned during grounding and bootstrapping. However, about half of the BabyAI high-level advice policies performed poorly on the test environments. In this case, we finetuned the policies with a few (<4k) samples collected with rollouts from a lower-level better grounded advice form.
As we can see in Fig 5, agents which are trained through distillation from an abstract coach on average train with less supervision than RL agents. Providing high-level advice can even sometimes outperform providing demonstrations, as the high-level advice allows the human to coach the agent through a successful trajectory without needing to provide an action at each timestep. It is about as efficient to provide low-level advice as to provide demos (when demos are available), as both involve providing one supervision unit per timestep.
Advice grounding on the new tasks is not always perfect, however. For Instance, in BabyAI Test Env 2 in Figure 5, occasional errors in the advice-conditional policy’s interpretation of high-advice result in it being just as efficient efficient to provide low-level advice or demos as it is to provide high-level advice (though both are more efficient than RL). When grounding is poor, the converged final policy may not be fully successful. Baseline methods, in contrast, may ultimately converge to higher rates, even if they take far more samples. For instance, RL never succeeds in AntMaze 3x3 and 6x6 in the plots in Figure 5, but if training is continued for 1e6 advice units, RL achieves near-perfect performance, whereas our method plateaus. This suggests our method is most useful when costly supervision is the main constraint.
The curve in Figure 5 is not entirely a fair comparison - after all, we are not taking into account the advice units used to train the advice-conditional surrogate policy. However, it’s also not fair to include this cost for each test env, since the up-front cost of grounding advice gets amortized over a large set of downstream tasks. Figure 6 summarizes the total number of samples needed to train each model to convergence on the Point Maze test environments, including all supervision provided during grounding and improvement. We see that when we use the best advice form, our method is 8x more efficient than demos, and over 20x more efficient than dense-reward RL. In the PointMaze environment, the cost of grounding becomes worthwhile with only 4 test envs. In other environments such as Ant, it may take many more test envs than the three we tested on. This suggests that our method is most appropriate when the agent will be used on a large set of downstream tasks.

heading: 4.6 Off-Policy Advice Relabeling
text: One limitation of the improvement phase as described Section 4.5 is that the human coach has to be continuously present as the agent is training to provide advice on every trajectory. We relax this requirement by providing the advice in hindsight rather than in-the-loop using the procedure from Section 3.3. Results are shown in Figure 7. IN the Point Maze and Ang envs, this DAgger-like scheme for soliciting advice performs greater than or equal to real-time advice. However, it performs worse in the BabyAI environment. In future work we will explore this approach further, as it removes the need for a human to be constantly present in the loop and opens avenues for using active learning techniques to label only the most informative trajectories.

heading: 4.7 Real Human Experiments
text: through a click interface. (For instance, they could click on the screen to provide a.) See Fig 8.
In the BabyAI environment, we provide OffsetWaypoint advice and compare against a behavioral cloning (BC) baseline where the human provided per-timestep demonstrations using arrow keys. Our method’s is higher variance and has a slightly lower mean success rate, but results are still largely consistent with Figure 5, which showed that for the BabyAI env BC is competitive with our method.
In the Ant environment, demonstrations aren’t possible, and the agent does not explore well enough to learn from sparse rewards. We compare against the performance of an agent coached by a scripted coach providing dense, shaped rewards. We see that the agent trained with 30 minutes of coaching by humans performs comparably to an RL agent trained with 3k more advice units.

heading: 5 Related Work
text: The learning problem studied in this paper belongs to a more general class of human-in-the-loop RL problems [1, 25, 30, 47, 12]. Existing frameworks like TAMER [25, 45] and COACH [30, 4] also use interactive feedback to train policies, but are restricted to scalar or binary rewards. In contrast, our work formalizes the problem of learning from arbitrarily complex feedback signals. A distinct line of work looks to learn how to perform tasks from binary feedback with human preferences, for example by indicating which of two trajectory snippets a human might prefer [10, 21, 47, 27].
These techniques receive only a single bit of information with every human interaction, making human supervision time-consuming and tedious. In contrast, the learning algorithm we describe uses higher-bandwidth feedback signals like language-based subgoals and directional nudges, provided sparsely, to reduce the required effort from a supervisor.
Learning from feedback, especially provided in the form of natural language, is closely related to instruction following in natural language processing [7, 3, 32, 41]. In instruction following problems, the goal is to produce an instruction-conditional policy that can generalize to new natural language specifications of behavior (at the level of either goals or action sequences [24] and held-out environments. Here, our goal is to produce an unconditional policy that achieves good task success autonomously—we use instruction following models to interpret interactive feedback and scaffold the learning of these autonomous policies. Moreover, the advice provided is not limited to task-level specifications, but instead allows for realtime, local guidance of behavior. This provides significantly greater flexibility in altering agent behavior.
The use of language at training time to scaffold learning has been studied in several more specific settings [29]: Co-Reyes et al. [11] describe a procedure for learning to execute fixed target trajectories via interactive corrections, Andreas et al. [2] use language to produce policy representations useful for reinforcement learning, while Jiang et al. [22] and Hu et al. [18] use language to guide the learning of hierarchical policies. Eisenstein et al. [13] and Narasimhan et al. [35] use side information from language to communicate information about environment dynamics rather than high-value action sequences. In contrast to these settings, we aim to use interactive human in the loop advice to learn policies that can autonomously perform novel tasks, even when a human supervisor is not present.

heading: 6 Discussion
text: Summary: In this work, we introduced a new paradigm for teacher in the loop RL, which we refer to as coaching augmented MDPs. We show that CAMPDs cover a wide range of scenarios and introduce a novel framework to learn how to interpret and utilize advice in CAMDPs. We show that doing so has the dual benefits of being able to learn new tasks more efficiently in terms of human effort and being able to bootstrap one form of advice off of another for more efficient grounding.
Limitations: Our method relies on accurate grounding of advice, which does not always happen in the presence of other correlated environment features (e.g. the advice to “open the door,” and the presence of a door in front of the agent). Furthermore, while our method is more efficient than BC or RL, it still requires significant human effort. These limitations are discussed further in Appendix G.
Societal impacts: As human in the loop systems such as the one described here are scaled up to real homes, privacy becomes a major concern. If we have learning systems operating around humans, sharing data and incorporating human feedback into their learning processes, they need to be careful about not divulging private information. Moreover, human in the loop systems are constantly operating around humans and need to be especially safe.
Acknowledgments: Thanks to experiment volunteers Yuqing Du, Kimin Lee, Anika Ramachandran, Philippe Hansen-Estruch, Alejandro Escontrela, Michael Chang, Sam Toyer, Ajay Jain, Dhruv Shah, Homer Walke. Funding by NSF GRFP and DARPA’s XAI, LwLL, and/or SemaFor program, as well as BAIR’s industrial alliance programs.

Review summaries:
This paper presents a framework for teaching agents using language advice. The framework consists of three stages: (i) grounding language advice to action, (ii) learning an unconditioned policy that mimics the advice-conditioned policy (iii) evaluating the unconditioned policy. Experiments are conducted in maze navigation environments. Results demonstrate the effectiveness of the framework compared with imitation learning and reinforcement learning.  

The authors propose a reinforcement learning approach where humans can  provide different forms of advice to complete a task. The proposed approach, called CAMDPs, follows a two-stage process: Grounding and Improvement.   During grounding the system learns how to interpret the advice. The result of this phase is a surrogate policy. Grounding learns a mapping from advice to actions using information from the true reward function. Grounding can be seen as an ordinarily multitask RL, where each task is a piece of advice, and the agent learns using normal RL how to achieve the advice.   Distillation is used to learn a policy using supervised learning and bootstrap distillation uses the previously learned policies for individual advice to learn a policy for a particular task.  During improvement the coach provides advice to solve a new task. A very similar approach is followed as bootstrap distillation, a new task is solved using advice followed by a distillation process.  It is expected for the learned policies to be general enough to be applied in other, although similar, domains.  The proposed system is evaluated in three, relatively similar, domains. As expected, it is experimentally shown that the proposed system learns faster with advice. 

The paper presents an interesting framework (CAMDPs) for teacher-in-the-loop-based RL, that takes into account rich interactive advice. The authors also show how the advice is interpreted in the Coaching augmented MDP. The framework comprises of 3 stages: grounding, improvement, and evaluation. In the grounding phase, the agent learns to interpret an 'advice', in the improvement phase the agent learns a policy using the advice and finally in the evaluation phase the learner deploys it's advice-independent policy. The claim is that the proposed pipeline helps a learner learn more efficiently, as it is able to account for richer forms of feedback. The method was tested on 3 rich RL environments: Baby AI, Ant Env, and Point Navigation Env, and evaluated based on its advice efficiency. The evaluations shpw that an agent learns quicker when presented with advice than when it is not.

The paper proposes a mechanism Coaching Augmented MDP for agents to learn from interactive expert feedback to avoid assumptions on reward functions and agent infrastructure. Here, structured advice is provided by experts to guide the agent learning process. A policy learning algorithm is designed for interpreting and acting on the advice to solve a problem. Experiments are conducted on discrete and continuous control problems to show how such mechanism outperforms traditional RL models.