Teachable Reinforcement Learning via Advice Distillation
Authors: Olivia Watkins
Abhishek Gupta
Trevor Darrell
Pieter Abbeel
Jacob Andreas
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning
Human in the Loop RL

Abstract: Training automated agents to complete complex tasks in interactive environments is challenging: reinforcement learning requires careful hand-engineering of reward functions, imitation learning requires specialized infrastructure and access to a human expert, and learning from intermediate forms of supervision (like binary preferences) is time-consuming and extracts little information from each human intervention. Can we overcome these challenges by building agents that learn from rich, interactive feedback instead? We propose a new supervision paradigm for interactive learning based on "teachable" decision-making systems that learn from structured advice provided by an external teacher. We begin by formalizing a class of human-in-the-loop decision making problems in which multiple forms of teacher-provided advice are available to a learner. We then describe a simple learning algorithm for these problems that first learns to interpret advice, then learns from advice to complete tasks even in the absence of human supervision. In puzzle-solving, navigation, and locomotion domains, we show that agents that learn from advice can acquire new skills with significantly less human supervision than standard reinforcement learning algorithms and often less than imitation learning.

Teachable Reinforcement Learning via Advice Distillation
Olivia Watkins
Trevor Darrell UC Berkeley trevorβeecs www.eeer.eeeeee...oo edu edu
UC Berkeley
oliviawadkinsseeeeeeeee.... edu
Pieter Abbeel UC Berkeley pabbeel@cs berkelee.beeeeee.c edu
Jacob Andreas
Abhishek Gupta
MIT
UC Berkeley
jda@mit.
abhigupta@berkeley.oo
Abstract
Training automated agents to complete complex tasks in interactive environments is challenging: reinforcement learning requires careful hand-engineering of reward functions, imitation learning requires specialized infrastructure and access to a human expert, and learning from intermediate forms of supervision (like binary preferences) is time-consuming and extracts little information from each human intervention. Can overcome these challenges by building agents that learn from rich, interactive feedback instead? We propose a new supervision paradigm for interactive learning based "teachable" decision-make makiing systems that learn from structured advice provided by an external teacher. We begin by formalizing decision making problems in which multiple forms
class of human-in-the-te of seacher-provide advice are available to a learner. We then describe simple learning algorithm for these problems that first learns to interpret advice, then learns from advice to complete tasks even in the absence of human supervision. In puzzle-solving, navigation, and locomotion domains, from advice can acquire new skills with significantly less human supervision than standard reinforcement learning algorithms and often less than imitation learning.
show that agents that learn
Introduction
Reinforcement learning (RL) offers a promising paradigm for building agents that can learn complex behaviors from autonomous interaction and minimal human effort. In practice, however, significant human effort is required to design and compute the reward functions that enable successful RI the reward functions underlying some of RL's most prominent success stories involve significant domain expertise and elaborate instrumentation of the agent and environment 37 Even with this complexity, reward ultimately more than a scalar indicator how good particular state is relative to others. Rewards provide limited information about how to perform tasks, and reward-dris driven RL agents must perform significant exploration and experimentation within an environment to lear effectively. A number of alternative paradigms for interactively learning policies have emerged alternatives, such as imitation learning [40 20 550 DAgger 43), and preference learning [1O, [. But these existing methods are either impractically low bandwidth (extracting little
49
15]
information from each human intervention) 25 30 IC or require costly data collection 4 23
has proven challenging to develop training methods that are simultaneously expressive and efficient
enough to rapidly train agents to acquire novel skills.
Human learners, by contrast, leverage numerous, rich forms of supervision: joint attention
physical corrections [ and natural language instruction 9. For human teachers, this kind of
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

Grounding
Improvement
(c) Evaluation
Couching
State
Tge. cooching g"
cuaching
x²a s)
Bam right
+1
wapgole
Browel
-022
stroight
canforenntt
advicc distilation
salvice
disillatioo
R g(a c,)
g(a s.
xT(a (s
(agent tales actions without advice))
simple conching- conditional policy
comples coaching conditional policy
policy
Figure 1: Three phases of teachable reinforcement learning. During the grounding phase (a). we train an advice-conditional policy through RL g(als, r, that can interpret simple form of advice c². During the improvement phase (b), an external coach provides real-time coaching, which the agent uses learn more complex advice forms and ultimately an advice-indepe indeeenddnnn policy (als, t). During the evaluation phase, the advice-indeppndent policy x(als, z) is executed to accomplish a task without additional human feedback.
coaching is often no more costly to provide than scalar measures of success, but significantly more bagh-bandwidth, low-effort communication
informative for learners. In this way, human learners as a means to flexibly acquire new concepts or skills 46. B3 Importantly, the interpretation of some of these feedback signals (like language), is itself learned, but can be bootstrapped from other forms of communication: for example, the function of gesture and attention can be learned from intrinsic rewards 691 these in turn play key role in language learning [1
This paper proposes a framework for training automated agents using similarly rich interactive supervision. For instance, given agent learning a policy to navigate and manipulate objects in simulated multi-room object manipulation problem (e.g., Fig Ileft), we train agents using not just reward signals but advice about what actions to take ('move left"), what waypoints move towards ("move towards (1,2)"). and what sub-goals to accomplish ("pick up the yellow ball"), offering human supervisors toolkit of rich feedback forms that direct and modify agent behavior. To do so, we introduce a new formulation interactive learning, the Coaching-Augmente Process (CAMDP), which formalizes the problem of learning from privileged supervisory signal provided via an observation channel. We then describe an algorithmic framework for learning in CAMDPs via alternating advice grounding and distillation phases. During the grounding phase, agents learn associations between teacher-provided advice and high-value actions in the environment; during distillation, agents collect trajectories with grounded models and interactive advice, then transfer information from these trajectories to fully autonomous policies that operate without advice. This formulation allows supervisors to guide agent behavior interactively, while enabling agents to internalize this guidance to continue performing tasks autonomously once the supervisor is no longer present. Moreover, this procedure can be extended enable bootstrapping of grounded models that use increasingly sparse and abstract advice types, leveraging some types of feedback to ground others. Experiments show that models trained via coaching can learn new tasks more efficiently and with 20x less human supervision than naïve methods for RL across puzzle-solving solving and locomotion domains
Markov Decision
g navigation [14]
In summary, this paper describes: (1) general framework (CAMDPs) for human-in-the-loop RL with rich interactive advice; (2) an algorithm for learning in CAMDPs with a single form of advice; (3) an extension of this algorithm that enables bootstrapped learning of multiple advice types; and finally (4) set of empirical evaluations on discrete and continuous control problems the BabyAl | and D4RL [E] environments. It thus offers a groundwork for moving beyond reward signals in interactive learning, and instead training agents with the full range of human communicative modalities.
2 Coaching Augmented Markov Decision Processes
To develop our procedure for learning from rich feedback, we begin by formalizing the environments and tasks for which feedback is provided. This formalization builds on the framework of multi-task RL and Markov decision processes (MDP), augmenting them with advice provided by coach the loop through an arbitrary prescriptive channel of communication. Conider the grid-world environment depicted in Fig 3 left [8) Tasks in this environment specify particular specific desired goal states; e.g. "place the yellow ball in the green box and the blue key in the green box" or "open all doors in

---

the blue room." In multi-task RL, a learner's objective is produce a policy I x(a r) that maximizes reward in expectation over tasks t. More formally, a multi-task MDP is defined by a 7-tuple P. p(s)). T.P(()) p()),, where y denotes the state space, a/ denotes the action space, M (5 T. RR
p: x L + R220 denotes the transition dynamics, r: r: a x I m> IR denotes the reward function, p: y + Ryo denotes the initial state distribution, Y [o, 1] denotes the discount factor and PTT) denotes the distribution over tasks. The objective in a multi-task MDP is to learn a policy Ty that maximizes the expected sum of discounted returns in expectation over tasks; maxe JE (Tg ((T) (a-an((he)))) - rss, at, r)]. Tr(p(c)
Why might additional supervision beyond the reward signal be useful for solving this optimization problem? Suppose the agent in Fig B is in the (low-value) state shown in the figure, but could reach a high-value state by going "right and up" towards the blue key. This fact is difficult to communicate through a scalar reward, which cannot convey information about alternative actions. side channel for providing this type of rich information at training -ime would be greatly beneficial.
We model this follows: a coaching-augmentte
MDP (CAMDP) consists of ordinary multi- task MDP augmented with a set of coaching functions & ==f 8 .8,...., ,".""",,)))))) 86', where each C provides different form of feedback to the agent. Like reward function, each coaching function models a form of supervision provided externally to the agent (by coach); these functions may produce informative outputs densely (at each timestep) or only infrequently. Unlike rewards, which give agents feedback on the desirability of states and actions they have already experienced, this coaching provides information about what the agent should do next. [ As shown in Figure advice can take many forms, for instance action advice (c"), waypoints (c') ), language sub- cub-goals (c ), or any other local information relevant task completion Coaching
CAMDP is useful if provides an agent local guidance on how to proceed toward a goal that is inferrable from the agent's 's current observation, when the mapping from observations and goals to actions has not yet been learned.
As in standard reinforcement learning in an multi-task MDP, the goal in a CAMDP is to learn a policy (. st +) that chooses an action based on Markovian state s, and high level task information without interacting with c². However, we allow learning algorithms to use the coaching signal el to learn this policy more efficiently at training time (although this is unavailable during deployment). For instance, the agent in Fig 3 can leverage hints "go left" or "move towards the blue key" to guide its exploration process but it eventually must learn how to perform the task without any coaching required. Section 3 decribes an algorithm for acquiring this independent, multi-task policy Tg (- sr, 7) from coaching feedback, and Section presents an empirical evaluation of this algorithm.
3 Leveraging Advice via Distillation
3.1 Preliminaries
The challenge of learning in a CAMDP is twofold: first, agents must learn to ground coaching signals in concrete behavior; second, agents must learn from these coaching signals to independently solve the task of interest in the absence of any human supervision. To accomplish this, we divide agent training into three phases: (1) a grounding phase, (2) an improvement phase and (3) an evaluation phase.
In the grounding phase, agents learn how to interpret coaching. The result of the grounding phase a surrogate policy q(a, Se, T,C) that can effectively condition on coaching when it is provided in the training loop. As we will discuss in Section 3.2 this phase can also make use of a bootstrapping process in which more complex forms of feedback are learned using signals from simpler ones.
During the improvement phase, agents use the ability to interpret advice to learn new skills. Specif- ically, the learner is presented with novel task Test that was not provided during the grounding phase, and must learn to perform this task using only a small amount of interaction in which advice is provided by human supervisor who is present in the loop. This advice, combined with the
While the design of optimal coaching strategies and explicit modeling of coaches are important research topics [16], this paper assumes that the coach is fixed and not explicitly modeled. Our empirical evaluation use both scripted coaches and human-in-the-logy feedback.
2 When only single form of advice is available the agent, we omit the superscript for clarity.
3

---

learned surrogate policy q(ar|s, r, can be used to efficiently acquire an advice-independent policy a(a (5. T). which can perform tasks without requiring any coaching.
Finally, in the evaluation phase, agent performance is evaluated on the task Test by executing the advice-independent, multi-task policy x( (s, Fesa)in the environment.
3.2 Grounding Phase: Learning to Interpret Advice
The goal of the grounding phase is to learn mapping from advice to contextually appropriate actions, so that advice can be used for quickly learning of training tasks p(t). As the purpose of these training environments is purely to ground coaching, sometimes called "advice", the tasks may be much simpler than test-time tasks. During this phase, the agent uses access to a reward function r(s,a.c) c), as well as the advice c(s,a) to learn a surrogate policy go (a)s, r.c). The reward function r(s.a.c) is provided by the coach during the grounding phase only and rewards the agent for correctly following the provided coaching, not just for accomplishing the task. Since coaching instructions (e.g. cardinal directions) are much easier to follow than completing a full task, grounding can be learned quickly. The process of grounding is no different than standard multi-task RL, incorporating advice c(s,a) a) another component of the observation space. This formulation makes minimal assumptions about the form of the coaching c.
tasks. In this phase, run RL on distribution
During this grounding process, the agent's 's optimization objective is:
Σ Ellnnaac)s
m() = E
r-p((t) ar arrugg (ar r.c)
(I)
Bootstrapping Multi-Level Advice The previous section described how to train an agent to in- terpret single form of advice c. In practice, a coach might find it useful to use multiple forms of advice-for for instance high-level language sub-goals for easy stages of the task and low-level action advice for more challenging parts of the task. While high-level advice can be very informative for guiding the learning of new tasks in the improvement phase, it can often be quite difficult to ground quickly pure RL. Instead of relying on RL, we can bootstrap the process of grounding one form of advice c from a policy g(a|s, 7, d that can interpret different form of advice d' In particular, we can use a surrogate policy which already understands (using the grounding scheme described above) low-level advice q(a|s, r,c') to bootstrap training of a surrogate policy which understands higher-level advice q(a)s, s, ). We call this process "bootstrap distillation".
Intuitively, we use a supervisor in the loop to guide an condiccnco policy that can interpret a low-level form of advice qo (als, r,c') to perform a training task, obtaining trajectories 9 {550, ao, cg. cb), (s,,a),,,, cf) ass,,n,,h cn, c ) then distilling the demonstrated behavior via supervised learning into a policy qo, (als, t, r,ch)) that can interpret higher-level advice to perform this
(a) In-the-loop advice
(5.0*, r.
coached rollouts from conditional policy
distillation unconditional policy
(b) Off-policy advice
(s,
(x.a
uncosched sollouts from unconditional policy
hindsight coaching and action relabeling
distillation
unconditional policy
Figure 2: Illustration of the procedure of advice distillation in the on-policy and off-policy settings. During on-policy advice distillation, the advico-conditional surrogate policy q(a|s, t,c) coached get optimal trajectories. These trajectories are then distilled into an unconditional model T[ass, t) using supervised learning. During oof-po distillation, trajectories are collected by the unconditional policy and trajectories are relabeled with advice after the fact. After this, we use the advice-conditional policy q(a)s, r.c) to relabel trajectories with optimal actions. These trajectories can then be distilled into unconditional policy.

---

new task without requiring the low level advice any longer. More specifically, we make use of an input remapping solution, seen Levine et [28], where the policy conditioned on advice c' is used to generate optimal action labels, which are then remapped to observations with a different form of advice e as input. To bootstrap the understanding of an abstract form of advice c² from a more low level one d', the agent optimizes the following objective to bootstrap the agent's s understanding of one advice type from another:
S = [s50 ao, do- d)), S1,,ai.c ai,,a,,,, ct). ... ( as (n)l))
so ~ p(sn), qn (s I, r,e),, p(s++1)s,,,)
max 16 (s. (.. A))0 loggor (,)ss,,,)))) 02
With this procedure, we only need to use RL to ground the simplest, fastest-learned advice form, and we can use more efficient bootstrapping to ground the others.
3.3 Improvement Phase: Learning New Tasks Efficiently with Advice
At the end of the grounding phase, we have an advice-following agent qo (ala,, r.g) that can interpret various forms of advice. Ultimately, we want policy x(a)s, r) which is able to succeed at performing the new test task Test, without requiring advice at evaluation time. To achieve this, we make use of a similar idea to the one described above for bootstrap distillation. In the improvement phase, we leverage supervisor in the loop to guide an advice-conditional surrogate policy q4 (a)s, t,c) to perform the new task Test, obtaining trajectories 2 = {50, 40: Co $1, "1, -.. ,H (0; j=1' then distill this behavior into an advice-independen policy na (als, T) via behavioral cloning. The result is a policy trained using coaching, but ultimately able to select tasks even when no coaching is provided. In Fig D left, this improvement process would involve a coach in the loop providing action advice or language sub- sub-goals to the agent during learning to coach it towards successfully accomplishing a task, and then distilling this knowledge into policy that can operate without seeing action advice or sub-goals at execution time. More formally, the agent optimizes the following objective:
{so, ap. Co. S1, AH ) sp ~ p(so), a ~ go go (, 8,41) ~ p(8++1)9,,))
max E +) [log (a (s,,
This improvement process, which call advice distillation, is depicted Fig This distillation process is preferable over directly providing demonstrations because the advice provided can be more convenient than providing an entire demonstration (for instance, compare the difficulty of producing a demo by navigating an agent through an entire maze to providing a few sparse waypoints). Interestingly, even if the new tasks being solved Test are quite different from the training distribution of tasks p(t), since advice (for instance waypoints) is provided locally and is largely invariant to this distribution shift, the agent's 's understanding of advice generalizes well.
Learning with Off-Policy Advice One limitation to the improvement phase procedure described above is that advice must be provided in real time. However, a small modification to the algorithm allows us to train with off-policy advice. During the improvement phase, we roll out an initially- untrained advicc--indepnnee
policy I[als, z). After the fact, the coach provides high-level advice ctt at multiple points along the trajectory. Next, use the go (o(a ss, r,c) to relabel this trajectory with near- optimal actions at each timestep. This lets us use behavioral cloning to update the advice-free agent on this trajectory. While this relabeling process must be performed multiple times during training, it allows a human to coach an agent without providing real-time advice, which can be more convenient. This process
surrogate policy
be thought of the coach performing DAgger [2] at the level of high-level advice (as was done in in ( rather than low-level actions. This procedure can be used for both the grounding and improvement phases. Mathematically, the agent optimizes the following objective:
2 = (so, N0: Co. $1,44101,, ,SH TH: CH NN 1=
~ p(so). T(a sy. t), ~ 1(st+1|s,,a) [log no (a" (s, t)
max (,,+))
at (4.6
5

---

3.4 Evaluation Phase: Executing tasks Without a Supervisor
In the evaluation phase, the agent simply needs to be able to perform the test tasks Gest without requiring coach in the loop. We run the udvice-independen agent learned in the improvement phase, ss T) on the test task Fest and record the average success rate.
4 Experimental Evaluation
We aim answer the following questions through our experimental evaluation (1) Can advice be grounded through interaction with the environment via supervisor the loop RL? (2) Can grounded advice allow agents to learn new tasks more efficiently than standard RL? (3) Can agents bootstrap grounding of one form advice from another?
4.1 Evaluation Domains
+3
up key"
Navigne
Navigate to
[,13,
Disetton
Deection [.1,, 23
atster:
Woppeia::
Woypoier (3.
Dinetio::
Dieetion: Wex
Subgoet:
"Gu to the yellow doxe"
Woypoint:
Waypurnt (3,4)
Waygoine
Figure 3: Evaluation Domains. (Left) BabyAl (Middle) Point Maze Navigation (Right) Ant Navigation. The associated task instructions shown, as well the types of advice available each domain.
BabyAI: In the open-source BabyAI grid-worke an agent is given tasks involving navigation, pick and place, door-opening and multi-step manipulation. We provide three types of advice:
1. Action Advice: Direct supervision of the next action to take.
2. WffsetWa Advice: A tuple (x, b), where (x, y) is the goal coordinate minus the agent's current position, and b tells the agent whether to interact with an object.
3. Subgoal Advice: language subgoal such as "Open the blue door."
2-D Maze Navigation (PM): In the 2D navigation environment, the goal is to reach random target within a procedurally generated maze. We provide the agent different types advice:
1. Direction Advice: The vector direction the agent should head in.
2. Cardinal Advice: Which of the cardinal directions (N, S, E, W) the agent should head in.
3. Waypoint Advice: The (x.y) position of coordinate along the agent's route.
4. OffsetWaypoint Advice: The (x.y) waypoint minus the agent's current position.
Ant-Maze Navigation (Ant): The open-source ant-maze navigation domain [4] replaces the simple point agent with quadrupedal "ant" robot. The forms of advice are the same as the ones described above for the point navigation domain.
In all domains, describe advice forms provided each timestep (Action Advice and Direction Advice) "low-level" advice, and advice provided less frequently as "high-level" advice. We present experiments involving both scripted coaches and real human-in-the- tho-loop advice.

---

4.2 Experimental Setup
For the environments listed above, we evaluate the ability of the agent to perform grounding efficiently on set of training tasks, to learn new test tasks quickly via advice distillation, and to leverage one form of advice bootstrap another. The details of the exact set of training and testing tasks, as well as architecture and algorithmic details, are provided in the appendix.
We evaluate all the environments using the metric of advice efficiency rather than sample efficiency. By advice efficiency, we are evaluating the number of instances of boach-in-the-lowe advice that are needed in order learn task. In real-world learning tasks, this coach is typically human, and the cost of training largely comes from the provision of supervision (rather than time the agent spends interacting with the environment). The same is true for other forms of supervision such as behavioral cloning and RL (unless the human spends extensive time instrumenting the environment to allow autonomous rewards and resets). This "advice units" metric more accurately reflects the true quantity we would like to measure: the amount of human time and effort needed to provide a particular course of coaching. For simplicity, we consider every time a supervisor provides any supervision, such as piece of advice or scalar reward, to constitute one advice unit. We measure efficiency in terms of how many advice units are needed to learn a task. We emphasize that this metric makes a strong simplifying assumption that all forms of advice have the same cost-which is certainly not true for real- world supervision. However, it is challenging to design a metric which accurately captures human effort. In Section 4.7 we validate our method by measuring the real human interaction time needed to train agents. We also plot more traditional sample efficiency measures in Appendix D
We compare our proposed framework to an RL baseline that is provided with a task instruction but no advice. In the improvement phase, we also compare with behavioral cloning from an expert for environments where it is feasible to construct an oracle.
4.3 Grounding Preseriptive Advice during Training
Mear
Directon (PN)
10
Cendiaal (11))
Wayesam
Diection (Ari)
45
Lo
0.5
Meric
Cardinal (ken)
1.0
Waypoirt (Ar))
(bubyal)
10
Suppoal (babyal
66
Outi: O recccoonniionn Advicc
9.0
655
1.
3.5
Outs: Waypcint Advkic
Addis
Advice
AdMoc
Ours: Offect Wiypoint Advice
Oums Subpos//Cardral Advice
RL Trarinn
Babion
Raselnn: Advice
All curves are trained with shaped-reward RL. We compare agents which condition on high-level advice (shades of blue) to ones with access to lowelevel advice (red) to an advice-free baseline (gray). Takeaways: (a) the agent able to ground advice, which suggests that our advice-condiiional policy may be useful for coaching; (b) grounding certain high-level advice forms through RL is slow, which is why bootstrapping is necessary. Right: Bootstrapping is able to quickly use existing grounded advice forms (OffsetWaypoint for Point Maze and Ant Maze envs, ActionAdvice for BabyAl) to ground additional forms of advice.
Figure 4: Left: Performance during the grounding phase (Section 3.2
Fig shows the results of the grounding phase, where the agent grounds advice by training an iivice-conditionas policy through RL. We observe the the agent learns the task more quickly when provided with advice, indicating that the agent is learning to interpret advice to complete tasks. However, we also see that the agent fails to improve much when conditioning some more abstract forms of advice, such as waypoint advice in the ant environment, This indicates that the advice form has not been grounded properly through RI. In cases like this, we instead must instead ground these advice forms through bootstrapping, as discussed in Section 3.2
4.4 Bootstrapping Multi-Level Feedback
Once we have successfully grounded the easiest form of advice, in each environment, we efficiently ground the other forms using the bootstrapping procedure from Section 3.2 As we see in Fig 4 bootstrap distillation is able to ground new forms of advice significantly more efficiently than if we start grounding things from scratch with naive RI. It performs exceptionally well even for advice forms where naive RL does not succeed at all, while providing additional speed up for environments
7

---

where it does. This suggests that advice is not just tool to solve new tasks, but also a tool for grounding more complex forms of communication for the agent.
4.5
Learning New Tasks with Grounded Prescriptive Advice
Point Mare 1005 Haze]
Point Maze (7x10 Mane)
Point Mase
Direction
Cardinal
Waypoint
Offsee
RL
Oncile
606 Muze
0.02
0.95
0.9910001
0.99:0001
0.27 0.01
0.87
Maze
0.75
6.9
0.73
10c10 Maze
0.00 +
0.07
86220000
0.85 0.00
0.11
+.0.4
0.64
1.4
Ant Maze 1363 Mozel
Supcoss Aahie
Art Maze 1636 Mozel
13x13 Mase
0. 0:004
0.35 0.08
0.22
0.45 1
0.08 a: 0.04
0.28 0.00
Aur Maze
Direction
Curiial
Waypoint
Offset
RL
8.5
4.5
3x3 Muze
0.25 0.117
0.38
0.77
0.210.11
0.0000
Mare
0.0110004
0.32.10.11
0.511025
0.55
00-00
7x10 Maze
0.0
0.0
0.000.00
0.-000
0.0=0.0
featback
feesback
RabyAl
Action Advice OrteetWayponn
Subgoai
RL.
Cracle BC
Bobyd Test
Bonydi
Test kiv
0.31 :: 0.15
0.51 0.14
0.53
0.00000
0.14
6.6
Test Eav 2
0.53:1
0.16
0.43
0.18
0.660.06
Test
0.04
0.26
0.240.08
0.4
0.16=00.04
lest
00=00
0:04::033
Test Eav 5
0.07
0.003
0.13
0.210.111
0.0.0.0
0.05 1.002
Advisz Lanis
Adtics Units
165
Test Eav 6
0.4
0.48
0.28
0.17=00.9
0.43
Ours: Drsccsorkciinn Adwis
Ours: Sulegaanaarchi Adeis
Test
0.32
0.42
0.14
0.01000011
0.26
Cors: Maypart Actice
Dead tnc Sahevirel Coning
Figure 5: Learning new tasks through distillation. The agent uses an already- advice channel to perform the distillation process from Section [.3 to train an advice-free agent. Results show the success rate of the advice-free new agent. Left, we show representative curves for a few environments. Colors designate supervision used: shades of blue high level advice; red = low level advice; black oracle demonstrations; gray shaped rewards. Right: We show success rates (mean, std) over 3 seeds for larger set of environments. Runs are bolded std intervals overlapped with the highest mean. Success rales are evaluated at 3e5 steps Point Maze and Ant Maze and 5e5 steps for Baby AL. Takeaway: once advice is grounded, in general it is most efficient to teach the agents new tasks by providing high-advice. There are occasional exceptions, discussed in Appendix G
Finally, we evaluate whether we can use grounded advice to guide the agent through new tasks. In most cases, we directly used advice- conditional policies learned during grounding and bootstrapping. However, about half of the BabyAl Al high-level advice policies performed poorly on the test environments. In this case, we finetuned the policies with a few (<4k) samples collected with rollouts from a lower-level better grounded advice form.
247
Total Soperstiion Cost
18
3.4
3.0
Figure 6: "Best advice" is OffsetAdvice. Y-axis includes advice from both grounding and improvement across all four Point Maze test envs. RL results stretch off the plot, indicating we unable run RI. for long enough converge to the success rates of the other methods.
As we can see in Fig 5 agents which are trained through distillation from an abstract coach on average train with less supervision than RL agents. Providing high-level advice can even sometimes outperform providing demonstrations, as the high-level advice allows the human to coach the agent through a successful trajectory without needing to provide an action at each timestep. It is about as efficient to provide low-level advice as to provide demos (when demos are available), as both involve providing one supervision unit per timestep.
Advice grounding on the new tasks is not always perfect, however. For Instance, in BabyAl Test Env 2 in Figure 5 occasional errors in the advice-conditional policy's 's interpretation of high- result in it being just as efficient efficient to provide low-level advice or demos as it is to provide high-level advice (though both are more efficient than RL). When grounding is poor, the converged final policy may not be fully successful. Baseline methods, in contrast, may ultimately converge to higher rates, even if they take far more samples. For instance, RL never succeeds in AntMaze 3x3 and 6x6 in the plots in Figure 5, but if training is continued for le6 advice units, RI. achieves near-perfect performance, whereas our method plateaus. This suggests our method is most useful when costly supervision is the main constraint.
8

---

The curve in Figure 5 is not entirely fair comparison after all, we are not taking into account the advice units used to train the advice-conditional surrogate policy. However, it's also not fair to include this cost for each test env, since the up-front cost of grounding advice gets amortized over a large set of downstream tasks. Figure summarizes the total number of samples needed to train each model to convergence on the Point Maze test environments, including all supervision provided during grounding and improvement. We see that when we use the best advice form, our method is 8x more efficient than demos, and over 20x more efficient than dense-reward RL. In the PointMaze environment, the cost of grounding becomes worthwhile with only 4 test envs. In other environments such as Ant, it may take many more test envs than the three tested on. This suggests that our method is most appropriate when the agent will be used on a large set of downstream tasks.
4.6 Off-Policy Advice Relabeling
One limitation of the improvement phase as described Section 4.5 is that the human coach has to be continuously present the agent is training provide advice on every trajectory. We relax this requirement by providing the advice in hindsight rather than in-the-loop using the procedure from Section 3.3 Results shown in Figure /
IN the Point Maze and Ang envs, this DAgger-like scheme for soliciting advice performs greater than or equal to real-time advice. However, it performs worse in the BabyAI environment. In future work we will explore this approach further, as it removes the need for a human to be constantly present in the loop and opens avenues for using active learning techniques label only the most informative trajectories.
4.7 Real Human Experiments
To validate the automated evaluation above (and determine whether our "advice unit" metric is good proxy for human effort), we performed an additional set of experiments with human-in-the-logg coaches. Advice- conditional surrogate policies were pre-trained to follow advice using a scripted coach. The coaches (all researchers at U.C. Berkeley) then coached these agents through solving new, more complex test environments. Afterwards, an an advice-fit free policy was distilled from the successful tra- jectories. Humans provided advice through a click interface. (For instance, they could click on the screen provide a.) See Fig
Garere
Malte lrbe 145
Matee Uniss 36
MiAe Usire 365
cae unis 1n
Babyy Test ber
6
921
1.39
Addio Units
165
AdACO Uhits 165 MMi this 185
DAgger Advice
Real-Time Advice
Figure 7: All curves show the success rate of advice-free free policy trained via distillation from an advice-conditional surrogate policy. All curves use the Offssetayyoontt advice form, and results are aver- aged over three seeds. Takeaway: DAgger performs well on some environments (Point Maze, Ant) but poorly on others (BabyAD).
In the Baby AI environment, we provide WffsetWaypoint advice and compare against a behavioral cloning (BC) baseline where the human provided per-timestep demonstrations using arrow keys. Our method's is higher variance and has a slightly lower mean success rate, but results are still largely consistent with Figure 5 which showed that for the BabyAl env BC is competitive with method.
In the Ant environment, demonstrations aren't 't possible, and the agent does not explore well enough to learn from sparse rewards. We compare against the performance of an agent coached by scripted coach providing dense, shaped rewards. We see that the agent trained with 30 minutes of coaching by humans performs comparably to an RI. agent trained with 3k more advice units.
Related Work
The learning problem studied in this paper belongs a more general class of comma--in-the-iee RL problems [1I 25 30
47 [2]. Existing frameworks like TAMER 25 [5] and COACH [30] 4 also use interactive feedback to train policies, but are restricted to sealar or binary rewards. contrast, our work formalizes the problem of learning from arbitrarily complex feedback signals. A distinct line of work looks to learn how to perform tasks from binary feedback with human preferences, for example by indicating which of two trajectory snippets a human might prefer
10
21 47 27

---

These techniques receive only single bit of information with every human interaction, making human supervision consuminn and tedious. In contrast, the learning algorithm we describe uses higher-bandwide feedback signals like language-based subgoals and directional nudges, provided sparsely, to reduce the required effort from a supervisor.
Learning from feedback, especially provided in the form of natural lan- guage, is closely related to instruction following in natural language process-
1.0
Ant Advice
1.0
Ant 0 -Samples
0.5
0.5
ing 7 3 32 41. In instruction fol- lowing problems, the goal to pro- duce an that can generalize to new natural lan- guage specifications of behavior (at the level of either goals or action se- quences A4I and held-out environ- ments. Here, our goal is to produce an unconditional policy that achieves good task success autonomously we instruction following models in- terpret interactive feedback and scaf- fold the learning of these autonomous policies. Moreover, the advice pro- vided is not limited to task-level spec- ifications, but instead allows for real- time, local guidance of behavior. This provides significantly greater flexibil- ity in altering agent behavior.
Ours
AL 1x
AL AL10x 10x
RL 1.00x
RL 1000x
RL 3000x
RL RL5000x
0.0
Ours
RL 70x
50x
RL 100x
RL
1.0
Human BabyA)
policy
Advice
Samples
1.0
1.0
Ours
0.5
0.5
0.0
Ours
AL 1x
AL 100x
RL 1000x
AL 3080x
RI RL5000x
0.0
Ours
RL 1x
RL 10x
RL 5.0x
RL 100x
AL
Figure 8: Left, Middle: We compare the success of an advice-free policy trained in two test envs with real human coaching RL policy trained with a scripted reward. "RL 10x" means RL policy received 10x more advice units (left) or samples (middle). Right: success of advice- fivice-free policies trained with 30 mins of human time. Humans either coach the agent with our method or provide demos. Sample sizes are n=2 per condition for Ant, n=3 per condition for Baby AI, so the results are suggestive not conclusive.
The use of language at training time to scaffold learning has been studied in several more specific settings [29]: Co- Co-Reyes et al. [I]] describe procedure for learning execute fixed target trajectories via interactive corrections, Andreas et al. [2] use language to produce policy representations useful for reinforcement learning, while Jiang al. [2 and Hu et al. [8]] use language to guide the learning of hierarchical policies. Eisenstein et al. [33 and Narasimhan et al. (35) use side information from language communicate information about environment dynamics rather than high-value action sequences. In contrast to these settings, we aim to use interactive human in the loop advice to learn policies that can autonomously perform novel tasks, even when human supervisor is not present.
6 Discussion
Summary: In this work, we introduced a new paradigm for teacher in the loop RL, which we refer to as coaching augmented MDPs. We show that CAMPDs cover wide range of scenarios and introduce a novel framework to learn how to interpret and utilize advice in CAMDPs. We show that doing so has the dual benefits of being able to learn new tasks more efficiently in terms of human effort and being able to bootstrap one form of advice off of another for more efficient grounding.
Limitations:
Our method relies accurate grounding of advice, which does not always happen in the presence of other correlated environment features (e.g. the advice to "open the door," and the presence of door in front of the agent). Furthermore, while our method is more efficient than BC RI., it still requires significant human effort. These limitations are discussed further in Appendix G
Societal impacts: As human in the loop systems such the described here are scaled up to real homes, privacy becomes a major concern. If we have learning systems operating around humans, sharing data and incorporating human feedback into their learning processes, they need to be careful about not divulging private information. Moreover, human in the loop systems are constantly operating around humans and need to be especially safe.
Thanks to experiment volunteers Yuqing Du, Kimin Lee, Anika Ramachandran, Philippe Hansen-Fstruch, Alejandro Escontrela, Michael Chang. Sam Toyer, Ajay Jain, Dhruv Shah, Homer Walke. Funding by NSF GRFP and DARPA's XAI, LwL.L, and/or SemaFor program, as well as BAIR's 's industrial alliance programs.
Acknowledgments:
10

---

References
[1] D. Abel, J. Salvatier, A. Stuhlmüller, and O. Evans. Agent-agnostic human--n-ith---oopreinneee reinfrree ment learning. CoRR, hts/1701.04079) 2017. http:///aarii.oorraa///////////
[2] J. Andreas, D. Klein, and S. Levine. Learning with latent language. In M. A. Walker, H. Ji, and
A. Stent, editors, NAACL, 2018.
[3] Y. Artzi and L. Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Trans. Assoc. Comput. Linguistics, 1:49 62, 2013. URL https
//tac12013. cs. cowwmbia.....om hataaaadddojjssssdeeaeeeeeeeeeee https////ndd.ppppaaaaaattttteeee pttp://acl//artiicl///eee////
[4] D. Arumugam, J. K. Lee, S. Saskin, and M. L. Littman. Deep reinforcement learning from volicy-dependens human feedback. CoRR, abs/1902.04257,, 2019. URL http:httpp/////// org/ abs/1902. 14221999.44257
[5] A. Bajesy, D. P. Losey, M. K. O'Malley, and A. D. Dragan. Learning robot objectives from physical human interaction. In Conference on Robot Learning (CoRL), 2017.
[6] D. S. Brown, W. Goo, and S. Niekum. Better-than-demmonsraaao imitation learning via automatically-raakeed demonstrations. In Conference on Robot Learning (CoRL), 2019.
[7] D. L. Chen and R. J. Mooney. Learning to interpret natural language navigation instructions from observations. In W. Burgard and D. Roth, editors, AAAI, 2011.
[8] M. Beevalier-Boisverr D. Bahdanau, Lahlou, Willems, C. Saharia, T. H. Nguyen, and Y. Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. ICLR, 2019.
[9] S. Chopra, M. H. Tessler, and N. D. Goodman. The first crank of the cultural ratchet: Learning and transmitting concepts through language. A. K. Goel, C. M. Seifert, and C. Freksa, editors, CogSei, 2019.
[10] F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, NeurlPS, 2017.
[11] D. Co-Reyes, A. Gupta, S. Sanjeev, Altieri, Andreas, J. DeNero, Abbeel, and Levine. Guiding policies with language via meta-learning. In ICLR, 2019.
[12] C. A. Cruz and T. Igarashi. survey on interactive reinforcement learning: Design principles and open challenges. In R. Wakkary, K. Andersen, W. Odom, A. Desjardins, and M. G. Petersen, editors, DIS '20: Designing Interactive Systems Conference 2020, Eindhoven, The Netherlands, July 6-10, 2020, pages 1195-1209. ACM, 2020. doi: (0.1145/3355223633355222 URL httpttttt///oo....
111111..11133333322 3395525
[13] J. Eisenstein, J. Clarke, D. Goldwasser, and D. Roth. Reading to learn: Constructing features from semantic abstracts. In EMNL, 2009.
[14] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4RL: datasets for deep data-driver reinforcement learning. CoRR, abs/2004.07219, 2020. URL httphtttt////xi.....o wwwpsii..rr//bb////0/e abs/2004
07219
[15] A. Gupta, J. Yu, T. Z. Zhao, V. Kumar, A. Rovinsky, K. Xu, T. Devlin, and S. Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. arXiv preprint 2rXiv:2104.11200 2021.
[16] D. Hadfield-Menell A. Dragan, Abbeel, and S. Russell. Cooperative inverse reinforcement learning. arXiv preprint 1606::6606.333 2016.
[17] D. Hejna, Pinto, and P. Abbeel. Hierarchically decoupled imitation for morphological transfer.
In International Conference on Machine Learning, pages 4159-4171. PMLR, 2020.
11

---

[18] H. Hu, D. Yarats, Q. Gong, Y. Tian, and M. Lewis. Hierarchical decision making by generating and following natural language instructions. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buu E. B. Fox, and R. Garnett, editors, NeurlPS, 2019.
[19] D. Y.-T. Hui, M. Chevalier-Boisven D. Bahdanau, and Y. Bengio. Babyai 1.1. arXiv preprint arXiv:2007.11 207777777777 2020.
[20] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation Iearning: A survey of learning methods. ACM Comput. Surv., 50(2):211--2::35, 2017. doi: 10. 10.1145/305491.. URL https //doi. oww...r..11.00 1145/3054912
[21] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in atari. In Bengio, H. M. Wallach, Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett, editors, NeurlPS, 2018.
[22] Y. Jiang, S. Gu, K. Murphy, and C. Finn. Language as an abstraction for hierarchical deep reinforcement learning. In NeurIPS, 2019.
[23] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine, Qt-opt: Scalable deep reinforcement learn- ing for vision-based robotic manipulation. CoRR, abs/1806. 10293, 2018. URL http: / //arxiv. /aaaxii.rrg//a awtaaaas/B10 18011109911111333
[24] S. Karamcheti, E. C. Williams, D. Arumugam, M. Rhee, N. Gopalan, L. L. S. Wong, and S. Tellexx A tale of two draggns: A hybrid approach for interpreting action-oriented and goal-oriented instructions. In M. Bansal, C. Matuszek, J. Andreas, Y. Artzi, and Y. Bisk, editors, RoboNLP@ACL, 2017.
[25] W. B. Knox and P. Stone. TAMER: Training Agent Manually via Evaluative Reinforcement. In IEEE 7th International Conference on Development and Learning, August 2008.
[26] H. Le, N. Jiang, A. Agarwal, M. Dudík, Y. Yue, and H. Daumé. Hierarchical imitation and reinforcement learning. In International conference on machine learning, pages 2917-2926. PMLR, 2018
[27] K. Lee, L. Smith, and P. Abbeel. Pebble: Feedback-efficien interactive reinforcement learning via relabeling experience and unsupervised pre-training In International Conference on Machine Learning, 2021.
[28] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-133,, 2016.
[29] J. Luketina, N. Nardelli, G. Farquhar, J. N. Foerster, Andreas, E. Grefenstette, S. Whiteson, and T. Rocktäschel. A survey of reinforcement learning informed by natural language. In IJCAI, 2019.
[30] MacGlashan, M. K. Ho, T. Loftin, Peng, G. Wang, D. L. Roberts, M. E. Taylor, and M. L.. Littman. Interactive learning from policy-dependent human feedback. In ICML, 2017.
[31] N. M. McNeil, M. W. Alibali, and J. L. Evans. The role of gesture in children's comprehension of spoken language:now they need it, now they don't. 't Journal of Nonverbal Behavior, 24 (2):131-150, 2000. doi: 10. 1006657929803
URL httpt:t////oo.. httpp:://o.l..rg//1.0002222 1023///
[32] H. Mei, M. Bansal, and M. R. Walter. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In D. Schuurmans and M. P. Wellman, editors, AAAI, 2016.
[33] J. H. Morgan, N. T. Uomini, L. E. Rendell, L. Chouinard- Chouinard-Thuly. Thuly, S. E. Street, H. M. Lewis, C. P. Cross, Evans, R. Kearney, I. de la Torre, A. Whiten, and K. N. Laland. Experimental evidence for the co-evolution hominin tool-making teaching and language. Nature Communications, (1):6029, 2015. doi: 10. 10.18//commms7099 URthttps:: //doi org/ 10. 1100///commm702222
[34] P. Mundy and W. Jarrold. Infant joint attention, neural networks and social cognition. Neural Networks, 23(99))9885 997, 2010. doi: 10. www1//..eumee.2010.ooo 000. URL, wtt.:/tttt/////t org h0:/10.20 1016/1 neunet. wewnet.2010.000
12

---

[35] K. Narasimhan, R. Barzilay, and T. S. Jaakkola. Deep transfer in reinforcement learning by language grounding. CoRR, abs/1 2017. URL http://ara.v..oggabb/// 00133
[36] K. Nguyen, D. Mísra, R. Schapire, M. Dudík, and P. Shafto. Interactive learning from activity description. arXiv preprint arXiv:2102.070244 2021.
[37] OpenAl. Openai five. arxiv, 2018.
[38] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej. M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving rubik's cube with robot hand. CoRR, abs/1910.07113, 2019. URL http://aaxi:xi org abs///b///// 1ttps/9911.00113
[39] Poli, G. Serino, R. B. Mars, and S. Humnius. Infants tailor their attention to maximize learning. Science Advances, 6(39), 2020, doi: 126//ccadv..bbb50000 URL https://
advances werraces.scieeccooo sciencemag. org/ tetg/content/eee 2039///bbb05555
[40] D. Pomerleau. ALVINN: an autonomous land vehicle in a neural network, In D. S. Touretzky, editor, NeurlPS, 1988
[41] Roh, C. Paxton, Pronobis, Farhadi, and D. Fox. Conditional driving from natural language instructions. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, CoRL, 2019.
[42] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. Proceedings of the fourteenth international conference on artifi- cial intelligence and stutistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[43] S. Ross, G. J. Gordon, and D. Bagnell. reduction of imitation learning and structured prediction no-regret online learning. In G. J. Gordon, D. B. Dunson, and M. Dudík, editors, AISTATS, 2011.
[44] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In ICML, 2015.
[45] G. Warnell, N. R. Waytowich, V. Lawhern, and P. Stone. Deep TAMER: interactive agent shaping in high-dimensional state spaces. In AAAI, 2018.
[46] S. R. Waxman and D. B. Markow. Words invitations form categories: evidence from 12- to 13-month-old infants. Cogn Psychol, 2953)257--02, Dec 1995.
[47] R. Zhang, F. Torabi, L. Guan, D. H. Ballard, and P. Stone. Leveraging human guidance for deep reinforcement learning tasks. In S. Kraus, editor, IJCAI, 2019.
[48] Zhu, Y. Zhu, X. Chang, and X. Liang. Vision-language navigation with self-supervised auxiliary reasoning tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10012-10022, 2020.
[49] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine. The ingredients of real world robotic reinforcement learning. In International Conference Learning Representations, 2020.
[50] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In D. Fox and C. Gomes, editors, AAAI, 2008.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section g
13

---

(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section x
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Ys]] This work does not actually use human subjects, and is largely done in simulation. But we have included a discussion in Section
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Math used as theory/formalism, but we don't I make any provable claims about it.
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Appendix for link to URL and run instructions in the README the github repo.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] See Appendix A.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] All plots were created with 3 random seeds with std error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Envs we used are cited in section 41
(b) Did you mention the license of the assets? [Yes] This is in Appendix
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We published the code and included all environments and assets as part of this
(d) Did you discuss whether and how consent was obtained from people whose data you'l using/curating? [Yes] We used three open source domains and collected our own data on these domains.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [No] We did not include full text since we didn't 't use an exact script, but we summarized the instructions and included images of the environments used.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] Only human involvement was data collection with our system.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] Human testers were volunteers
14


Reviews:
[{"review_id": "xprLPzlQaB", "reviewer": "Reviewer_TZyF", "summary": "The paper presents an interesting framework (CAMDPs) for teacher-in-the-loop-based RL, that takes into account rich interactive advice. The authors also show how the advice is interpreted in the Coaching augmented MDP. The framework comprises of 3 stages: grounding, improvement, and evaluation. In the grounding phase, the agent learns to interpret an 'advice', in the improvement phase the agent learns a policy using the advice and finally in the evaluation phase the learner deploys it's advice-independent policy. The claim is that the proposed pipeline helps a learner learn more efficiently, as it is able to account for richer forms of feedback. The method was tested on 3 rich RL environments: Baby AI, Ant Env, and Point Navigation Env, and evaluated based on its advice efficiency. The evaluations shpw that an agent learns quicker when presented with advice than when it is not.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n\n* The paper tackles an ambitious problem of learning from high-level advice in teacher-in-the-loop-based RL systems. The problem formulation was clean, and the three steps of the framework were well explained. The idea of using bootstrapping to interpret multi-level advice was particularly interesting.\n* The experimental evaluation carried out on the rich RL environments was promising. The experimental analysis on the 3 RL environments was exhaustive.\n\n\n\n**Cons**\n\n* Evaluation: While the evaluation on the 3 rich RL environments was done in terms of the advice efficiency metric, it would be interesting to see how this method compares to the existing methods such as imitation learning, in terms of their sample efficiency. Furthermore, it would also be interesting to investigate the effect of richer/ more structured advice grammar and analyze further if there was a way to determine the optimal sequence of different advice types within a task, to improve performance fastest. It would be great to see the effect of this framework on a more real setting: perhaps on teaching an agent to solve mathematical equations, which comprises a more challenging advice set.\n\n\n\n**General Comments**\nThis paper opens avenues for analysis of such rich feedback-based RL systems. It presents an interesting framework to analyze the richness of advice-based feedback w.r.t task. The different advice types investigated, and the framework presented for their distillation into a policy is novel. The evaluation of such a framework on the simulated RL environments looks promising, and there is scope to explore more interesting/complex advice grammar.\n\n\n*Originality*: Novel Approach\n\n*Clarity*: Largely clear. \n\n*Quality*: Good\n\n*Significance*: Important stepping stone to future work in this direction\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper addresses its limitations of assuming that the provided advice by the teacher is always optimal. They have left incorporating noisy advice for future work. However, this work is still far from deployment to be concerned about its negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "j0BxjMr9HVp", "reviewer": "Reviewer_df4E", "summary": "The authors propose a reinforcement learning approach where humans can \nprovide different forms of advice to complete a task. The proposed\napproach, called CAMDPs, follows a two-stage process: Grounding and\nImprovement. \n\nDuring grounding the system learns how to interpret the advice. The\nresult of this phase is a surrogate policy. Grounding learns a mapping\nfrom advice to actions using information from the true reward\nfunction. Grounding can be seen as an ordinarily multitask RL, where\neach task is a piece of advice, and the agent learns using normal RL\nhow to achieve the advice. \n\nDistillation is used to learn a policy using supervised learning and\nbootstrap distillation uses the previously learned policies for\nindividual advice to learn a policy for a particular task.\n\nDuring improvement the coach provides advice to solve a new task. A very\nsimilar approach is followed as bootstrap distillation, a new task\nis solved using advice followed by a distillation process.\n\nIt is expected for the learned policies to be general enough to be\napplied in other, although similar, domains.\n\nThe proposed system is evaluated in three, relatively similar,\ndomains. As expected, it is experimentally shown that the proposed\nsystem learns faster with advice.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Some positive aspects of this paper are that it can use different\ntypes of advice and it is tested on different domains. It is also\npositive that it can combine low-level advice and policies into\nhigher-level policies to solve more complex tasks.\n\nOn the negative side, the description of the method is not completely\nclear and the proposed method seems to heavily depend on human\nintervention in order to work. It also depends on some strong\nassumptions in terms of how the advice is given, which makes it\ndifficult to assess how useful the proposed system will behave under\nmore realistic conditions. \n\n\nIt is not clear how the sub-goals are related to higher goals. It\nseems to be that the user is responsible for providing this, but needs\nclarification. \n\nAlso it is not clear how the pieces of advice are learned. Since more\nabstract advice can be learned from more concrete advice, does it mean\nthat the system needs to first learn the less abstract advice (e.g.,\ndirection) before more abstract advises (e.g, cardinal and then\nway-points) or can the advises be learned independently.\n\nIt is not clear how the proposed approach of combining low level\npolicies to learn more abstract policies compares to learning\noptions or other hierarchical learning approaches.\n\nThere are different types of advice (at different abstraction levels),\nbut in the evaluation phase they are treated equally. A more detailed\nevaluation, considering these differences would certainly help to\nunderstand the role of each type of advice.\n\nAt every state the teacher chooses the coaching function (if any) to\napply. This is currently hardwired, but it is not clear what was the\nrationale to define this process (when to give advice and which type\nof advice to use).\n\nThe authors should clarify the the types of advice is restricted to\nthose that suggest possible actions. For instance, a user may tell the\nagent that what it is doing is wrong without providing any specific\nhint of which action may be the right one.\n\nIt would be very helpful to give an example of one \"typical\" episode\nwith advice (e.g., when and what type of advice is given, how often, etc.)\n\nIt not clear how general is the approach. It is shown that the\nlearned advice can be used in larger versions of the same original\ntask. Can the advice be transferred between different tasks, e.g., from\nBabyAI to Ant Navigation?\n\nAlso, the system assumes ideal conditions with perfect advice and no\nnoise. It is not clear how the proposed approach will behave under\nmore realistic conditions. It is easy to image scenarios where a user\ngives wrong pieces of advises, with delays, or gives an advice that it\nis too general from which the system is not able to learn, etc.\n\nSmall typos:\n- a advice\n- other We\n\n----------\nI have read the comments of the other reviewers and the authors' response.\nI appreciate that the authors have performed further experiments to \nstrengthen their paper and I think that the paper could be publish, \nhowever I still maintain my original acceptance score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes the authors adequately addressed the limitations and positive societal impact", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "P4CQaAXNd9W", "reviewer": "Reviewer_NG65", "summary": "The paper proposes a mechanism Coaching Augmented MDP for agents to learn from interactive expert feedback to avoid assumptions on reward functions and agent infrastructure. Here, structured advice is provided by experts to guide the agent learning process. A policy learning algorithm is designed for interpreting and acting on the advice to solve a problem. Experiments are conducted on discrete and continuous control problems to show how such mechanism outperforms traditional RL models.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThough the proposed Coaching Augmented MDP is closely related to multi-task MDP and distillation, it still has novelty in processing more interactive feedback with sub-goals along with an un-conditional policy \n\nQuality\nThe paper proposes a framework for multi-task MDP with teacher in the loop using different forms of coaching functions called advice. They also propose an algorithm to perform exploration using these coaching signals while the actual task can be performed without these signals. The stage-wise learning helps to achieve this. The technical details of  these method are explained in detail in the paper. \nExperiments are conducted both in discrete and continuous control problems to verify that the proposed method outperforms traditional RL methods. The authors can clarify if the proposed mechanism is more time-consuming than the traditional RL techniques.\n\nClarity\nThe paper is well written and easy to follow. \n\nSignificance:\nThe work is significant to the community as it introduces the paradigm of teacher in the loop RL which considers interactive feedback during learning. The work looks like a promising step towards a broader class of feedback loop during RL training. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper points out that the teacher in the loop is not a real-time human which can be noisy and compromised which are scenarios that need to be considered. The proposed method is a work towards having autonomous systems operate with rich interactive teacher feedback instead of just reward signals", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1cKHCnGBzr2", "reviewer": "Reviewer_F7yY", "summary": "This paper presents a framework for teaching agents using language advice. The framework consists of three stages: (i) grounding language advice to action, (ii) learning an unconditioned policy that mimics the advice-conditioned policy (iii) evaluating the unconditioned policy. Experiments are conducted in maze navigation environments. Results demonstrate the effectiveness of the framework compared with imitation learning and reinforcement learning.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: this work combines several pre-existing ideas, namely grounded language learning and policy distillation, in a novel way to enable teaching using language advice. It shares a lot of similarities with [6] (in hierarchical learning) and [3,4] (in learning to interpret advice) but the specific frameworks are not exactly the same. I would like to request comparison with these works in the paper and in the rebuttal. I also suggest reading [2] for a discussion on the drawbacks of reinforcement learning and imitation learning compared to a language-based learning framework. \n\nMissing related work:\n\n[1] Learning Rewards from Linguistic Feedback. https://arxiv.org/pdf/2009.14715.pdf\n\n[2] Interactive Learning from Activity Description. https://arxiv.org/pdf/2102.07024.pdf\n\n[3] Vision-based navigation with language-based assistance via imitation learning with indirect intervention. https://arxiv.org/pdf/1812.04155.pdf\n\n[4] Help, Anna! Visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. https://arxiv.org/pdf/1909.01871.pdf\n\n[5] Vision-and-Dialog Navigation. https://arxiv.org/pdf/1907.04957.pdf\n\n[6] Hierarchical Imitation and Reinforcement Learning. https://arxiv.org/pdf/1803.00590.pdf\n\nIn fact, I strongly suggest the authors add more citations from the related work sections of these papers. \n\n**Quality**: \n\nThe method is technically sound as it combines well-established methods. My concerns are mostly about evaluation:\n- Using advice unit to measure communication efficiency oversimplifies comparison. A piece of language advice contains a lot of more information than a scalar reward. A scalar reward is represented by a 64-bit floating-point number in most machines; a 5-word advice in which each word is drawn from a 10^5-word vocabulary can be represented with log(10^5) * 5 ~ 80 bits; an action in a space consisting of 4 actions can be represented by only 2 bits. The bottom line is that communication efficiency of an advice should be based on the amount of information conveyed in the advice. A more sophisticated metric is to add the cognitive effort of humans when eliciting the advice but that may be too difficult to measure. Nevertheless, the advice unit metric in this work is overly naive and I do not advocate using it as a standard metric for later work to follow. \n- The results for imitation learning are absurdly low. Is it because the training set is too small? I request the authors to describe the dataset used for training with this baseline. I strongly suggest using \"behavior cloning\" instead of \"imitation learning\" because the latter is a much broader term. In fact, when the advice is an Action Advice, the approach is equivalent to DAgger [7], which is a form of imitation learning. I encourage the authors to clarify this point in the paper. \n\n[7] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. https://www.ri.cmu.edu/pub_files/2011/4/Ross-AISTATS11-NoRegret.pdf\n\n**Clarity**: The paper is well-written and is easy to follow. However, I would like to suggest the authors merge the \"bootstrapping with multi-level advice\" with section 4.3 because they implement the same idea. In fact, this framework would be more general if the authors adapted a goal-conditioned RL framework (e.g. [8]). Then, this framework can be described as low-level policies being composed to form higher-level policies until the highest-level policies are learned. \n\n[8] Hindsight Experience Replay. https://arxiv.org/pdf/1707.01495.pdf\n\n**Significance**: The paper studies an important direction: learning from language interaction, which enables non-expert humans to teach ML-based agents using their natural mode of communication. The presented framework is relatively general and can be potentially applied to a diverse range of tasks, especially that it captures the hierarchical structure of linguistically described tasks. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors admit the limitations of the language and environments used in their experiments. The language is scripted and the environment is relatively simple (compared to the Matterport3D environment in [4]). I would like to point out several potential challenges when scaling this framework to more complex environments:\n- Advice arises in contexts. In reality, we may not be able to ask humans to just \"give me an advice\" without grounding the question in the context of a task. Hence, it may make more sense to merge the grounding phase and the improvement phase as in [4], where humans give advice and also demonstrate the meaning of the advice at the same time so that the humans have contexts to give advice. \n- Giving advice too frequently could be tedious for humans. To reduce human effort, for example, [4] learns a policy that allows the agent to actively decide when it is necessary to ask humans for advice. \n- In complex environments, reinforcement learning could be too inefficient for language grounding. In that case, imitation learning could be used instead (as in [3, 4, 6]).  \n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were very impressed by the amount of additional material provided during the author response phase. The extensive replies addressed all major concerns of the reviewers. Incorporating all that in the paper will be a challenge though.
Decision: Accept (Poster)