Learning When and What to Ask: a Hierarchical Reinforcement Learning Framework
Authors: Khanh Xuan Nguyen
Yonatan Bisk
Hal Daumé III
Venue: ICLR 2022 Submitted (2022)
Keywords: human-agent interaction
reinforcement learning
navigation

Abstract: Reliable AI agents should be mindful of the limits of their knowledge and consult humans when sensing that they do not have sufficient knowledge to make sound decisions. We formulate a hierarchical reinforcement learning framework for learning to decide when to request additional information from humans and what type of information would be helpful to request. Our framework extends partially-observed Markov decision processes (POMDPs) by allowing an agent to interact with an assistant to leverage their knowledge in accomplishing tasks. Results on a simulated human-assisted navigation problem demonstrate the effectiveness of our framework: aided with an interaction policy learned by our method, a navigation policy achieves up to a 7× improvement in task success rate compared to performing tasks only by itself.  The interaction policy is also efficient: on average, only a quarter of all actions taken during a task execution are requests for information. We analyze benefits and challenges of learning with a hierarchical policy structure and suggest directions for future work.

Under review as a conference paper at ICLR 2022
LEARNING WHEN AND WHAT TO ASK: A HIERARCHI- CAL REINFORCEMENT LEARNING FRAMEWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Reliable Al agents should be mindful of the limits of their knowledge and con- sult humans when sensing that they do not have sufficient knowledge make sound decisions. We formulate hierarchical reinforcement learning framework for learning to decide when to request additional information from humans and what type of information would be helpful to request. Our framework extends partially-obseredd Markov decision processes (POMDPs) by allowing an agent interact with assistant to leverage their knowledge in accomplishing tasks. Results on a simulated human-assisted navigation problem demonstrate the ef- fectiveness of framework: aided with interaction policy learned by our method, a navigation policy achieves up to a 7x improvement in task success rate compared to performing tasks only by itself. We find that the ability to request subgoals enables the agent to generalize effectively to tasks in unseen environ- ments. We analyze benefits and challenges of learning with a hierarchical policy structure and suggest directions for future work.
INTRODUCTION
Human-agent communication at deployment time has been under-explored in machine learning, where the traditional focus has been on building agents that can accomplish tasks on their own (full autonomy). Nevertheless, enabling an agent to exchange information with humans during its operation can potentially enhance its helpfulness and trustworthiness. The ability to request and interpret human advice would help the agent accomplish tasks beyond its built-in knowledge, while the ability to accurately convey when and why it is about to fail would make the agent safer to use.
In his classical work, Grice (9755) outlines the desired characteristics of cooperative communica- tion, commonly known as the Gricean maxims of cooperation. Among these characteristics are informativeness (the maxim of quantity) and faithfulness (the maxim of quality). Human-agent communication in current work has fallen short in these two aspects. Traditional frameworks like imitation learning and reinforcement learning employ limited communication protocols where the agent and the human can only exchange simple intentions (requesting low-level actions or rewards Torrey & Taylor (2013); Knox & Stone (2009). More powerful frameworks like (Nguyen & Daumé IIII 2019] Nguyen et al.) 2019; Kim et al.] 2019) allow the agent to process high-level instructions from humans, but the agent still only requests generic help. Recent work in natural language processing endows the agent with the ability to generate rich natural language utterances (Camburu et al. 2018 Rao & Daumé III 2018 De Vries al.] 2017 Das al.] 2017), but the communication is not faithful in the sense that the agent only mirrors human human-generater utterances without grounding its communication in self-perception of its (in)capabilities and (un)certainties. Essentially it learns to convey what human may be concerned about, not what if is concerned about.
This paper presents hierarchical reinforcement learning framework named HARI Hfuman-Assiste Reinforced Interaction), which supports richer and more faithful human-agent communication. Our framework allows the agent to learn to convey intrinstic needs for specific information and to incor- porate diverse types of information from humans to make better decisions. Specifically, the agent in HARI is equipped with three information seeking intentions: in every step, it can choose to re- quest more information about (i) its current state, (ii) the goal state, (iii) subgoal state which, reached, helps it make progress on the current task. Upon receiving a request, the human can transfer new information to the agent by giving new descriptions of the requested state. These de-

---

Under review as a conference paper at ICLR 2022
Environment
OOAL yor locadry
SUB
Peat whhe
Hallway D
CUR
BONE jatreed
th mosh
DOME joineet
Kicceen
Figure 1: An illustration of the HARI framework in a human- navigation task. An agent can only observe part of an environment and is asked to find a mug in the kitchen. An assistant communicates with the agent and can provide it with information about the environment and the task. Initially (A) it may request more information about the goal, but may not know enough about where it currently is. For example, at location B, due to limited perception, it does not recognize that it is in living room and stands next to a couch. can obtain such information from the assistant. If the current task becomes too difficult (like at location C), the agent can require the assistant to provide a simpler subtask which, if accomplished. helps it make progress on the main task. The agent maintains stack of tasks and always executes the task at the top. When the agent receives a (sub)taskk pushes the tsub) to the top of the stack. When wants to stop executing (sub) tsub) it pops the (sub)task from the stack (e.g.. at location D). At location E, the agent empties the stack and terminates its exccution.
scriptions will be incorporated as new inputs to the agent's decision-making policy. The human thus can transfer any form of information that can be interpreted by the policy (e.g., asking the agent to execute skills that it has learned, giving side information that connects the agent to a situation it is more familiar with). Because the agent's s policy can implement a variety of model architectures and learning algorithms, our framework opens up many possibilities for human-agent communication.
To enable faithful communication, we teach the agent to understand its intrinsic needs by interacting with the human and the environment (rather than imitating human behaviors). By requesting differ- ent types of information and observing how much each type of information enhances its decisions, the agent gradually learns to determine which information is most useful to obtain in given situa- tion. With this capability, at deployment time, it can choose when and what information to ask from the human to improve its task performance. To demonstrate the effectiveness of HARI, we simulate a numan-assisted navigation problem where an agent has access to only sparse information about its current state and the goal, and can request additional information about these states. On tasks that take place in previously unseen environments, the ability to ask for help improves the agent's 's suc- cess rate by 7x higher compared performing tasks only by itself. This human-assisted agent even outperforms an agent that always has access to dense information in unseen environments, thanks to the ability to request subgoals. We show that performance of the agent can be further improved by recursively asking for subgoals of subgoals. We discuss limitations of the policy's 's model and feature representation, which suggest room for future improvements.
2 MOTIVATION: LIMITATIONS OF THE STANDARD POMDP FRAMEWORK
We consider environment defined by a partially-observed Markov decision process (POMDP)
E = A, D, p) with state space S. action space A. state- transition function T : S x A (SS), cost function c: S x A + R. description space D. and description function p : - A A(D Here, A()) denotes the set of all probability distributions over a set y. We refer to this environment as the operation environment because is where the agent operates to accomplish tasks.
Each task in the environment is defined as tuple ($1. 91, df) where is the start state, 91 is the goal state, and di is a limited description of gi- Initially, a task (#1, $1,,91,43 dd) is sampled from a task
We use the term "description" in lieu of "observation" in the POMDP formulation to emphasize two prop- erties of the information the agent has access to for making decisions: (i) the information can he in various modalities and (ii) the information can be obtained via not only perception, but also communication.
2

---

Under review as a conference paper at ICLR 2022
distribution X. An agent starts in $1 and is only given the goal description df. It has to reach the goal state 91 within H time steps. Let 9i and de be the goal state and goal description being executed at time step t, respectively. In standard POMDP, 9, 91 and de di for 1 S $ H. But later, will enable the agent to set new goals via communication with humans.
At any time step f, the agent does not know its true state 82 but only receives a description dł p ( of the state. Generally, the description can include any information coming from any knowledge source (e.g,, an RGB image and/or verbal description describing the current view). Given da and dy. the agent then makes a decision az E A, transitions to the next state 814-1 ~ T(s ax), and receives cost C2 _ c(s;, az). A special action Odone E A is taken when the agent decides to terminate its execution. The goal of the agent is to reach 91 with minimum cumulative cost C(7) = -i=1 Ct, where r = ($1, dj, d1, $2, d2. Q2,..., SH dir is an execution of the task.
As the agent does not have access to its true state, it can only make decisions based the (ob- servable) partial execution Tı. = (di, ai...., df). Kaelbling et al. (9998) introduce the notion of a belief state b E (S)), which sufficiently summarizes a partial execution distribution over the state space S. In practice, when S is continuous or high-dimensional, representing and updating full belief state (whose dimension is (S]) is intractable. We follow Hausknecht & Stone (2015), using recurrent neural networks to learn compact representation of partial executions. We denote by by a representation of the partial execution T1: and by B the set of all possible representations.
.........
The agent maintains an operation policy मे : B x DD + A(A) that maps a belief state be and a goal description d to a distribution over A. The learning objective for solving a standard POMDP is to estimate an operation policy that minimizes the expected cumulative cost of performing tasks:
min df) (11....)))) [C(r)]
where Px(- | (1,d)) is the distribution over executions generated by policy given start state 81 and goal description d². In a standard POMDP, an agent performs tasks by executing its own operation policy without asking for any external assistance. Moreover, the description function p and the goal description di are assumed to be fixed during a task execution. As seen from Equation 1, given fixed environment and task distribution, the expected performance of the agent solely determined by the operation policy Å. Thus, the standard POMDP framework does not provide any mechanism for improving the agent's performance other than enhancing the operation policy.
3 LEVERAGING HUMAN KNOWLEDGE TO BETTER ACCOMPLISH TASKS
We introduce an assistant into the operation environment, who can provide information about the environment's 's states. We assume the agent possesses a pre-learned operation policy Ω. This pol- icy serves as the common ground between the agent and the assistant, which is prerequisite for communication between them to occur. For example, this policy represents a set of basic tasks that agent has mastered and the assistant can ask the agent to perform. In general, the more knowledge encoded in this policy, the more effectively the agent can communicate with and leverage help from the assistant. Our goal is to learn interaction policy vho (parametrized by 0) that controls how the agent communicates with the assistant to gather additional information. The operation policy will be invoked by the interaction policy if the latter decides that the agent does not need new information and wants to take an operating action.
The assistant aids the agent by giving new (current or goal) state descriptions, connecting the agent to situations on which it can make better decisions. Consider an abject-finding navigation problem, where a robot has been trained reliably navigate the kitchen from the living room of house. Suppose the robot is then asked to "find a mug", an object that it has never heard of. The assistant can help the robot accomplish this task by giving a more informative goal description "find a mug in the kitchen", relating the current task to the kitchen-findee finding task that the robot has been familiar with. The robot may also have problems with localization: it knows how to get the kitchen from the living room but it may not realize that it is currently the living room. In this case, giving current- state description that specifies this information provides the robot with a useful hint on what actions to take next.
Our framework allows the assistant to convey any form of information that the agent can incorpo- rate into its input. As discussed in 2 the notion of "state description" in our framework is general,
3

---

Under review as a conference paper at ICLR 2022
capturing various types of information, including but not limited to visual perception and verbal de- scription. Communication between the agent and the assistant can be flexibly enriched by designing the agent's operation policy to be able to consume the forms of information of interest (e.g., a policy that takes natural language as input).
Communication with the Assistant. The assistant is present all the time and knows the agent's s current state and the goal state Gt. It is specified by two functions: a description function P.A S D > (D) and subgoal function WA : S - A(S). P d) specifies the probability of giving d' as the new description of state given current description d. W 8.9) indicates the probability proposing g' as a subgoal given a current state and a goal state g.
At time step t, the assistant accepts three types of request from the agent:
(a) CUR: requests a new description of 8, and receives di+1 ~ PA (: Se, di);
(b) GoAL: requests a new description of gt and receives d ~ PA (: gt, di);
(c) SUB: requests a description of subgoal I++1 and receives d' ~ PA (- 94+1, where
-t-1 ~ WA (- 91) and @ is an empty description.
Interaction Policy. The action space of the interaction policy te consists of five actions: {CUR, GOAL,, SUB, Do, DONE}. The first three actions correspond to making the three types of request that the assistants accepts. The remaining two actions are used to traverse in the operation environment:
(d) Do: executes the action at - arg max.cA & (a bf, d.) The agent transitions to new operation state St+1 ~ T(tt, at
(e) DONE: determines that the current goal 9r has been reached If gr is main goal (s: 91), the task episode ends. If Sr is a subgoal (g. * 91), the agent may choose a new goal to follow. Our problem formulation leaves it open on what goal should be selected next.
By selecting among these actions, the interaction policy essentially decides when to ask the assistant for additional information, and what types of information to ask for. Our formulation does not specifyy the input space of the interaction policy, as this space depends on how the agent implements its goal memory (i.e. how stores and retrieves the subgoals). the next section, we introduce an instantiation where the agent uses stack data structure to manage (sub)goals.
4 HIERARCHICAL REINFORCEMENT LEARNING FRAMEWORK
In this section, we describe the HARI framework. We first formulate the POMDP environment that the interaction policy acts in, referred to as the interaction environment ( 4.T. Our construction employs goal stack to manage multiple levels of (sub)gools (5.2)) A goal stack stores all the tasks the agent has been assigned but has not yet decided to terminate (by choosing the DONE action). It is updated in every step depending the taken action. We design cost function (6.44) that specifies a trade-off between the cost of taking actions (acting cost) and the cost of not completing a task (task error).
4.1 INTERACTION ENVIRONMENT
Given operation environment E (S.A,T, D.,), the interaction environment constructed on with:
top of E' is a POMDP =
State space S = S D GL where Gr is the set of all goal stacks containing at most L elements (L a hyperparameter). Each state ø = (s, d=, G) E S a tuple of an operation state S. its description d' D, and goal stack G Gz. Each element in the goal stack G is tuple (g, dd) ) of a goal state E S and its description d E D;
Action space A (CUR, GOAL, SUB, DO, DONE);
State-transition function T = T₃ TG where T, S x D - A(S x D) and To
Gr x A AG
Cost function ē: (Sx9L) A R (defined in $4.4 to trade off acting cost and task error);
Note that the agent may falsely decide that goal has been reached.
4

---

Under review as a conference paper at ICLR 2022
Description space D D x 9% where 9% is the set of all coal-deseription stacks of size L. At any time, the agent cannot access the environment's 's goal stack G, which contains true goal states. Instead, it can only observe the descriptions in G. We call this partial stack goal-description stack, denoted by G Description function p
DD where p(s) p(s, d², G) (d' Gd). Unlike in the standard POMDP formulation, this description function is deterministic.
A belief state by summarizes partial execution (3), a,,.... , 3t). We formally define the interaction policy as vy B + ((A), where B is the set of all interaction belief states.
4.2
GOAL STACK
A goal stack is an ordered set of tasks that the agent has not declared completion (by calling DONE). The initial stack G₁ {g), d contains the main goal 91 and its description de.. Let G, be the goal stack at time step L. The agent executes the goal Jr at the top of this stack. Only the GOAL, SUB, DONE actions alter the stack. The GOAL action replaces the top goal description with d++1 the new description given by the assistant. The SUB action pushes stack. The DONE action pops the top (sub)task from the stack.
subtask (gc-1, d(+1) the
(t.pop())push(g push d(41)) Gr.puh((g+ d1++)
if a GOAL, if SUB, a = DONE, otherwise
G.uupate(a))
G..pop() Ge
The SUB action is not available to the agent when the current stack contains L elements, guarantee- ing that goal stack always has at most L elements. The goal-stack transition function Ta is defined as To(Gu+1 G, = 1 [( = (ā.) where 1 {-) is indicator function.
4.3 TRANSITION OF CURRENT OPERATION STATE AND ITS DESCRIPTION
To complete the definition of the state-transine function T, we define the transition function T,- This function factored into two terms by the chain rule:
(s db+1 Sts di, āz) P(s+41 st, a,, P(d 1141, df,
(3)
Only taking the Do action may change the current operation state
[T (st+1 8t, a P(st+1 St, a.) = { [1{s441 = =t}
if DO, otherwise
(4)
The description de may vary when the agent moves to new operation state (by taking the Do action) or requests a new description of s, (by taking the CUR action)
p(dī-1 S441) (dj st41, de)
if ā Do. if a. CUR, otherwise
P(di+1 d1,
(5)
1 1(di+1 = di}
4.4 COST FUNCTION
The interaction policy needs to balance between two types of cost: the cost of taking actions (acting cost) and the cost of not completing task (task error). The acting cost also subsumes the cost of communicating with the assistant because, in reality, such interactions consume time, human effort, and possibly trust. Assuming that the assistant is helpful, acting cost and task error usually conflict with each other; for example, the agent may lower its task error if it is willing to suffer a larger acting cost by increasing the number of requests the assistant.
We employ a simplified model where all types of cost are non-negative real numbers the same unit. Making request of type is assigned constant cost %a The cost of taking the Do action c(s,, ofn) the cost of executing the at action in the operation environment. Calling DONE to terminate execution of the main goal 9i incurs task error c(st, Adone). We exclude the task emors
5

---

Under review as a conference paper at ICLR 2022
of executing subgoals because the interaction policy is only evaluated on reaching the main goal. The magnitudes of the costs naturally specify a trade-off between acting cost and task error. For example, setting the task errors much larger than the other costs indicates that completing tasks is prioritized over taking few actions.
5 LEARNING WHEN AND WHAT TO ASK IN HUMAN-ASSISTED NAVIGATION
Problem. We apply HARI to modeling a human-assisted navigation (HAN) problem. In HAN, human requests an agent to find an object in an indoor environment. Each task request asks the agent to go to a room of type and find an object of type o (e.g., find a mug in kitchen). The agent is equipped with a camera and shares its camera view with the human. We assume that the human is sufficiently familiar with the environment that they can recognize the agent's location by looking at its current view. Before issuing a task request, the human imagines a goal location (but do not reveal it to the agent). We are primarily interested in evaluating success in goal-finding, i.c. whether the agent can arrive at the human's 's intended goal location. Even though there could be multiple locations that match a request, the agent only succeeds it arrives exactly at the chosen goal location. We also determine success in request-fuffilling where the agent successfully fulfills request it navigates to any node that is within two meters of an object that matches the request.
While agent is performing task, it may request the human to provide additional information via telecommunication (e.g.. a chat app). Specifically, it can ask for a description of its current location (CUR), the goal location (GOAL), subgoal location that is on the path from its current location to the goal location (SUB). Detail about how the subgoals are determined is in the Appendix.
Operation Environment. We construct the operation environments using the environment graphs provided by the Matterport3D simulator (Anderson al. 2018). Each environment graph is gener- ated from a 3D model of house where each node is location in the house and each edge connects two nearby unobstructed locations. Each operation state s corresponds to a node in the graph. At any time, the agent's 's operation action space A consists of traversing to any of the nodes that are adjacent to its current node.
We employ a discrete bag-of-features representation for state descriptions A bag of features rep- resents the information that the agent extracts from the raw input that the agent perceives (e.g., an image, language sentence). Working with this intermediate input allows us to easily vary the type and amount of information given to the agent. Specifically, we simulate two settings of descriptions: dense and sparse. At evaluation time, the agent perceives sparse descriptions and request the assis- tant for dense descriptions. A dense description of a current location contains the room name at the location, and the features of M objects restricted be within ò meters of the location. The features of each object consists of (i) its name, (ii) horizontal and vertical angles (relative to the current view- point), and (iii) distance (in meters) from the object the current location. A dense description of goal follows the same representation scheme. In the sparse setting, the current-location description does not include the room name. Moreover, we remove the features of objects that are not the top 100 most frequent objects, emulating an imperfect object detector module. The sparse goal descrip- tion (the task request) has only features of the target object and the room name where the object located at. Especially, if subgoal location is adjacent or coincides with the agent's 's current location, instead of describing room and object features, the human specifies the ground-trutt truth action to go to the subgoal (an action is specified by its horizontal and vertical angles, and travel distance)
Experimental Procedure. We conduct our experiments in three phases. In the pre-training phase, we learn an operation policy â with dense descriptions of the current location and the goal. In the training phase, the agent perceives a sparse description of its current location and is given sparse initial goal description. We use advantage actor-critic (Mnih et al.] 2016) to learn an interaction pol- icy the that controls how the agent communicates with the human and navigates in an environment.
While representation of state descriptions simplifies the object/room detection problem for the agent, does necessarily make the navigation problem easier than with image input, as images may contain information that is not captured by our representation (e.g., object shapes and colors, visualization of paths). Here, we emulate practical scenario where if a destination is visible in the current view, save effort, human would concisely tell agent what to do rather than giving verbose description of the destination.

---

Under review as a conference paper at ICLR 2022
Table 1: Main results on test sets. For success rate, we report both goal-finding (normal font) and request-fulfilling results (smaller grey font in parentheses). We also report the average number of different types of actions taken by the agent (across all task types).
Success Rate % 1
Avg. number of actions
Unseen Start
Unseen Object
Unseen Environment
Agent
CUR GOAL SUB
Do
No assistant and interaction policy the
(d": surrent state description, d": goal description)
Sparse d' and d
43.4 (50.4)
16.4 (23.2)
3.0 (6.8)
13.1
Sparse d", dense d
67.2 (68.4)
56.6 (58.2)
9.7 (12.3)
12.6
Dense d", sparse d
77.9 (86.0)
30.6 (40.3)
4.1 (7.5)
12.0
Dense d" and d
97.8 (98.3)
81.7 (83.3)
9.4 (11.9)
11.0
With assistant and interaction policy the
Rulc-based the (baseline)
78.8 (78.8)
68.5 (68.5)
12.7 (12.7)
2.0
1.0
1.7
11.3
RL-learned the (ours)
85.8 (86.8)
78.2 (79.6)
19.8 (22.5)
2.1
1.0
1.7
11.1
Perfect on sub-goals (skyline)
94.3 (95.8)
95.1 (96.1)
92.6 (94.3)
0.0
0.0
6.3
7.3
The human always returns dense descriptions. The interaction policy is trained in environments that are previously seen as well as unseen during pre- pre-training Finally, in the evaluation phase, the interaction policy is tested three conditions: from a new room (UNSEENSTR), seen environment but new target object type (UNSEENOB)), and new environment (UNSFENENV). The pre-trained operation policy è is fixed during the training and evaluation phases. We create 82, 104 examples for pre-training, 133 for training, and approx- imately 2,000 for each validation or test set. Details about the training procedure and the dataset are included in the Appendix.
environment and target object type but starting
RESULTS AND ANALYSES
Settings. In our main experiments, we set: the cost of taking a CUR, GOAL, SUB, or Do action be 0.01 (we will consider other settings subsequently), the cost of calling DONE to terminate the main goal (i.e. task error) equal the (unweighted) length of the shortest-path from the agent's location to the goal, and the goal stack's size (L) be 2. We compare our RL-learned interaction policy with rule-based baseline that first takes the GOAL action and then randomly selects actions. In each episode, we enforce that the rule-based policy can take most [Xa] y actions of type a,
where y ~ Demoulli((X [) and X, is a constant. We tune each X. on the validation sets so that the rule-based policy has the same average count of each action as the RL-learned policy. To prevent early termination, we enforce that the mueebbaed policy cannot take more DONE actions than SUB actions unless when its SUB action's 's budget is exhausted. We also construct skyline where the interaction policy is also learned by RL but with operation policy that executes subgoals perfectly. As discussed in $5] we are primarily interested in goal-finding success rate and will refer to this metric briefly as success rate.
Main Results Tabll 1). To inspect the potential benefits of asking for additional information, we compute how much the operation policy स improves when it is supplied with dense information about the current and/or goal states. As seen, success rate of the operation policy is lifted dramat- ically when both the current-state and goal descriptions are dense (~>2x increase on UNSEENSTR, 55 on UNSEENOBI, and ~3x on UNSEENENV). We find that dense information about the cur- rent state is more helpful on UNSEENSTR TR, while dense information about the goal is more valuable on UNSEENOBJ and UNSEENENV. This is reasonable because on UNSEENSTR, the agent has been trained to find similar goals. In contrast, the initial goal descriptions in UNSEENOBJ and UNSEE- NENV are completely new to the agent, thus gathering more information about them is necessary.
Aided by our RL-learned interaction policy, the agent observes substantial ~2x increase in suc- cess rate on UNSEENSTR, ~5x on UNSEENOBJ, and ~7x on UNSEENENV, compared to when performing tasks using only the operation policy. In unseen environments, with its capability of requesting subgoals, the agent impressively doubles the success rate of the operation policy that has access to dense descriptions. The RI-learned interaction policy is significantly more effective than
7

---

Under review as a conference paper at ICLR 2022
CCR
sim
Action
Fradton ephode
(a) How froquently does the it- leraction policy execute difler- ent actions, on average acrows different types tasks? Sub- gualls are requested much more unseen
(b) Over the course of a trajec- lory, how does the frequency of different types of actions change unseen environments? Sub- guals are requested in the mid- goal information at the ginning.
Effect of
Effect of cost action counts.
rate.
Figure 3: Analyzing the effect of simultane- ously varying the cost of the CUR, GOAL, SUB, Do actions (on validation environ- ments), thus trading off success rate versus number of actions taken.
Figure 2: Analyzing the behavior of the RL- learned interaction policy (on validation en- vironments).
the rule-based baseline (+7.1% % on UNSEENENV). Compared to policy that calls GOAL at the beginning and calls CUR at every step (which is equivalent to the dense-d"-and-d9 baseline), our policy achieves higher success rate on UNSEENENV while making two times fewer requests. This is due to the ability to request subgoals.
On UNSEENSTR and UNSEENOBJ, the RL-learned policy has not closed the gap with the operation policy that performs tasks with dense descriptions. Our investigation finds that limited information often causes the policy not request information about the current state (i.e. taking CUR) and terminate prematurely or go to a wrong place. Encoding uncertainty in the current-state description Finkel et al., (2006); Nguyen& & O' O'Connor (2015)) is a plausible future direction for tackling this issue. Finally, results obtained by replacing the learned operation policy with one that behaves optimally on subgoals shows that further improving performance of the operation policy on short- distance goals would effectively enhance the agent's 's performance on long- -ist-ncetncee goals.
Behavior of the RL.-Lerneed Interaction Policy. Figure 2a characterizes behaviors of the RL- leamned interaction policy in three evaluation conditions. We expect that tasks in UNSEENSTR are the easiest and those in UNSEENENV are the hardest. As the difficulty of the evaluation condition increases, the interaction policy issues more CUR, SUB, and Do actions. The average number of GOAL actions does not vary, showing that the interaction policy has correctly learned that making more than one goal-clarifying request is unnecessary. Figure 2b) illustrates the distribution of each action along the length of an episode in the validation UNSEENENV dataset. The GOAL action, if taken, is always taken only once and immediately in the first step. The number of CUR actions gradually decreases over time. The agent makes most SUB requests in the middle of an episode, after its has attempted but failed accomplish the main goals. We observe similar patterns on the other two validation sets.
Effects of Varying Action Cost. As mentioned, we assign the same cost to each CUR, GOAL, SUB, Do action. Figure 3al demonstrates the effects of changing this cost on the success rate of the agent. Setting the cost equal to 0.5 makes it too costly to take any action, inducing a policy that always calls DONE in the first step and thus fails all tasks. Overall, the success rate of the agent rises as we reduce the action cost. The increase in success rate is most visible in UNSEENENV and least visible in UNSEENSTR. Figure 3b provides more insights. As the action cost decreases, we observe a growth in the number of SUB and Do actions taken by the interaction policy. Meanwhile, the numbers of CUR and GOAL actions are mostly static. Since requesting subgoals is more helpful in unseen environments than in seen environments, the increase in the number of SUB actions leads the more visible boost in success rate on UNSEENENV tasks.
Performing Tasks with Deeper Goal Stacks. In Table 2, we test the functionality of our frame- work with a stack size 3, allowing the agent to request subgoals of subgoals. As expected, success rate UNSEENENV is boosted significantly (+11.9% compared to using a stack of size 2). Success rate on UNSEENOBJ is largely unchanged; we find that the agent makes more SUB requests (aver- agely 4.5 requests per episode compared to 1.0 request made when the stack size is 2), but doing
8

---

Under review as a conference paper at ICLR 2022
Success rates and numbers of actions taken with different stack sizes (on validation). Larger stack sizes significantly aid success rates in unseen environments, but not in seen environments.
Table 2:
Goal Goal-finding success rate (%)+
Average number of actions !
Unseen Start
Unseen Object
Unseen Environment
Stack size
CUR
GOAL
SUB
DO
1 (no subgouls)
92.2
78.4
12.5
5.1
1.9
0.0
10.7
2
86.9
77.6
21.6
2.1
1.0
1.7
11.2
3
83.2
78.6
33.5
1.3
1.0
5.0
8.2
so does not further enhance performance. The agent makes less CUR requests, possibly in order to offsee the cost of making more SUB requests. Due to this behavior, success rate on UNSEENSTR declines with larger stack sizes, as information about the current state is more valuable for these tasks than subgoals. These results show that the critic model overestimates the V values in states where SUB actions are taken, leading to the agent learning to request subgoals more than needed.
7 RELATED WORK AND CONCLUSION
Transfer Learning in Reinforcement Learning. Various frameworks have been proposed to model knowledge transfer from a more capable agent novice one (Da Silva Costa| 2019). Torrey & Taylor (013) introduce the action-advising framework where learner strategically re- strate- gies for deciding when to request in this framework. In agent-to-agent setting, (Da Silva et al.) 2017 Zimmer et al. 2014; Omidshafiei et al., 2019) focus on learning a teaching policy in addition to advice-requesting policy. An important assumption in these papers is that the teacher must share a common action space with the learner. More recent frameworks (Kim et al.] 2019 Nguyen al. 2019; Nguyen & Daumé III, 2019) relax this assumption by allowing the teacher to specify high-level subgoals instead of low-level actions. HARI can be viewed as a strict extension of these frameworks. It allows the human to specify not only subgoals, but also any additional information about the current state and the goal that agent can interpret. Moreover, HARI equips the agent with select the most useful intention convey in
quests reference actions from a teacher. [a Silva et al.] (2020) investigate uncertainty-
multiple communication intentions and teaches given situation. Another line of work employs standard RL communication protocol, where the hu- man can only transfer knowledge through numerical scores or categorical feedback (Knox & Stone, 2009 Judah et al.] 22100) Peng et al. 2016)] Griffith et al.) 2013). Maclin & Shavlik (1996) propose a framework where the human advises the agent using a domain-speciff language, specifying rules that can be incorporated into the agent's model. In contrast, HARI operates with a black-box agent model. SSumers et al. (2020) extract features from various types of language feedback to construct a reward function for reinforcement learning. We instead focus on melloyymen-timee communication and directly incorporate the human feedback as input to agent's operation policy.
Task-Oriented Dialog and Generating Natural Language Questions. HARI models a task- oriented dialog problem. Many variants of this problem requires the agent compose specific questions (De Vries et al. 2017 Das et al., 2017] [Toomsoo et al.) 2020). The dominant approach in these problems is to mimic pre-collected human utterances. As discussed previously, naively mir- roring human external behavior cannot enable agents to understand the limits of their knowledge. We teach the agent to understand its intrinsic needs through interaction with the human and the en- vironment rather than through imitation of human behaviors. Another related line of work concerns generating natural language explanations of model decisions (Camburu et al. 2018 Hendricks al., 2016 Rajani et al. 2019).
In summary, this paper presents a general POMDP framework for modeling human-agent commu- nication. While we demonstrate this framework on simplified navigation problem, our framework can theoretically capture richer types of human-agent communication. Hence, an important empir- ical question is how well our formulation generalizes to richer environments with more complex interactions and state spaces (Shridhar et al. 2020). Enhancing the sample efficiency of the learning policy by exploiting the hierarchical policy structure is an exciting future direction. Furthermore, techniques for generating faithful explanations (Kumar & Talukdar 2020: Madsen et al. 202T can be applied to enhance the specificity of the generated questions.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 3674-3683, 2018.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. In Proceedings of Advances in Neural Information Processing Systems, 2018.
Felipe Leno Da Silva and Anna Helena Reali Costa. A survey transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research, 64:645-703, 2019.
Felipe Leno Da Silva, Ruben Glatt, and Anna Helena Reali Costa. Simultaneously learning and advising in multiagent reinforcement learning. In Proceedings of the 16th conference tonomous agents and multiagent systems, pp. 1100-1108, 2017.
Felipe Leno Da Silva, Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. Uncertainty- aware action advising for deep reinforcement learning agents. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 34, pp. 5792-5799, 2020.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 326-335, 2017.
Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. Guesswhat?! visual object discovery through multi-modal dialogue. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5503-5512, 2017.
Jenny Rose Finkel, Christopher D Manning, and Andrew Ng. Solving the problem cascading errors: Approximate bayesian inference for linguistic annotation pipelines. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pp.
2006.
Herbert P Grice. Logic and conversation. In Speech acts pp. 41-58. Brill, 1975.
Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. Georgia Institute of Technology, 2013.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposiun series, 2015.
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating visual explanations. In European conference on computer vision. pp. 3-19. Springer, 2016.
Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. A recurrent aaboo-aad-language bert for navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Kshitij Judah, Saikat Roy, Alan Fern, and Thomas Dietterich. Reinforcement learning via practice and critique advice. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, 2010.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-34, 1998.
Dong-Ki Ki Kim, Miao Liu, Shayegan Omidshafici. Sebastian Lopez-Cot, Matthew Riemer, Golnaz Habibi, Gerald Tesauro, Sami Mourad, Murray Campbell, and Jonathan P How. Learning hierar- chical teaching policies for cooperative agents. arXiv preprint arXivv/903.03:
2019.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, pp. 9-16, 2009.
10

---

Under review as a conference paper at ICLR 2022
Sawan Kumar and Partha Talukdar. : language inference with faithful natural language explanations. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 3730-2742, Online, July 2020. Association for Computational Linguisties. doi: 10. 10.18665/11/2200.cch haap://000..mmmmm.777777077 URL.Bttps://accaaahholoog org. acl mocl-main... 771
Richard Maclin and Jude w Shavlik. Creating advice-taking reinforcement learners. Machine Learning, 22(1):251-281, 1996.
Andreas Madsen, Nicholas Meade, Vaibhav Adlakha, and Siva Reddy. Evaluating the faithfulness of importance measures in nip by recursively masking allegedly important tokens and retraining. arXiv preprint arXiv:2110.08411 2021.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuogiu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. In leml, volume 99, pp. 278-2877 1999.
Khanh Nguyen and Hal Daumé III. Help, anna! visual navigation with natural multimodal assistance via retrospective crriossty-encouraginn imitation learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), November 2019. URL https:
//arxiv. org/ abs/ 1909. 01871
Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing models. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1587-1598, Lishon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 111188/11/D111111110
(ttt/hhtts:///w// www.w///bb...................... orgg org/
aathology/D055-1138
Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In The IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), June 2019. hRlph:ts:://art
orrg/bbs/18/ 11112.011111.11
Shayegan Omidshafiei, Dong-Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher Amato, Murray Campbell, and Jonathan P How. Learning teach in cooperative multiagent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pp. 3128-6136 2019.
Bei Peng, James MacGlashan, Robert Loftin, Michael I. Littman, David Roberts, and Matthew Taylor. A need for speed: Adapting agent action speed to improve task learning from non-expert humans. In Proceedings of the International Joint Conference on Autonomous Agents and Multi- agent Systems, 2016.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv: 110..:6966...61,, 2019.
Sudha Rao and Hal Daumé III. Learning to ask good questions: Ranking clarification questions us- ing neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 2737-2746, Melbourne, Australia. July 2018 Association for Computational Linguistics. doi: 10.18653/v1/P18. 1255.
URL https::/aclanthoott
2wgg/P8--io 1255
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statistics, pp. 635. JMLR Workshop and Conference Proceedings, 2011.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020.
11

---

Under review as a conference paper at ICLR 2022
Theodore R Sumers, Mark K Ho, Robert D Hawkins, Karthik Narasimhan, and Thomas L Griffiths. Learning rewards from linguistic feedback. arXiv preprint arXiv:2009.1715, 2020.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navi- gation. In Conference on Robot Learning, pp. 394-406. PMLR, 2020.
Lisa Torrey and Matthew Taylor. Teaching on a budget: Agents advising agents in reinforcement learning. In Proceedings of the 2013 international conference on Autonomous agents and multi- agent systems, pp. 1053-1060, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Matthieu Zimmer, Paolo Viappiani, and Paul Weng. Teacher-student framework: reinforcement learning approach. In AAMAS Workshop Autonomous Robots and Multirobot Systems, 2014.
12

---

Under review as a conference paper at ICLR 2022
A
TRAINING PROCEDURE
Cost function.
The cost function our framework given as follows
c(st, ai
if a = Do if ā E [CUR, GOAL, SUB}. if a, DONE, |G if a DONE, |Gy| 1,
d(tt, Gt, as)
Ya.
(6)
c(st, Odamee
Subgoals. Let pt be the shortest path from the agent's current state 8, the current goal Jt, and Pt.: be the i-th node on the path (0 < < lP.). The subgoal location is chosen as Pt, where k min [1pp/1//110 maxx, where max a pre-defined constant.
Training Algorithms. We pre-train the operation policy म with DAgger (Ross et al. 2011), mini- mizing the cross entropy between its action distribution with that of shortest-path oracle (which is a one-hot distribution with all probability concentrated on the optimal action).
We use advantage actor-critic (Mnih al. 2016) train the interaction policy ve. This method simultancously estimates an actor policy wia B + A(A) and a critic function V6 B + R. Given an execution ($1, 01, G1 ... ,H), , the gradients with respect the actor and critic are
Ve CaCoeto
(Vo(B)) C₁) V log ( bg)
t=1
H
@ frrii
Σ
(Vo(b,) - Ct) VoVa(())
(8)
1=1
where C. = EØ Cy. by is belief state that summarizes the partial execution Ti.t for the actor, and oy is a belief state for the critic.
Cost function. The cost function introduced in 4.4 is not effective for learning the interaction policy because the task error is given only at the end of an episode. We extend the reward-shaping method proposed by Ng et al. (1999) to goal-conditioned policies, augmenting the original cost function with a shaping function ((s.g) g) with s.g E S. We set ((s,g) to be the (unweighted) shortest-path distance from
g. The cost received by the agent at time step & is & à + @ (8t-1.9(+1) ((s:. gt). We assume that the agent transitions to a special terminal state 9tem E S and remains there after it terminates execution of the main goal. We set (Stemm None) = O, where 9i = None signals that the episode has ended. Hence, the cumulative cost of an execution under the new cost function is
H 2a=
H
H
c. + (86411.944)) (et.ge))
c
(1).
Since (()), 91) does not depend on the action taken in s1, minimizing the new cumulative cost does not change the optimal policy for the task (81,91).
Model Architecture. We adapt the V&I. BERT architecture (Hong et al. 2020) for modeling the operation policy स. Our model has two components: encoder and a decoder; both are imple- mented as Transformer models (Vaswani et al.) 2017). The encoder takes as input a description di or dy and generates sequence of hidden vectors. In every step, the decoder takes input the previous hidden vector b1-1. the sequence of vectors representing d', and the sequence of vectors representing dV. It then performs self-attention on these vectors to compute the current hidden vector bj and probability distribution over navigation actions Pt-
The interaction policy th (the actor) an LSTM-based recurrent neural network. The input of this model is the operation policy's model outputs, b; and Pt, and the embedding of the previously taken action ū--1- The critic model also has a similar architecture but outputs a real number (the V value) rather than action distribution. When training the interaction policy, always fix the parameters of the operation policy. We find it necessary to pre-train the critic before training jointly with the actor.
13

---

Under review as a conference paper at ICLR 2022
Table 3: Dataset statistics.
Split
Number examples
Pre-training
82,104
Pre-training validation
3,000
Training
65,133
Validation UNSEENSTR
1,901
Validation UNSEENOBJ
1,912
Validation UNSEENENY
1,967
Test UNSEENSTR
1,653
Test UNSEENOBJ
1,913
Test UNSEENENV
1,777
Representation of State Descriptions. The representation each object, room, or action is com- puted as follows. Let frume, fhhrr, f flist, and fore are the features of an object f. consisting of its name, horizontal angle, vertical angle, distance, and type (a type is either Object, Room, or Action; in this case, the type is jeject). For simplicity, we discretize real-valued features, result- ing horizontal angles (corresponding to π 7/6- k; 0 S k < 12), 3 vertical angles (corresponding to π /76-A. k -1 < k < 1), and 5 distance values (we round down real-valued distance to the nearest integer). We then lookup the embedding of each feature from an embedding table and sum all the embeddings into single vector that represents the corresponding object. For room, fluez, fvent fui are zeroes. For an action, fnume is either ActionStop for the stop action Adone or ActionGo otherwise.
During pre- vre-training, we randomly drop features in dł and de so that the operation policy is familiar with making decisions under sparse information. Concretely, room or action as a feature set. For dÿ. let M be the number objects in a description. We uniformly randomly keep feature sets among the M 1 feature sets of dī (the plus one is the room's 's feature set), where m ~ Uniform(ming
refer to all features of an object,
M + 1), M + 1).
adjacent equals 81, uniformly randomly alternate between giving a dense and sparse description. In this case, the sparse description contains the features of the target object and the goal room's 's name. Otherwise, with probability of 1/, we give (b) a (sparse) description that contains the target object's 's features and the goal room's name, or (c) (sparse) description that describes the next ground-truth action.
For di, we have two cases. gi
either (a) dense description
We pre-train the operation policy on various path lengths (ranging from to 10 graph nodes) so that learns to accomplish both long-distance main goals and short-distance subgoals.
Data. Table summarizes the data splits. From total of 72 environments provided by the Matter- port3D dataset, we use 36 environments for pre- pre-training, 18 as unseen environments for training, for validation UNSEENENV, and 11 for test UNSERNENV. We vocabulary of size 1738, which includes object and room names, and special tokens representing the distance and direction values. The length of navigation path ranges from 10 graph nodes.
Hyperparameters.
See Table 4]
14

---

Under review as a conference paper at ICLR 2022
Table 4: Hyperparameters.
Hyperparameter Name
Value
Environment
Max. subgoal distance (1mm))
3 nodes
Max. stack size (T.)
Max. object distance for
5 meters meters
Max. object distance for de
Max. number of objects (Mmx))
20
Cost of taking each CUR, GOAL. SUB, Do action
0.01
Operation policy
Hidden size
256
Number of hidden layers
Attention dropout probability
0.1
Hidden dropout probability
0.1
Number of attention heads Optimizer
Adam
Learning rate
10-4
Batch size
32
Number of training iterations Max. number of time steps (H)
10a
Interaction policy
Hidden size
Number of hidden layers
Entropy regularization weight Optimizer
0,001
Adam
Learning rate
10-5
Batch size
32
Number of critic pre- traininn iterations
x 10
Number of training iterations
x 10a
Max. number of time steps (H)
30
Max. number of time steps for executing subgoal
3 shortest distancc to the subgoal
15


Reviews:
[{"review_id": "ywodw790ukN", "reviewer": "Reviewer_LvSE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors present a \"transfer learning\" (TL) framework in which the agent is able to \"ask for help\" as one of the action options. While the authors propose an \"unusual\" modeling centered around POMDPs and robotic navigation (which could be good, bridging the gap between general TL for RL methods and this particular application), there has been a large amount of works with the same high-level idea in the last years. Yet, most of the related works are simply neglected and not even mentioned in the manuscript. \n\nThe survey below summarizes dozens of related works that follow this same general idea, and not even the Survey itself is cited:\n\nSilva, Felipe Leno, and Anna Helena Reali Costa. \"A survey on transfer learning for multiagent reinforcement learning systems.\" Journal of Artificial Intelligence Research 64 (2019): 645-703.\n\nIn particular, works from the \"action advising\" area are very relevant to the present manuscript. The seminal Teacher-Student framework [1] introduced the idea of proactively \"asking for help\" when needed in the RL context. The RCMP advising method [2], based on Adhoc Advising [3], propose a more modern takeaway on how to compute the agent \"uncertainty\" (very relevant to the manuscript at hand). Works that focus on the specific points claimed as contributions in the manuscript also exist, such as considering as a MDP the problem of learning \"when to ask for help\" [4, 5], or adding a hierarchical component to task selection before the low-level resolution of the task [6].\n\n[1] Torrey, Lisa, and Matthew Taylor. \"Teaching on a budget: Agents advising agents in reinforcement learning.\" Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems. 2013.\n\n[2] Silva, Felipe Leno, et al. \"Uncertainty-aware action advising for deep reinforcement learning agents.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.\n\n[3] Silva, Felipe Leno, Ruben Glatt, and Anna Helena Reali Costa. \"Simultaneously learning and advising in multiagent reinforcement learning.\" Proceedings of the 16th conference on autonomous agents and multiagent systems. 2017.\n\n[4] Zimmer, Matthieu, Paolo Viappiani, and Paul Weng. \"Teacher-student framework: a reinforcement learning approach.\" AAMAS Workshop Autonomous Robots and Multirobot Systems. 2014.\n\n[5] Omidshafiei, Shayegan, et al. \"Learning to teach in cooperative multiagent reinforcement learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n\n[6] Kim, Dong-Ki, et al. \"Learning hierarchical teaching in cooperative multiagent reinforcement learning.\" (2019).\n\nI would expect that several of those related works were included in the empirical evaluation. Yet, none of them are even mentioned in the paper! At the very least the distinction between the contribution of the present manuscript and those related works should be discussed in deep, as well as an explanation of why those methods were not added to the empirical evaluation.\n\nDue to the aforementioned major omission, I cannot judge the contribution of the paper.\n\n\n\n-------------------\nAfter Rebuttal\n-------------------\n\nAlthough I agree the method has its merits and contribution, the manuscript does a poor job in situating the proposed work in the current state-of-the-art literature. Most of the discussion period was devoted to explaining where the method differs from current literature, which should have been clear since the beginning by reading the paper. I am also not entirely convinced the method is that different from the action advising framework. Sure, the present manuscript chooses to extract more information from the advice by enforcing the advisor to be a human (more restrictions = more information to be extract from the advice at the cost of less flexibility), while most of those methods are very general and focus on information that exist in any RL environment (i.e., states and actions). Yet, the same general idea holds, and the present method is not the first one that extracted from advice more than action suggestions. Those similarities should be embraced and throughfully explained in the manuscript, as in my view the proposed method is a specialized advisor-advisee framework for this specific situation that matters most to the authors. Therefore, my grades are maintained so that the authors can improve the manuscript for the next submission.", "summary_of_the_paper": "The paper presents a reinforcement learning framework where the agent is able to \"ask for help\" to receive from a human additional information to more easily solve the task. The paper is kinda Robotic Navigation-oriented by the way tasks and \"additional information\" about the environment are described, as well as in the empirical evaluation. It's uncertain if the paper presents any significant contribution due to major omissions on the related works list (explanation in the main review)", "summary_of_the_review": "Paper investigates an interesting and relevant topic. However, there is a major omission of related works. Therefore, my recommendation is to reject the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "I1kRdLoAbOf", "reviewer": "Reviewer_9mjj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am not convinced that this paper is ready for ICLR, it looks like this project is in an early stage with writing often unclear, results are presented without confidence intervals, in tables rather than figures. I am also not convinced of the strength of the contributions, even if the presentation was improved. The authors do a good job implementing existing algorithms in the context of the studied task, however the applicability of this framework seems to be very narrow. While reading this paper I kept wondering why is this a valuable research question, however I do not have a good answer.\n\nA detailed list of my comments is below: \n\n1. The writing is often unclear -- for example:\n\n - Page 1: \"On every step, the agent can ask for more information about its current state, the goal state, or a subgoal state...\" \"The agent learns when to convey an intent to the assistant...\" \nthis is confusing... so, does the agent convey intent, or ask questions? But the assistant is supposed to be omniscient, and always aware of the agent's intent?\n\n - Page 2:  states that agent has only a partial description of a goal state. So, does the uncertainty in this task reduce to uncertainty about the goal state? Toward the end of the paper I discovered that the uncertainty actually  comes from missing environment annotations. This needs to be clarified.\n\n- Page 2: \"The agent achieves this level of performance while issuing only 4.8 requests to the assistant on average, representing less than 1⁄4 of the total number of actions taken in a task execution.\" Why is this significant or relevant? The number of times the agent has to make requests is determined by how much information is missing in the evaluation task, compared to the task it was trained on -- which means that the exact numbers are arbitrary. Such references to arbitrary numbers happen in a few places through the paper.\n\n2. Related work is very brief, and given in the end of the paper -- instead of at the beginning to motivate the current work. This makes it hard to evaluate the novelty of the contribution. \nThe goal stack maintenance section could be moved to the supplement to make space for motivation of this work.\n\n3. It appears that there are multiple goals at the same time in the goal stack - if so, how does the agent arbitrate between multiple goals? I expected that there should be only one goal, which can have subgoals, but this is not clear in the text.\n\n4. A computational experiment is used to show that the ability to request information was useful to the agent. However, it seems obvious that an agent with less uncertainty will do better. Section 6 describes the model comparison as comparing the agent that requests assistance to an agent that takes random actions. However, with missing annotations a rational agent that can not request information should still be able to rationally search the graph until the object is found - which seems a more reasonable  performance benchmark.  Alternatively, the authors could compare the current framework to a state-of-the-art agent that can solve the same benchmark problem. Was the indoor object-finding task used in previous work, or as part of benchmark problem sets? \n\n\n5. Practical applications of this framework seem to be very limited, to a context of running RL agents with a near-perfect incomplete policy to evaluation tasks with missing annotations. \n\n6. What is the point of using the Matterport3D dataset, as a basis for environment graphs? This 3D nature of this dataset, or that it is made to resemble a living space, is not in any way leveraged by the current algorithm.\n\n7. The data presented in tables would be better readable as a plot.\n", "summary_of_the_paper": "This paper formulates a hierarchical RL agent that learns to request information from human assistant, when task-critical information is missing. The assistant is assumed to be omniscent, present at all times, know the agent’s current state, know the agents goal state, and be able to accurately provide the requested information. On each step the agent has 5 types of actions: it can either issue one of the three information queries (request current state description, goal state description, subgoal description), or execute the highest value action according to its beliefs, or terminate execution. Such a framework can be applicable to situations where an RL agent is trained on a certain task, but deployed on a similar task with some information missing (for example, a missing object annotation). \nThe framework is demonstrated on simulated human-assisted object finding task.  The paper shows that an agent equipped with a human assistant achieves an improved rate of successfully completing the task, compared to an unassisted agent. \n\n", "summary_of_the_review": "I am not convinced that this paper is ready for ICLR, it looks like the project is in an early stage. The writing is unclear, results are presented in tables rather than figures -- and are mostly descriptions of model performance rather than model comparisons. I am also not convinced of the strength of the contributions, even if the presentation is improved. The authors do a good job implementing existing algorithms in the context of the given task, however the applicability of this framework is very narrow, to a context of running RL agents with a near-perfect incomplete policy to evaluation tasks with missing annotations. The paper claims that the agent that can request assistance from a human to reduce uncertainty performs better on the given task, than an agent that can not request assistance -- this claim is correct, however it also seems obvious.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "35r6n1vFIJw", "reviewer": "Reviewer_rgjt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**:\n* Writing and Presentation: The paper is well structured. All the graphs were well annotated.\n* Modelling: The POMDP designed to obtain the interaction policy was succinct and rich enough to represent the assistive task. It presents a framework that can be extended further/utilized in several other multi-agent problems and opens up avenues for future research.\n* Goal Stack and using multiple Sub-goals: The idea of introducing a goal stack and including sub-goals as a possible action for the interaction policy was very interesting. The generalization capabilities of the method on unseen environments also look promising.\n* Limitations of the work are well addressed.\n\n**Cons**:\n* Experimental Evaluation: The experiments done on the Matterport3D simulator under three different conditions was exhaustive. However, it would have helped to know a few more details on the SUB action: how was the subgoal given to the agent during training -- particularly when the subgoal location was not adjacent /coincides with the agent's current location.  I did not fully understand this action as presented by the human during training. Experimental details were missing and given that the problem is heavily dependent on modeling the POMDP, it would have helped to know the details of the setup elaborately.\n\n* Baselines: The rule-based baseline seemed to be particularly weak in the context of this work. Perhaps a slightly stronger baseline -- including some more domain knowledge could have led to interesting ablation studies.\n\n* Typos in Notation: Typo on Page 4, where the subgoal function was first introduced -- the symbol used was \\sigma_{A}.\n\n\n**Additional Comments**:\n\nI find the premise of the work very interesting and was curious to know if there were any thoughts on how the presented algorithm/model would perform (robustness of the technique to noise) in case there was noise induced in the feedback from the assistant -- like the assistant presented a noisy estimate of the current state description or a possible sub-goal?\n\n\n**General Comments**:\nOverall I liked the approach of learning interaction policies using the hierarchical RL and think that it presents a good starting point for future research in the direction of learning policies under assistance. The success rate on unseen tasks guided by sub-goals is particularly promising. There could have been some more clarity in the experimental setup (for training), and additional baselines used for evaluation. \n\n*Originality*:  Moderate\n\n*Clarity*: Moderate\n\n*Quality*: Good\n\n*Significance*: Moderate to High", "summary_of_the_paper": "The paper operates in a 2-agent setting, where one agent is a learning agent and the other agent is the assistant. The objective is for the learning agent to seek help from the assistant while solving a task, and the decision of what to ask the assistant and when to do so is made by learning an interaction policy using hierarchical reinforcement learning -- in particular using a variant of POMDPs. The assumption is that the learning agent's operational policy on the task environment is not changed and an additional interaction (hierarchical) policy is learned on top of the operational policy which dictates when and what to ask the assistant. The authors have empirically evaluated their approach on a (simulated) human-assisted navigation task, and show that the proposed method achieves close to seven times an increase in the success rate of solving the task compared to the policy obtained by learning without assistance. Additionally, the results also show that the frequency of seeking assistance was only 25 percent of all actions executed to solve the task. ", "summary_of_the_review": "The paper presents a good starting point for future research in the area and a model for application to several real-world problems such as assisted teaching (when to provide students help and how, for instance). However, the experimental section could be made clearer since the results presented look promising. An additional stronger baseline (though I understand this might be non-trivial to formulate) would have made the claims stand out. At the moment, both the baselines are weak (no assistance/rule-based assistance). Hence. I am inclined to reject the paper at the moment.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is on the theme of active reinforcement learning with a human/assistant in the loop. Under partial observability, an agent acts as per an interaction policy that gathers state/goal information from the assistant, while an operational policy assumed pre-learnt in this paper executes low-level actions.  The reviews acknowledge the relevance of this topic and that the paper is well structured and coherently presented overall. However, there are unanimous concerns around experimental evaluation being unconvincing, lack of strong baselines and lack of thorough coverage of related work precluding an accurate assessment of claimed contributions. As such, the paper is not in a form that can be accepted at ICLR --  the authors are encouraged to revise their submission as per review feedback.
Decision: Reject