Year: 2021

Abstract:
Reliable AI agents should be mindful of the limits of their knowledge and consult humans when sensing that they do not have sufficient knowledge to make sound decisions. We formulate a hierarchical reinforcement learning framework for learning to decide when to request additional information from humans and what type of information would be helpful to request. Our framework extends partially-observed Markov decision processes (POMDPs) by allowing an agent to interact with an assistant to leverage their knowledge in accomplishing tasks. Results on a simulated human-assisted navigation problem demonstrate the effectiveness of our framework: aided with an interaction policy learned by our method, a navigation policy achieves up to a 7× improvement in task success rate compared to performing tasks only by itself. We find that the ability to request subgoals enables the agent to generalize effectively to tasks in unseen environments. We analyze benefits and challenges of learning with a hierarchical policy structure and suggest directions for future work.

heading: 1 INTRODUCTION
text: Human-agent communication at deployment time has been under-explored in machine learning, where the traditional focus has been on building agents that can accomplish tasks on their own (full autonomy). Nevertheless, enabling an agent to exchange information with humans during its operation can potentially enhance its helpfulness and trustworthiness. The ability to request and interpret human advice would help the agent accomplish tasks beyond its built-in knowledge, while the ability to accurately convey when and why it is about to fail would make the agent safer to use.
In his classical work, Grice (1975) outlines the desired characteristics of cooperative communication, commonly known as the Gricean maxims of cooperation. Among these characteristics are informativeness (the maxim of quantity) and faithfulness (the maxim of quality). Human-agent communication in current work has fallen short in these two aspects. Traditional frameworks like imitation learning and reinforcement learning employ limited communication protocols where the agent and the human can only exchange simple intentions (requesting low-level actions or rewards Torrey & Taylor (2013); Knox & Stone (2009)). More powerful frameworks like (Nguyen & Daumé III, 2019; Nguyen et al., 2019; Kim et al., 2019) allow the agent to process high-level instructions from humans, but the agent still only requests generic help. Recent work in natural language processing endows the agent with the ability to generate rich natural language utterances (Camburu et al., 2018; Rao & Daumé III, 2018; De Vries et al., 2017; Das et al., 2017), but the communication is not faithful in the sense that the agent only mirrors human-generated utterances without grounding its communication in self-perception of its (in)capabilities and (un)certainties. Essentially it learns to convey what a human may be concerned about, not what it is concerned about.
This paper presents a hierarchical reinforcement learning framework named HARI (Human-Assisted Reinforced Interaction), which supports richer and more faithful human-agent communication. Our framework allows the agent to learn to convey intrinstic needs for specific information and to incorporate diverse types of information from humans to make better decisions. Specifically, the agent in HARI is equipped with three information-seeking intentions: in every step, it can choose to request more information about (i) its current state, (ii) the goal state, or (iii) a subgoal state which, if reached, helps it make progress on the current task. Upon receiving a request, the human can transfer new information to the agent by giving new descriptions of the requested state. These de-
scriptions will be incorporated as new inputs to the agent’s decision-making policy. The human thus can transfer any form of information that can be interpreted by the policy (e.g., asking the agent to execute skills that it has learned, giving side information that connects the agent to a situation it is more familiar with). Because the agent’s policy can implement a variety of model architectures and learning algorithms, our framework opens up many possibilities for human-agent communication.
To enable faithful communication, we teach the agent to understand its intrinsic needs by interacting with the human and the environment (rather than imitating human behaviors). By requesting different types of information and observing how much each type of information enhances its decisions, the agent gradually learns to determine which information is most useful to obtain in a given situation. With this capability, at deployment time, it can choose when and what information to ask from the human to improve its task performance. To demonstrate the effectiveness of HARI, we simulate a human-assisted navigation problem where an agent has access to only sparse information about its current state and the goal, and can request additional information about these states. On tasks that take place in previously unseen environments, the ability to ask for help improves the agent’s success rate by 7× higher compared to performing tasks only by itself. This human-assisted agent even outperforms an agent that always has access to dense information in unseen environments, thanks to the ability to request subgoals. We show that performance of the agent can be further improved by recursively asking for subgoals of subgoals. We discuss limitations of the policy’s model and feature representation, which suggest room for future improvements.

heading: 2 MOTIVATION: LIMITATIONS OF THE STANDARD POMDP FRAMEWORK
text: We consider an environment defined by a partially-observed Markov decision process (POMDP) E = (S,A, T, c,D, ρ) with state space S, action space A, state-transition function T : S × A → ∆(S), cost function c : S ×A → R, description space D, and description function ρ : S → ∆(D).1 Here, ∆(Y) denotes the set of all probability distributions over a set Y . We refer to this environment as the operation environment because it is where the agent operates to accomplish tasks.
Each task in the environment is defined as a tuple (s1, g1, d g 1) where s1 is the start state, g1 is the goal state, and dg1 is a limited description of g1. Initially, a task (s1, g1, d g 1) is sampled from a task
1We use the term “description” in lieu of “observation” in the POMDP formulation to emphasize two properties of the information the agent has access to for making decisions: (i) the information can be in various modalities and (ii) the information can be obtained via not only perception, but also communication.
distribution T. An agent starts in s1 and is only given the goal description d g 1. It has to reach the goal state g1 within H time steps. Let gt and d g t be the goal state and goal description being executed at time step t, respectively. In a standard POMDP, gt = g1 and d g t = d g 1 for 1 ≤ t ≤ H . But later, we will enable the agent to set new goals via communication with humans.
At any time step t, the agent does not know its true state st but only receives a description dst ∼ ρ(st) of the state. Generally, the description can include any information coming from any knowledge source (e.g., an RGB image and/or a verbal description describing the current view). Given dst and dgt , the agent then makes a decision at ∈ A, transitions to the next state st+1 ∼ T (st, at), and receives a cost ct , c(st, at). A special action adone ∈ A is taken when the agent decides to terminate its execution. The goal of the agent is to reach g1 with minimum cumulative cost C(τ) = ∑H t=1 ct, where τ = (s1, d s 1, a1, s2, d s 2, a2, . . . , sH , d s H) is an execution of the task.
As the agent does not have access to its true state, it can only make decisions based on the (observable) partial execution τ1:t = (ds1, a1, . . . , d s t ). Kaelbling et al. (1998) introduce the notion of a belief state b ∈ ∆(S), which sufficiently summarizes a partial execution as a distribution over the state space S. In practice, when S is continuous or high-dimensional, representing and updating a full belief state (whose dimension is |S|) is intractable. We follow Hausknecht & Stone (2015), using recurrent neural networks to learn compact representation of partial executions. We denote by bst a representation of the partial execution τ1:t and by B the set of all possible representations. The agent maintains an operation policy π̂ : B × D → ∆(A) that maps a belief state bs and a goal description dg to a distribution over A. The learning objective for solving a standard POMDP is to estimate an operation policy that minimizes the expected cumulative cost of performing tasks:
min π E(s1,g1,dg1)∼T,τ∼Pπ(·|s1,dg1) [C(τ)] (1)
where Pπ(· | s1, dg1) is the distribution over executions generated by a policy π given start state s1 and goal description d g 1. In a standard POMDP, an agent performs tasks by executing its own operation policy without asking for any external assistance. Moreover, the description function ρ and the goal description dg1 are assumed to be fixed during a task execution. As seen from Equation 1, given a fixed environment and task distribution, the expected performance of the agent is solely determined by the operation policy π̂. Thus, the standard POMDP framework does not provide any mechanism for improving the agent’s performance other than enhancing the operation policy.

heading: 3 LEVERAGING HUMAN KNOWLEDGE TO BETTER ACCOMPLISH TASKS
text: We introduce an assistant into the operation environment, who can provide information about the environment’s states. We assume the agent possesses a pre-learned operation policy π̂. This policy serves as the common ground between the agent and the assistant, which is a prerequisite for communication between them to occur. For example, this policy represents a set of basic tasks that agent has mastered and the assistant can ask the agent to perform. In general, the more knowledge encoded in this policy, the more effectively the agent can communicate with and leverage help from the assistant. Our goal is to learn an interaction policy ψθ (parametrized by θ) that controls how the agent communicates with the assistant to gather additional information. The operation policy π̂ will be invoked by the interaction policy if the latter decides that the agent does not need new information and wants to take an operating action.
The assistant aids the agent by giving new (current or goal) state descriptions, connecting the agent to situations on which it can make better decisions. Consider an object-finding navigation problem, where a robot has been trained to reliably navigate to the kitchen from the living room of a house. Suppose the robot is then asked to “find a mug”, an object that it has never heard of. The assistant can help the robot accomplish this task by giving a more informative goal description “find a mug in the kitchen”, relating the current task to the kitchen-finding task that the robot has been familiar with. The robot may also have problems with localization: it knows how to get the kitchen from the living room but it may not realize that it is currently the living room. In this case, giving a currentstate description that specifies this information provides the robot with a useful hint on what actions to take next.
Our framework allows the assistant to convey any form of information that the agent can incorporate into its input. As discussed in §2, the notion of “state description” in our framework is general,
capturing various types of information, including but not limited to visual perception and verbal description. Communication between the agent and the assistant can be flexibly enriched by designing the agent’s operation policy to be able to consume the forms of information of interest (e.g., a policy that takes natural language as input).
Communication with the Assistant. The assistant is present all the time and knows the agent’s current state st and the goal state gt. It is specified by two functions: a description function ρA : S ×D → ∆(D) and a subgoal function ωA : S ×S → ∆(S). ρA(d′ | s, d) specifies the probability of giving d′ as the new description of state s given a current description d. ωA(g′ | s, g) indicates the probability of proposing g′ as a subgoal given a current state s and a goal state g.
At time step t, the assistant accepts three types of request from the agent:
(a) CUR: requests a new description of st and receives dst+1 ∼ ρA (· | st, dst ); (b) GOAL: requests a new description of gt and receives d g t+1 ∼ ρA (· | gt, d g t ); (c) SUB: requests a description of a subgoal gt+1 and receives d g t+1 ∼ ρA (· | gt+1, ∅) where
gt+1 ∼ ωA (· | st, gt) and ∅ is an empty description.
Interaction Policy. The action space of the interaction policy ψθ consists of five actions: {CUR,GOAL, SUB,DO,DONE}. The first three actions correspond to making the three types of request that the assistants accepts. The remaining two actions are used to traverse in the operation environment:
(d) DO: executes the action adot , arg maxa∈A π̂ (a | bst , d g t ). The agent transitions to a new
operation state st+1 ∼ T (st, adot ); (e) DONE: determines that the current goal gt has been reached.2 If gt is a main goal (gt = g1),
the task episode ends. If gt is a subgoal (gt 6= g1), the agent may choose a new goal to follow. Our problem formulation leaves it open on what goal should be selected next.
By selecting among these actions, the interaction policy essentially decides when to ask the assistant for additional information, and what types of information to ask for. Our formulation does not specify the input space of the interaction policy, as this space depends on how the agent implements its goal memory (i.e. how it stores and retrieves the subgoals). In the next section, we introduce an instantiation where the agent uses a stack data structure to manage (sub)goals.

heading: 4 HIERARCHICAL REINFORCEMENT LEARNING FRAMEWORK
text: In this section, we describe the HARI framework. We first formulate the POMDP environment that the interaction policy acts in, referred to as the interaction environment (§ 4.1). Our construction employs a goal stack to manage multiple levels of (sub)goals (§4.2). A goal stack stores all the tasks the agent has been assigned but has not yet decided to terminate (by choosing the DONE action). It is updated in every step depending on the taken action. We design a cost function (§ 4.4) that specifies a trade-off between the cost of taking actions (acting cost) and the cost of not completing a task (task error).

heading: 4.1 INTERACTION ENVIRONMENT
text: Given an operation environment E = (S,A, T, c,D, ρ), the interaction environment constructed on top of E is a POMDP Ē = (S̄, Ā, T̄ , c̄, D̄, ρ̄) with:
• State space S̄ = S × D × GL where GL is the set of all goal stacks containing at most L elements (L is a hyperparameter). Each state s̄ = (s, ds, G) ∈ S̄ is a tuple of an operation state s ∈ S, its description ds ∈ D, and a goal stack G ∈ GL. Each element in the goal stack G is a tuple (g, dg) of a goal state g ∈ S and its description dg ∈ D; • Action space Ā = {CUR,GOAL, SUB,DO,DONE}; • State-transition function T̄ = Ts · TG where Ts : S × D × Ā → ∆(S × D) and TG : GL × Ā → ∆(GL);
• Cost function c̄ : (S×GL)×Ā → R (defined in §4.4 to trade off acting cost and task error); 2Note that the agent may falsely decide that a goal has been reached.
• Description space D̄ = D×GdL where GdL is the set of all goal-description stacks of size L. At any time, the agent cannot access the environment’s goal stack G, which contains true goal states. Instead, it can only observe the descriptions in G. We call this partial stack a goal-description stack, denoted by Gd; • Description function ρ̄ : S̄ → D̄, where ρ̄(s̄) = ρ̄(s, ds, G) = (ds, Gd). Unlike in the standard POMDP formulation, this description function is deterministic.
A belief state b̄t summarizes a partial execution (s̄1, ā1, · · · , s̄t). We formally define the interaction policy as ψθ : B̄ → ∆(A), where B̄ is the set of all interaction belief states.

heading: 4.2 GOAL STACK
text: A goal stack is an ordered set of tasks that the agent has not declared completion (by calling DONE). The initial stack G1 = {(g1, dg1)} contains the main goal g1, and its description d g 1. Let Gt be the goal stack at time step t. The agent executes the goal gt at the top of this stack. Only the GOAL, SUB, DONE actions alter the stack. The GOAL action replaces the top goal description with dgt+1, the new description given by the assistant. The SUB action pushes a new subtask (gt+1, d g t+1) to the stack. The DONE action pops the top (sub)task from the stack.
Gt.update(a) =  Gt.pop().push(gt, d g t+1) if a = GOAL, Gt.push(gt+1, d g t+1) if a = SUB,
Gt.pop() if a = DONE, Gt otherwise
(2)
The SUB action is not available to the agent when the current stack contains L elements, guaranteeing that goal stack always has at most L elements. The goal-stack transition function TG is defined as TG(Gt+1 | Gt, āt) = 1{Gt+1 = Gt.update(āt)} where 1{.} is an indicator function.

heading: 4.3 TRANSITION OF THE CURRENT OPERATION STATE AND ITS DESCRIPTION
text: To complete the definition of the state-transition function T̄ , we define the transition function Ts. This function is factored into two terms by the chain rule:
Ts(st+1, d s t+1 | st, dst , āt) = P (st+1 | st, āt) · P (dst+1 | st+1, dst , āt) (3)
Only taking the DO action may change the current operation state P (st+1 | st, āt) = { T ( st+1 | st, adot ) if āt = DO,
1{st+1 = st} otherwise (4)
The description dst may vary when the agent moves to a new operation state (by taking the DO action) or requests a new description of st (by taking the CUR action)
P (dst+1 | dst , st+1, āt) =  ρ(dst+1 | st+1) if āt = DO, ρA(d s t+1 | st+1, dst ) if āt = CUR,
1{dst+1 = dst} otherwise (5)

heading: 4.4 COST FUNCTION
text: The interaction policy needs to balance between two types of cost: the cost of taking actions (acting cost) and the cost of not completing a task (task error). The acting cost also subsumes the cost of communicating with the assistant because, in reality, such interactions consume time, human effort, and possibly trust. Assuming that the assistant is helpful, acting cost and task error usually conflict with each other; for example, the agent may lower its task error if it is willing to suffer a larger acting cost by increasing the number of requests to the assistant.
We employ a simplified model where all types of cost are non-negative real numbers of the same unit. Making a request of type a is assigned a constant cost γa. The cost of taking the DO action is c(st, adot ), the cost of executing the a do t action in the operation environment. Calling DONE to terminate execution of the main goal g1 incurs a task error c(st, adone). We exclude the task errors
of executing subgoals because the interaction policy is only evaluated on reaching the main goal. The magnitudes of the costs naturally specify a trade-off between acting cost and task error. For example, setting the task errors much larger than the other costs indicates that completing tasks is prioritized over taking few actions.

heading: 5 LEARNING WHEN AND WHAT TO ASK IN HUMAN-ASSISTED NAVIGATION
text: Problem. We apply HARI to modeling a human-assisted navigation (HAN) problem. In HAN, a human requests an agent to find an object in an indoor environment. Each task request asks the agent to go to a room of type r and find an object of type o (e.g., find a mug in a kitchen). The agent is equipped with a camera and shares its camera view with the human. We assume that the human is sufficiently familiar with the environment that they can recognize the agent’s location by looking at its current view. Before issuing a task request, the human imagines a goal location (but do not reveal it to the agent). We are primarily interested in evaluating success in goal-finding, i.e. whether the agent can arrive at the human’s intended goal location. Even though there could be multiple locations that match a request, the agent only succeeds if it arrives exactly at the chosen goal location. We also determine success in request-fulfilling, where the agent successfully fulfills a request if it navigates to any node that is within two meters of an object that matches the request.
While an agent is performing a task, it may request the human to provide additional information via telecommunication (e.g., a chat app). Specifically, it can ask for a description of its current location (CUR), the goal location (GOAL), or a subgoal location that is on the path from its current location to the goal location (SUB). Detail about how the subgoals are determined is in the Appendix.
Operation Environment. We construct the operation environments using the environment graphs provided by the Matterport3D simulator (Anderson et al., 2018). Each environment graph is generated from a 3D model of a house where each node is a location in the house and each edge connects two nearby unobstructed locations. Each operation state s corresponds to a node in the graph. At any time, the agent’s operation action space A consists of traversing to any of the nodes that are adjacent to its current node.
We employ a discrete bag-of-features representation for state descriptions.3 A bag of features represents the information that the agent extracts from the raw input that the agent perceives (e.g., an image, a language sentence). Working with this intermediate input allows us to easily vary the type and amount of information given to the agent. Specifically, we simulate two settings of descriptions: dense and sparse. At evaluation time, the agent perceives sparse descriptions and request the assistant for dense descriptions. A dense description of a current location contains the room name at the location, and the features of M objects restricted to be within δ meters of the location. The features of each object consists of (i) its name, (ii) horizontal and vertical angles (relative to the current viewpoint), and (iii) distance (in meters) from the object to the current location. A dense description of a goal follows the same representation scheme. In the sparse setting, the current-location description does not include the room name. Moreover, we remove the features of objects that are not in the top 100 most frequent objects, emulating an imperfect object detector module. The sparse goal description (the task request) has only features of the target object and the room name where the object is located at. Especially, if a subgoal location is adjacent or coincides with the agent’s current location, instead of describing room and object features, the human specifies the ground-truth action to go to the subgoal (an action is specified by its horizontal and vertical angles, and travel distance).4
Experimental Procedure. We conduct our experiments in three phases. In the pre-training phase, we learn an operation policy π̂ with dense descriptions of the current location and the goal. In the training phase, the agent perceives a sparse description of its current location and is given a sparse initial goal description. We use advantage actor-critic (Mnih et al., 2016) to learn an interaction policy ψθ that controls how the agent communicates with the human and navigates in an environment.
3While our representation of state descriptions simplifies the object/room detection problem for the agent, it does not necessarily make the navigation problem easier than with image input, as images may contain information that is not captured by our representation (e.g., object shapes and colors, visualization of paths).
4Here, we emulate a practical scenario where if a destination is visible in the current view, to save effort, a human would concisely tell an agent what to do rather than giving a verbose description of the destination.
The human always returns dense descriptions. The interaction policy is trained in environments that are previously seen as well as unseen during pre-training. Finally, in the evaluation phase, the interaction policy is tested on three conditions: seen environment and target object type but starting from a new room (UNSEENSTR), seen environment but new target object type (UNSEENOBJ), and new environment (UNSEENENV). The pre-trained operation policy π̂ is fixed during the training and evaluation phases. We create 82,104 examples for pre-training, 65,133 for training, and approximately 2,000 for each validation or test set. Details about the training procedure and the dataset are included in the Appendix.

heading: 6 RESULTS AND ANALYSES
text: Settings. In our main experiments, we set: the cost of taking a CUR, GOAL, SUB, or DO action to be 0.01 (we will consider other settings subsequently), the cost of calling DONE to terminate the main goal (i.e. task error) equal the (unweighted) length of the shortest-path from the agent’s location to the goal, and the goal stack’s size (L) to be 2. We compare our RL-learned interaction policy with a rule-based baseline that first takes the GOAL action and then randomly selects actions. In each episode, we enforce that the rule-based policy can take at most bXac+ y actions of type a, where y ∼ Bernoulli(Xa − bXac) and Xa is a constant. We tune each Xa on the validation sets so that the rule-based policy has the same average count of each action as the RL-learned policy. To prevent early termination, we enforce that the rule-based policy cannot take more DONE actions than SUB actions unless when its SUB action’s budget is exhausted. We also construct a skyline where the interaction policy is also learned by RL but with an operation policy that executes subgoals perfectly. As discussed in §5, we are primarily interested in goal-finding success rate and will refer to this metric briefly as success rate.
Main Results (Table 1). To inspect the potential benefits of asking for additional information, we compute how much the operation policy π̂ improves when it is supplied with dense information about the current and/or goal states. As seen, success rate of the operation policy is lifted dramatically when both the current-state and goal descriptions are dense (∼2× increase on UNSEENSTR, ∼5× on UNSEENOBJ, and ∼3× on UNSEENENV). We find that dense information about the current state is more helpful on UNSEENSTR, while dense information about the goal is more valuable on UNSEENOBJ and UNSEENENV. This is reasonable because on UNSEENSTR, the agent has been trained to find similar goals. In contrast, the initial goal descriptions in UNSEENOBJ and UNSEENENV are completely new to the agent, thus gathering more information about them is necessary.
Aided by our RL-learned interaction policy, the agent observes a substantial ∼2× increase in success rate on UNSEENSTR, ∼5× on UNSEENOBJ, and ∼7× on UNSEENENV, compared to when performing tasks using only the operation policy. In unseen environments, with its capability of requesting subgoals, the agent impressively doubles the success rate of the operation policy that has access to dense descriptions. The RL-learned interaction policy is significantly more effective than
0
5
10
CUR DO GOAL SUB Action type
A ve
ra ge
a ct
io ns
unseen_str unseen_obj unseen_env
(a) How frequently does the interaction policy execute different actions, on average across different types of tasks? Subgoals are requested much more in unseen environments.
●
●
●
● ●
0.0
0.5
1.0
1.5
2.0
0.2 0.4 0.6 0.8 1.0 Fraction of episode
A ve
ra ge
a ct
io ns
● CUR GOAL SUB
(b) Over the course of a trajectory, how does the frequency of different types of actions change in unseen environments? Subgoals are requested in the middle, goal information at the beginning.
Figure 2: Analyzing the behavior of the RLlearned interaction policy (on validation environments).
●
● ● ● ● ● ● ●
●
0
25
50
75
−3−2−1 log10(cost)
S uc
ce ss
r at
e (%
)
● unseen_str unseen_obj unseen_env
(a) Effect of cost on success rate.
●
● ● ● ● ● ● ● ●
0
3
6
9
−3−2−1 log10(cost)
A ve
ra ge
a ct
io ns
● CUR GOAL SUB DO
(b) Effect of cost on action counts.
Figure 3: Analyzing the effect of simultaneously varying the cost of the CUR, GOAL, SUB, DO actions (on validation environments), thus trading off success rate versus number of actions taken.
the rule-based baseline (+7.1% on UNSEENENV). Compared to a policy that calls GOAL at the beginning and calls CUR at every step (which is equivalent to the dense-ds-and-dg baseline), our policy achieves higher success rate on UNSEENENV while making two times fewer requests. This is due to the ability to request subgoals.
On UNSEENSTR and UNSEENOBJ, the RL-learned policy has not closed the gap with the operation policy that performs tasks with dense descriptions. Our investigation finds that limited information often causes the policy to not request information about the current state (i.e. taking CUR) and terminate prematurely or go to a wrong place. Encoding uncertainty in the current-state description (e.g., Finkel et al. (2006); Nguyen & O’Connor (2015)) is a plausible future direction for tackling this issue. Finally, results obtained by replacing the learned operation policy with one that behaves optimally on subgoals shows that further improving performance of the operation policy on shortdistance goals would effectively enhance the agent’s performance on long-distance goals.
Behavior of the RL-Learned Interaction Policy. Figure 2a characterizes behaviors of the RLlearned interaction policy in three evaluation conditions. We expect that tasks in UNSEENSTR are the easiest and those in UNSEENENV are the hardest. As the difficulty of the evaluation condition increases, the interaction policy issues more CUR, SUB, and DO actions. The average number of GOAL actions does not vary, showing that the interaction policy has correctly learned that making more than one goal-clarifying request is unnecessary. Figure 2b illustrates the distribution of each action along the length of an episode in the validation UNSEENENV dataset. The GOAL action, if taken, is always taken only once and immediately in the first step. The number of CUR actions gradually decreases over time. The agent makes most SUB requests in the middle of an episode, after its has attempted but failed to accomplish the main goals. We observe similar patterns on the other two validation sets.
Effects of Varying Action Cost. As mentioned, we assign the same cost to each CUR, GOAL, SUB, or DO action. Figure 3a demonstrates the effects of changing this cost on the success rate of the agent. Setting the cost equal to 0.5 makes it too costly to take any action, inducing a policy that always calls DONE in the first step and thus fails on all tasks. Overall, the success rate of the agent rises as we reduce the action cost. The increase in success rate is most visible in UNSEENENV and least visible in UNSEENSTR. Figure 3b provides more insights. As the action cost decreases, we observe a growth in the number of SUB and DO actions taken by the interaction policy. Meanwhile, the numbers of CUR and GOAL actions are mostly static. Since requesting subgoals is more helpful in unseen environments than in seen environments, the increase in the number of SUB actions leads the more visible boost in success rate on UNSEENENV tasks.
Performing Tasks with Deeper Goal Stacks. In Table 2, we test the functionality of our framework with a stack size 3, allowing the agent to request subgoals of subgoals. As expected, success rate on UNSEENENV is boosted significantly (+11.9% compared to using a stack of size 2). Success rate on UNSEENOBJ is largely unchanged; we find that the agent makes more SUB requests (averagely 4.5 requests per episode compared to 1.0 request made when the stack size is 2), but doing
so does not further enhance performance. The agent makes less CUR requests, possibly in order to offset the cost of making more SUB requests. Due to this behavior, success rate on UNSEENSTR declines with larger stack sizes, as information about the current state is more valuable for these tasks than subgoals. These results show that the critic model overestimates the V values in states where SUB actions are taken, leading to the agent learning to request subgoals more than needed.

heading: 7 RELATED WORK AND CONCLUSION
text: Transfer Learning in Reinforcement Learning. Various frameworks have been proposed to model knowledge transfer from a more capable agent to a novice one (Da Silva & Costa, 2019). Torrey & Taylor (2013) introduce the action-advising framework where a learner strategically requests reference actions from a teacher. Da Silva et al. (2020) investigate uncertainty-based strategies for deciding when to request in this framework. In an agent-to-agent setting, (Da Silva et al., 2017; Zimmer et al., 2014; Omidshafiei et al., 2019) focus on learning a teaching policy in addition to an advice-requesting policy. An important assumption in these papers is that the teacher must share a common action space with the learner. More recent frameworks (Kim et al., 2019; Nguyen et al., 2019; Nguyen & Daumé III, 2019) relax this assumption by allowing the teacher to specify high-level subgoals instead of low-level actions. HARI can be viewed as a strict extension of these frameworks. It allows the human to specify not only subgoals, but also any additional information about the current state and the goal that agent can interpret. Moreover, HARI equips the agent with multiple communication intentions and teaches it to select the most useful intention to convey in a given situation. Another line of work employs standard RL communication protocol, where the human can only transfer knowledge through numerical scores or categorical feedback (Knox & Stone, 2009; Judah et al., 2010; Peng et al., 2016; Griffith et al., 2013). Maclin & Shavlik (1996) propose a framework where the human advises the agent using a domain-specific language, specifying rules that can be incorporated into the agent’s model. In contrast, HARI operates with a black-box agent model. Sumers et al. (2020) extract features from various types of language feedback to construct a reward function for reinforcement learning. We instead focus on deployment-time communication and directly incorporate the human feedback as input to agent’s operation policy.
Task-Oriented Dialog and Generating Natural Language Questions. HARI models a taskoriented dialog problem. Many variants of this problem requires the agent to compose specific questions (De Vries et al., 2017; Das et al., 2017; Thomason et al., 2020). The dominant approach in these problems is to mimic pre-collected human utterances. As discussed previously, naively mirroring human external behavior cannot enable agents to understand the limits of their knowledge. We teach the agent to understand its intrinsic needs through interaction with the human and the environment rather than through imitation of human behaviors. Another related line of work concerns generating natural language explanations of model decisions (Camburu et al., 2018; Hendricks et al., 2016; Rajani et al., 2019).
In summary, this paper presents a general POMDP framework for modeling human-agent communication. While we demonstrate this framework on a simplified navigation problem, our framework can theoretically capture richer types of human-agent communication. Hence, an important empirical question is how well our formulation generalizes to richer environments with more complex interactions and state spaces (Shridhar et al., 2020). Enhancing the sample efficiency of the learning policy by exploiting the hierarchical policy structure is an exciting future direction. Furthermore, techniques for generating faithful explanations (Kumar & Talukdar, 2020; Madsen et al., 2021) can be applied to enhance the specificity of the generated questions.

heading: A TRAINING PROCEDURE
text: Cost function. The cost function in our framework is given as follows
c̄(st, Gt, āt) =  c(st, a do t ) if āt = DO
γāt if āt ∈ {CUR,GOAL, SUB}, c(st, adone) if āt = DONE, |Gt| = 1 0 if āt = DONE, |Gt| > 1,
(6)
Subgoals. Let pt be the shortest path from the agent’s current state st to the current goal gt, and pt,i be the i-th node on the path (0 ≤ i < |pt|). The subgoal location is chosen as pt,k where k = min(b|p|/2c, lmax), where lmax is a pre-defined constant.
Training Algorithms. We pre-train the operation policy π̂ with DAgger (Ross et al., 2011), minimizing the cross entropy between its action distribution with that of a shortest-path oracle (which is a one-hot distribution with all probability concentrated on the optimal action).
We use advantage actor-critic (Mnih et al., 2016) to train the interaction policy ψθ. This method simultaneously estimates an actor policy ψθ : B̄ → ∆(Ā) and a critic function Vφ : B̄ → R. Given an execution τ̄ = (s̄1, ā1, c̄1 · · · , s̄H), the gradients with respect to the actor and critic are
∇θLactor = H∑ t=1 ( Vφ(b̄ v t )− Ct ) ∇θ logψθ(āt | b̄at ) (7)
∇φLcritic = H∑ t=1 ( Vφ(b̄ v t )− Ct ) ∇φVφ(b̄vt ) (8)
where Ct = ∑H j=t cj , b̄ a t is a belief state that summarizes the partial execution τ̄1:t for the actor, and b̄vt is a belief state for the critic.
Cost function. The cost function introduced in § 4.4 is not effective for learning the interaction policy because the task error is given only at the end of an episode. We extend the reward-shaping method proposed by Ng et al. (1999) to goal-conditioned policies, augmenting the original cost function with a shaping function Φ(s, g) with s, g ∈ S. We set Φ(s, g) to be the (unweighted) shortest-path distance from s to g. The cost received by the agent at time step t is c̃t , c̄t + Φ(st+1, gt+1)−Φ(st, gt). We assume that the agent transitions to a special terminal state sterm ∈ S and remains there after it terminates execution of the main goal. We set Φ(sterm, None) = 0, where gt = None signals that the episode has ended. Hence, the cumulative cost of an execution under the new cost function is
H∑ t=1 c̃t = H∑ t=1 c̄t + Φ(st+1, gt+1)− Φ(st, gt) = H∑ t=1 c̄t − Φ(s1, g1) (9)
Since Φ(s1, g1) does not depend on the action taken in s1, minimizing the new cumulative cost does not change the optimal policy for the task (s1, g1).
Model Architecture. We adapt the V&L BERT architecture (Hong et al., 2020) for modeling the operation policy π̂. Our model has two components: an encoder and a decoder; both are implemented as Transformer models (Vaswani et al., 2017). The encoder takes as input a description dst or d g t and generates a sequence of hidden vectors. In every step, the decoder takes as input the previous hidden vector bst−1, the sequence of vectors representing d s t , and the sequence of vectors representing dgt . It then performs self-attention on these vectors to compute the current hidden vector bst and a probability distribution over navigation actions pt.
The interaction policy ψθ (the actor) is an LSTM-based recurrent neural network. The input of this model is the operation policy’s model outputs, bst and pt, and the embedding of the previously taken action āt−1. The critic model also has a similar architecture but outputs a real number (the V value) rather than an action distribution. When training the interaction policy, we always fix the parameters of the operation policy. We find it necessary to pre-train the critic before training it jointly with the actor.
Representation of State Descriptions. The representation of each object, room, or action is computed as follows. Let f name, f horz, f vert, f dist, and f type are the features of an object f , consisting of its name, horizontal angle, vertical angle, distance, and type (a type is either Object, Room, or Action; in this case, the type is Object). For simplicity, we discretize real-valued features, resulting in 12 horizontal angles (corresponding to π/6 · k, 0 ≤ k < 12), 3 vertical angles (corresponding to π/6 · k,−1 ≤ k ≤ 1), and 5 distance values (we round down a real-valued distance to the nearest integer). We then lookup the embedding of each feature from an embedding table and sum all the embeddings into a single vector that represents the corresponding object. For a room, f horz, f vert f dist are zeroes. For an action, f name is either ActionStop for the stop action adone or ActionGo otherwise.
During pre-training, we randomly drop features in dst and d g t so that the operation policy is familiar with making decisions under sparse information. Concretely, we refer to all features of an object, room or action as a feature set. For dst , let M be the number objects in a description. We uniformly randomly keep m feature sets among the M + 1 feature sets of dst (the plus one is the room’s feature set), where m ∼ Uniform(min(5,M + 1),M + 1). For dst , we have two cases. If g1 is not adjacent or equals to s1, we uniformly randomly alternate between giving a dense and a sparse description. In this case, the sparse description contains the features of the target object and the goal room’s name. Otherwise, with a probability of 1⁄3, we give either (a) a dense description (b) a (sparse) description that contains the target object’s features and the goal room’s name, or (c) a (sparse) description that describes the next ground-truth action.
We pre-train the operation policy on various path lengths (ranging from 1 to 10 graph nodes) so that it learns to accomplish both long-distance main goals and short-distance subgoals.
Data. Table 3 summarizes the data splits. From a total of 72 environments provided by the Matterport3D dataset, we use 36 environments for pre-training, 18 as unseen environments for training, 7 for validation UNSEENENV, and 11 for test UNSEENENV. We use a vocabulary of size 1738, which includes object and room names, and special tokens representing the distance and direction values. The length of a navigation path ranges from 5 to 10 graph nodes.
Hyperparameters. See Table 4.

Review summaries:
The paper presents a reinforcement learning framework where the agent is able to "ask for help" to receive from a human additional information to more easily solve the task. The paper is kinda Robotic Navigation-oriented by the way tasks and "additional information" about the environment are described, as well as in the empirical evaluation. It's uncertain if the paper presents any significant contribution due to major omissions on the related works list (explanation in the main review)

The paper operates in a 2-agent setting, where one agent is a learning agent and the other agent is the assistant. The objective is for the learning agent to seek help from the assistant while solving a task, and the decision of what to ask the assistant and when to do so is made by learning an interaction policy using hierarchical reinforcement learning -- in particular using a variant of POMDPs. The assumption is that the learning agent's operational policy on the task environment is not changed and an additional interaction (hierarchical) policy is learned on top of the operational policy which dictates when and what to ask the assistant. The authors have empirically evaluated their approach on a (simulated) human-assisted navigation task, and show that the proposed method achieves close to seven times an increase in the success rate of solving the task compared to the policy obtained by learning without assistance. Additionally, the results also show that the frequency of seeking assistance was only 25 percent of all actions executed to solve the task. 

This paper formulates a hierarchical RL agent that learns to request information from human assistant, when task-critical information is missing. The assistant is assumed to be omniscent, present at all times, know the agent’s current state, know the agents goal state, and be able to accurately provide the requested information. On each step the agent has 5 types of actions: it can either issue one of the three information queries (request current state description, goal state description, subgoal description), or execute the highest value action according to its beliefs, or terminate execution. Such a framework can be applicable to situations where an RL agent is trained on a certain task, but deployed on a similar task with some information missing (for example, a missing object annotation).  The framework is demonstrated on simulated human-assisted object finding task.  The paper shows that an agent equipped with a human assistant achieves an improved rate of successfully completing the task, compared to an unassisted agent.