Learning When and What to Ask: a Hierarchical Reinforcement Learning Framework

1 INTRODUCTION . Human-agent communication at deployment time has been under-explored in machine learning , where the traditional focus has been on building agents that can accomplish tasks on their own ( full autonomy ) . Nevertheless , enabling an agent to exchange information with humans during its operation can potentially enhance its helpfulness and trustworthiness . The ability to request and interpret human advice would help the agent accomplish tasks beyond its built-in knowledge , while the ability to accurately convey when and why it is about to fail would make the agent safer to use . In his classical work , Grice ( 1975 ) outlines the desired characteristics of cooperative communication , commonly known as the Gricean maxims of cooperation . Among these characteristics are informativeness ( the maxim of quantity ) and faithfulness ( the maxim of quality ) . Human-agent communication in current work has fallen short in these two aspects . Traditional frameworks like imitation learning and reinforcement learning employ limited communication protocols where the agent and the human can only exchange simple intentions ( requesting low-level actions or rewards Torrey & Taylor ( 2013 ) ; Knox & Stone ( 2009 ) ) . More powerful frameworks like ( Nguyen & Daumé III , 2019 ; Nguyen et al. , 2019 ; Kim et al. , 2019 ) allow the agent to process high-level instructions from humans , but the agent still only requests generic help . Recent work in natural language processing endows the agent with the ability to generate rich natural language utterances ( Camburu et al. , 2018 ; Rao & Daumé III , 2018 ; De Vries et al. , 2017 ; Das et al. , 2017 ) , but the communication is not faithful in the sense that the agent only mirrors human-generated utterances without grounding its communication in self-perception of its ( in ) capabilities and ( un ) certainties . Essentially it learns to convey what a human may be concerned about , not what it is concerned about . This paper presents a hierarchical reinforcement learning framework named HARI ( Human-Assisted Reinforced Interaction ) , which supports richer and more faithful human-agent communication . Our framework allows the agent to learn to convey intrinstic needs for specific information and to incorporate diverse types of information from humans to make better decisions . Specifically , the agent in HARI is equipped with three information-seeking intentions : in every step , it can choose to request more information about ( i ) its current state , ( ii ) the goal state , or ( iii ) a subgoal state which , if reached , helps it make progress on the current task . Upon receiving a request , the human can transfer new information to the agent by giving new descriptions of the requested state . These de- scriptions will be incorporated as new inputs to the agent ’ s decision-making policy . The human thus can transfer any form of information that can be interpreted by the policy ( e.g. , asking the agent to execute skills that it has learned , giving side information that connects the agent to a situation it is more familiar with ) . Because the agent ’ s policy can implement a variety of model architectures and learning algorithms , our framework opens up many possibilities for human-agent communication . To enable faithful communication , we teach the agent to understand its intrinsic needs by interacting with the human and the environment ( rather than imitating human behaviors ) . By requesting different types of information and observing how much each type of information enhances its decisions , the agent gradually learns to determine which information is most useful to obtain in a given situation . With this capability , at deployment time , it can choose when and what information to ask from the human to improve its task performance . To demonstrate the effectiveness of HARI , we simulate a human-assisted navigation problem where an agent has access to only sparse information about its current state and the goal , and can request additional information about these states . On tasks that take place in previously unseen environments , the ability to ask for help improves the agent ’ s success rate by 7× higher compared to performing tasks only by itself . This human-assisted agent even outperforms an agent that always has access to dense information in unseen environments , thanks to the ability to request subgoals . We show that performance of the agent can be further improved by recursively asking for subgoals of subgoals . We discuss limitations of the policy ’ s model and feature representation , which suggest room for future improvements . 2 MOTIVATION : LIMITATIONS OF THE STANDARD POMDP FRAMEWORK . We consider an environment defined by a partially-observed Markov decision process ( POMDP ) E = ( S , A , T , c , D , ρ ) with state space S , action space A , state-transition function T : S × A → ∆ ( S ) , cost function c : S ×A → R , description space D , and description function ρ : S → ∆ ( D ) .1 Here , ∆ ( Y ) denotes the set of all probability distributions over a set Y . We refer to this environment as the operation environment because it is where the agent operates to accomplish tasks . Each task in the environment is defined as a tuple ( s1 , g1 , d g 1 ) where s1 is the start state , g1 is the goal state , and dg1 is a limited description of g1 . Initially , a task ( s1 , g1 , d g 1 ) is sampled from a task 1We use the term “ description ” in lieu of “ observation ” in the POMDP formulation to emphasize two properties of the information the agent has access to for making decisions : ( i ) the information can be in various modalities and ( ii ) the information can be obtained via not only perception , but also communication . distribution T. An agent starts in s1 and is only given the goal description d g 1 . It has to reach the goal state g1 within H time steps . Let gt and d g t be the goal state and goal description being executed at time step t , respectively . In a standard POMDP , gt = g1 and d g t = d g 1 for 1 ≤ t ≤ H . But later , we will enable the agent to set new goals via communication with humans . At any time step t , the agent does not know its true state st but only receives a description dst ∼ ρ ( st ) of the state . Generally , the description can include any information coming from any knowledge source ( e.g. , an RGB image and/or a verbal description describing the current view ) . Given dst and dgt , the agent then makes a decision at ∈ A , transitions to the next state st+1 ∼ T ( st , at ) , and receives a cost ct , c ( st , at ) . A special action adone ∈ A is taken when the agent decides to terminate its execution . The goal of the agent is to reach g1 with minimum cumulative cost C ( τ ) = ∑H t=1 ct , where τ = ( s1 , d s 1 , a1 , s2 , d s 2 , a2 , . . . , sH , d s H ) is an execution of the task . As the agent does not have access to its true state , it can only make decisions based on the ( observable ) partial execution τ1 : t = ( ds1 , a1 , . . . , d s t ) . Kaelbling et al . ( 1998 ) introduce the notion of a belief state b ∈ ∆ ( S ) , which sufficiently summarizes a partial execution as a distribution over the state space S. In practice , when S is continuous or high-dimensional , representing and updating a full belief state ( whose dimension is |S| ) is intractable . We follow Hausknecht & Stone ( 2015 ) , using recurrent neural networks to learn compact representation of partial executions . We denote by bst a representation of the partial execution τ1 : t and by B the set of all possible representations . The agent maintains an operation policy π̂ : B × D → ∆ ( A ) that maps a belief state bs and a goal description dg to a distribution over A . The learning objective for solving a standard POMDP is to estimate an operation policy that minimizes the expected cumulative cost of performing tasks : min π E ( s1 , g1 , dg1 ) ∼T , τ∼Pπ ( ·|s1 , dg1 ) [ C ( τ ) ] ( 1 ) where Pπ ( · | s1 , dg1 ) is the distribution over executions generated by a policy π given start state s1 and goal description d g 1 . In a standard POMDP , an agent performs tasks by executing its own operation policy without asking for any external assistance . Moreover , the description function ρ and the goal description dg1 are assumed to be fixed during a task execution . As seen from Equation 1 , given a fixed environment and task distribution , the expected performance of the agent is solely determined by the operation policy π̂ . Thus , the standard POMDP framework does not provide any mechanism for improving the agent ’ s performance other than enhancing the operation policy . 3 LEVERAGING HUMAN KNOWLEDGE TO BETTER ACCOMPLISH TASKS . We introduce an assistant into the operation environment , who can provide information about the environment ’ s states . We assume the agent possesses a pre-learned operation policy π̂ . This policy serves as the common ground between the agent and the assistant , which is a prerequisite for communication between them to occur . For example , this policy represents a set of basic tasks that agent has mastered and the assistant can ask the agent to perform . In general , the more knowledge encoded in this policy , the more effectively the agent can communicate with and leverage help from the assistant . Our goal is to learn an interaction policy ψθ ( parametrized by θ ) that controls how the agent communicates with the assistant to gather additional information . The operation policy π̂ will be invoked by the interaction policy if the latter decides that the agent does not need new information and wants to take an operating action . The assistant aids the agent by giving new ( current or goal ) state descriptions , connecting the agent to situations on which it can make better decisions . Consider an object-finding navigation problem , where a robot has been trained to reliably navigate to the kitchen from the living room of a house . Suppose the robot is then asked to “ find a mug ” , an object that it has never heard of . The assistant can help the robot accomplish this task by giving a more informative goal description “ find a mug in the kitchen ” , relating the current task to the kitchen-finding task that the robot has been familiar with . The robot may also have problems with localization : it knows how to get the kitchen from the living room but it may not realize that it is currently the living room . In this case , giving a currentstate description that specifies this information provides the robot with a useful hint on what actions to take next . Our framework allows the assistant to convey any form of information that the agent can incorporate into its input . As discussed in §2 , the notion of “ state description ” in our framework is general , capturing various types of information , including but not limited to visual perception and verbal description . Communication between the agent and the assistant can be flexibly enriched by designing the agent ’ s operation policy to be able to consume the forms of information of interest ( e.g. , a policy that takes natural language as input ) . Communication with the Assistant . The assistant is present all the time and knows the agent ’ s current state st and the goal state gt . It is specified by two functions : a description function ρA : S ×D → ∆ ( D ) and a subgoal function ωA : S ×S → ∆ ( S ) . ρA ( d′ | s , d ) specifies the probability of giving d′ as the new description of state s given a current description d. ωA ( g′ | s , g ) indicates the probability of proposing g′ as a subgoal given a current state s and a goal state g. At time step t , the assistant accepts three types of request from the agent : ( a ) CUR : requests a new description of st and receives dst+1 ∼ ρA ( · | st , dst ) ; ( b ) GOAL : requests a new description of gt and receives d g t+1 ∼ ρA ( · | gt , d g t ) ; ( c ) SUB : requests a description of a subgoal gt+1 and receives d g t+1 ∼ ρA ( · | gt+1 , ∅ ) where gt+1 ∼ ωA ( · | st , gt ) and ∅ is an empty description . Interaction Policy . The action space of the interaction policy ψθ consists of five actions : { CUR , GOAL , SUB , DO , DONE } . The first three actions correspond to making the three types of request that the assistants accepts . The remaining two actions are used to traverse in the operation environment : ( d ) DO : executes the action adot , arg maxa∈A π̂ ( a | bst , d g t ) . The agent transitions to a new operation state st+1 ∼ T ( st , adot ) ; ( e ) DONE : determines that the current goal gt has been reached.2 If gt is a main goal ( gt = g1 ) , the task episode ends . If gt is a subgoal ( gt 6= g1 ) , the agent may choose a new goal to follow . Our problem formulation leaves it open on what goal should be selected next . By selecting among these actions , the interaction policy essentially decides when to ask the assistant for additional information , and what types of information to ask for . Our formulation does not specify the input space of the interaction policy , as this space depends on how the agent implements its goal memory ( i.e . how it stores and retrieves the subgoals ) . In the next section , we introduce an instantiation where the agent uses a stack data structure to manage ( sub ) goals .

Summary: The paper operates in a 2-agent setting, where one agent is a learning agent and the other agent is the assistant. The objective is for the learning agent to seek help from the assistant while solving a task, and the decision of what to ask the assistant and when to do so is made by learning an interaction policy using hierarchical reinforcement learning -- in particular using a variant of POMDPs. The assumption is that the learning agent's operational policy on the task environment is not changed and an additional interaction (hierarchical) policy is learned on top of the operational policy which dictates when and what to ask the assistant. The authors have empirically evaluated their approach on a (simulated) human-assisted navigation task, and show that the proposed method achieves close to seven times an increase in the success rate of solving the task compared to the policy obtained by learning without assistance. Additionally, the results also show that the frequency of seeking assistance was only 25 percent of all actions executed to solve the task.