Question: Page 2: "The agent achieves this level of performance while issuing only 4.8 requests to the assistant on average, representing less than 1⁄4 of the total number of actions taken in a task execution." Why is this significant or relevant? The number of times the agent has to make requests is determined by how much information is missing in the evaluation task, compared to the task it was trained on -- which means that the exact numbers are arbitrary. Such references to arbitrary numbers happen in a few places through the paper.
Evidence: Settings. In our main experiments, we set: the cost of taking a CUR, GOAL, SUB, or DO action to be 0.01 (we will consider other settings subsequently), the cost of calling DONE to terminate the main goal (i.e. task error) equal the (unweighted) length of the shortest-path from the agent's location to the goal, and the goal stack's size (L) to be 2. We compare our RL-learned interaction policy with a rule-based baseline that first takes the GOAL action and then randomly selects actions.
In each episode, we enforce that the rule-based policy can take at most X a + y actions of type a, where y ∼ Bernoulli(X a − X a ) and X a is a constant. We tune each X a on the validation sets so that the rule-based policy has the same average count of each action as the RL-learned policy. To prevent early termination, we enforce that the rule-based policy cannot take more DONE actions than SUB actions unless when its SUB action's budget is exhausted. We also construct a skyline where the interaction policy is also learned by RL but with an operation policy that executes subgoals perfectly. As discussed in §5, we are primarily interested in goal-finding success rate and will refer to this metric briefly as success rate.
Main Results (Table). To inspect the potential benefits of asking for additional information, we compute how much the operation policy π improves when it is supplied with dense information about the current and/or goal states. As seen, success rate of the operation policy is lifted dramatically when both the current-state and goal descriptions are dense (∼2× increase on UNSEENSTR, ∼5× on UNSEENOBJ, and ∼3× on UNSEENENV). We find that dense information about the current state is more helpful on UNSEENSTR, while dense information about the goal is more valuable on UNSEENOBJ and UNSEENENV. This is reasonable because on UNSEENSTR, the agent has been trained to find similar goals. In contrast, the initial goal descriptions in UNSEENOBJ and UNSEE-NENV are completely new to the agent, thus gathering more information about them is necessary.
Aided by our RL-learned interaction policy, the agent observes a substantial ∼2× increase in success rate on UNSEENSTR, ∼5× on UNSEENOBJ, and ∼7× on UNSEENENV, compared to when performing tasks using only the operation policy. In unseen environments, with its capability of requesting subgoals, the agent impressively doubles the success rate of the operation policy that has access to dense descriptions. The RL-learned interaction policy is significantly more effective than (a) How frequently does the interaction policy execute different actions, on average across different types of tasks? Subgoals are requested much more in unseen environments.
q q q q q 0.0 0.5 1.0 1.5 2.0 0.2 0.4 0.6 0.8 1.0
Fraction of episode Average actions q CUR GOAL SUB (b) Over the course of a trajectory, how does the frequency of different types of actions change in unseen environments? Subgoals are requested in the middle, goal information at the beginning.
Figure: Analyzing the behavior of the RLlearned interaction policy (on validation environments).
q q q q q q q q q 0 25 50 75 −3 −2 −1 log10(cost) Success rate (%) q unseen_str unseen_obj unseen_env (a) Effect of cost on success rate.
q q q q q q q q q 0 3 6  the rule-based baseline (+7.1% on UNSEENENV). Compared to a policy that calls GOAL at the beginning and calls CUR at every step (which is equivalent to the dense-d s -and-d g baseline), our policy achieves higher success rate on UNSEENENV while making two times fewer requests. This is due to the ability to request subgoals.
On UNSEENSTR and UNSEENOBJ, the RL-learned policy has not closed the gap with the operation policy that performs tasks with dense descriptions. Our investigation finds that limited information often causes the policy to not request information about the current state (i.e. taking CUR) and terminate prematurely or go to a wrong place. Encoding uncertainty in the current-state description (e.g.,;) is a plausible future direction for tackling this issue. Finally, results obtained by replacing the learned operation policy with one that behaves optimally on subgoals shows that further improving performance of the operation policy on shortdistance goals would effectively enhance the agent's performance on long-distance goals.
Behavior of the RL-Learned Interaction Policy. Figure characterizes behaviors of the RLlearned interaction policy in three evaluation conditions. We expect that tasks in UNSEENSTR are the easiest and those in UNSEENENV are the hardest. As the difficulty of the evaluation condition increases, the interaction policy issues more CUR, SUB, and DO actions. The average number of GOAL actions does not vary, showing that the interaction policy has correctly learned that making more than one goal-clarifying request is unnecessary. Figure illustrates the distribution of each action along the length of an episode in the validation UNSEENENV dataset. The GOAL action, if taken, is always taken only once and immediately in the first step. The number of CUR actions gradually decreases over time. The agent makes most SUB requests in the middle of an episode, after its has attempted but failed to accomplish the main goals. We observe similar patterns on the other two validation sets.
Effects of Varying Action Cost. As mentioned, we assign the same cost to each CUR, GOAL, SUB, or DO action. Figure demonstrates the effects of changing this cost on the success rate of the agent. Setting the cost equal to 0.5 makes it too costly to take any action, inducing a policy that always calls DONE in the first step and thus fails on all tasks. Overall, the success rate of the agent rises as we reduce the action cost. The increase in success rate is most visible in UNSEENENV and least visible in UNSEENSTR. Figure provides more insights. As the action cost decreases, we observe a growth in the number of SUB and DO actions taken by the interaction policy. Meanwhile, the numbers of CUR and GOAL actions are mostly static. Since requesting subgoals is more helpful in unseen environments than in seen environments, the increase in the number of SUB actions leads the more visible boost in success rate on UNSEENENV tasks.
Performing Tasks with Deeper Goal Stacks. In Table, we test the functionality of our framework with a stack size 3, allowing the agent to request subgoals of subgoals. As expected, success rate on UNSEENENV is boosted significantly (+11.9% compared to using a stack of size 2). Success rate on UNSEENOBJ is largely unchanged; we find that the agent makes more SUB requests (averagely 4.5 requests per episode compared to 1.0 request made when the stack size is 2), but doing focus on learning a teaching policy in addition to an advice-requesting policy. An important assumption in these papers is that the teacher must share a common action space with the learner. More recent frameworks relax this assumption by allowing the teacher to specify high-level subgoals instead of low-level actions. HARI can be viewed as a strict extension of these frameworks. It allows the human to specify not only subgoals, but also any additional information about the current state and the goal that agent can interpret. Moreover, HARI equips the agent with multiple communication intentions and teaches it to select the most useful intention to convey in a given situation. Another line of work employs standard RL communication protocol, where the human can only transfer knowledge through numerical scores or categorical feedback. propose a framework where the human advises the agent using a domain-specific language, specifying rules that can be incorporated into the agent's model. In contrast, HARI operates with a black-box agent model. extract features from various types of language feedback to construct a reward function for reinforcement learning. We instead focus on deployment-time communication and directly incorporate the human feedback as input to agent's operation policy.
Task-Oriented Dialog and Generating Natural Language Questions. HARI models a taskoriented dialog problem. Many variants of this problem requires the agent to compose specific questions. The dominant approach in these problems is to mimic pre-collected human utterances. As discussed previously, naively mirroring human external behavior cannot enable agents to understand the limits of their knowledge. We teach the agent to understand its intrinsic needs through interaction with the human and the environment rather than through imitation of human behaviors. Another related line of work concerns generating natural language explanations of model decisions.
In summary, this paper presents a general POMDP framework for modeling human-agent communication. While we demonstrate this framework on a simplified navigation problem, our framework can theoretically capture richer types of human-agent communication. Hence, an important empirical question is how well our formulation generalizes to richer environments with more complex interactions and state spaces. Enhancing the sample efficiency of the learning policy by exploiting the hierarchical policy structure is an exciting future direction. Furthermore, techniques for generating faithful explanations can be applied to enhance the specificity of the generated questions.
Compared to a policy that calls GOAL at the beginning and calls CUR at every step (which is equivalent to the dense-ds -and-dg baseline), our policy achieves higher success rate on UNSEENENV while making two times fewer requests. This is due to the ability to request subgoals.
Answer: ¼ actions are requests: we are comparing our agent with the agent that asks all the time (the dense ds and dg skyline). Our agent outperforms this agent on unseen environments, while making fewer requests. We will revise this claim.