Question: A computational experiment is used to show that the ability to request information was useful to the agent. However, it seems obvious that an agent with less uncertainty will do better. Section 6 describes the model comparison as comparing the agent that requests assistance to an agent that takes random actions. However, with missing annotations a rational agent that can not request information should still be able to rationally search the graph until the object is found - which seems a more reasonable  performance benchmark.  Alternatively, the authors could compare the current framework to a state-of-the-art agent that can solve the same benchmark problem. Was the indoor object-finding task used in previous work, or as part of benchmark problem sets?
Evidence: Problem. We apply HARI to modeling a human-assisted navigation (HAN) problem. In HAN, a human requests an agent to find an object in an indoor environment. Each task request asks the agent to go to a room of type r and find an object of type o (e.g., find a mug in a kitchen). The agent is equipped with a camera and shares its camera view with the human. We assume that the human is sufficiently familiar with the environment that they can recognize the agent's location by looking at its current view. Before issuing a task request, the human imagines a goal location (but do not reveal it to the agent). We are primarily interested in evaluating success in goal-finding, i.e. whether the agent can arrive at the human's intended goal location. Even though there could be multiple locations that match a request, the agent only succeeds if it arrives exactly at the chosen goal location. We also determine success in request-fulfilling, where the agent successfully fulfills a request if it navigates to any node that is within two meters of an object that matches the request.
While an agent is performing a task, it may request the human to provide additional information via telecommunication (e.g., a chat app). Specifically, it can ask for a description of its current location (CUR), the goal location (GOAL), or a subgoal location that is on the path from its current location to the goal location (SUB). Detail about how the subgoals are determined is in the Appendix.
Operation Environment. We construct the operation environments using the environment graphs provided by the Matterport3D simulator. Each environment graph is generated from a 3D model of a house where each node is a location in the house and each edge connects two nearby unobstructed locations. Each operation state s corresponds to a node in the graph. At any time, the agent's operation action space A consists of traversing to any of the nodes that are adjacent to its current node.
We employ a discrete bag-of-features representation for state descriptions. 3 A bag of features represents the information that the agent extracts from the raw input that the agent perceives (e.g., an image, a language sentence). Working with this intermediate input allows us to easily vary the type and amount of information given to the agent. Specifically, we simulate two settings of descriptions: dense and sparse. At evaluation time, the agent perceives sparse descriptions and request the assistant for dense descriptions. A dense description of a current location contains the room name at the location, and the features of M objects restricted to be within δ meters of the location. The features of each object consists of (i) its name, (ii) horizontal and vertical angles (relative to the current viewpoint), and (iii) distance (in meters) from the object to the current location. A dense description of a goal follows the same representation scheme. In the sparse setting, the current-location description does not include the room name. Moreover, we remove the features of objects that are not in the top 100 most frequent objects, emulating an imperfect object detector module. The sparse goal description (the task request) has only features of the target object and the room name where the object is located at. Especially, if a subgoal location is adjacent or coincides with the agent's current location, instead of describing room and object features, the human specifies the ground-truth action to go to the subgoal (an action is specified by its horizontal and vertical angles, and travel distance). 4   Experimental Procedure. We conduct our experiments in three phases. In the pre-training phase, we learn an operation policy π with dense descriptions of the current location and the goal. In the training phase, the agent perceives a sparse description of its current location and is given a sparse initial goal description. We use advantage actor-critic to learn an interaction policy ψ θ that controls how the agent communicates with the human and navigates in an environment. The human always returns dense descriptions. The interaction policy is trained in environments that are previously seen as well as unseen during pre-training. Finally, in the evaluation phase, the interaction policy is tested on three conditions: seen environment and target object type but starting from a new room (UNSEENSTR), seen environment but new target object type (UNSEENOBJ), and new environment (UNSEENENV).
We introduce an assistant into the operation environment, who can provide information about the environment's states. We assume the agent possesses a pre-learned operation policy π. This policy serves as the common ground between the agent and the assistant, which is a prerequisite for communication between them to occur. For example, this policy represents a set of basic tasks that agent has mastered and the assistant can ask the agent to perform. In general, the more knowledge encoded in this policy, the more effectively the agent can communicate with and leverage help from the assistant. Our goal is to learn an interaction policy ψ θ (parametrized by θ) that controls how the agent communicates with the assistant to gather additional information. The operation policy π will be invoked by the interaction policy if the latter decides that the agent does not need new information and wants to take an operating action.
The assistant aids the agent by giving new (current or goal) state descriptions, connecting the agent to situations on which it can make better decisions. Consider an object-finding navigation problem, where a robot has been trained to reliably navigate to the kitchen from the living room of a house. Suppose the robot is then asked to "find a mug", an object that it has never heard of. The assistant can help the robot accomplish this task by giving a more informative goal description "find a mug in the kitchen", relating the current task to the kitchen-finding task that the robot has been familiar with. The robot may also have problems with localization: it knows how to get the kitchen from the living room but it may not realize that it is currently the living room. In this case, giving a currentstate description that specifies this information provides the robot with a useful hint on what actions to take next.
Our framework allows the assistant to convey any form of information that the agent can incorporate into its input. As discussed in §2, the notion of "state description" in our framework is general, capturing various types of information, including but not limited to visual perception and verbal description. Communication between the agent and the assistant can be flexibly enriched by designing the agent's operation policy to be able to consume the forms of information of interest (e.g., a policy that takes natural language as input).
Communication with the Assistant. The assistant is present all the time and knows the agent's current state s t and the goal state g t . It is specified by two functions: a description function ρ A : S × D → ∆(D) and a subgoal function ω A : S × S → ∆(S). ρ A (d | s, d) specifies the probability of giving d as the new description of state s given a current description d. ω A (g | s, g) indicates the probability of proposing g as a subgoal given a current state s and a goal state g. By selecting among these actions, the interaction policy essentially decides when to ask the assistant for additional information, and what types of information to ask for. Our formulation does not specify the input space of the interaction policy, as this space depends on how the agent implements its goal memory (i.e. how it stores and retrieves the subgoals). In the next section, we introduce an instantiation where the agent uses a stack data structure to manage (sub)goals.
Our model has two components: an encoder and a decoder; both are implemented as Transformer models (Vaswani et al., 2017).****Operation Environment. We construct the operation environments using the environment graphs
provided by the Matterport3D simulator (Anderson et al., 2018). Each environment graph is generated from a 3D model of a house where each node is a location in the house and each edge connects
two nearby unobstructed locations.****We introduce an assistant into the operation environment, who can provide information about the environment’s states. We assume the agent possesses a pre-learned operation policy πˆ.
Answer: We are not sure what “rationally search the graph” means. Our "no assistance" baselines search the graph with pre-learned knowledge. An approach that exhaustively searches the graph overly exploits the structure of the simulator and may not scale to real continuous environments. We impose a time constraint so the agent cannot simply visit an entire house for each task. Even if the agent performed an exhaustive search, it would still need to detect objects (i.e. learning when to call DONE). This navigation problem is proposed by our paper. We implement a transformer-based model trained with imitation learning, which is a strong baseline for the popular VLN problem (https://arxiv.org/abs/2011.13922). Our task is harder than VLN in that the task command is high-level (“find O in room R” vs “turn left... go to X... stop there”).