# Incorporating External Knowledge Reasoning for Vision-and-Language Navigation with Assistant's Help
###### Abstract  
Vision-and-Language Navigation (VLN) is a task designed to enable embodied agents carry out natural language instructions in realistic environments. Most VLN tasks, however, are guided by an elaborate set of instructions that is depicted step-by-step. This approach deviates from real-world problems in which humans only describe the object and its surroundings and allow the robot to ask for help when required. Vision-based Navigation with Language-based Assistance (VNLA) is a recently proposed task that requires an agent to navigate and find a target object according to a high-level language instruction. Due to the lack of step-by-step navigation guidance, the key to VNLA is to conduct goal-oriented exploration. In this paper, we design an Attention-based Knowledge-enabled Cross-modality Reasoning with Assistant's Help (AKCR-AH) model to address the unique challenges of this task. AKCR-AH learns a generalized navigation strategy from three new perspectives: (1) external commonsense knowledge is incorporated into visual relational reasoning, so as to take proper action at each viewpoint by learning the internal-external correlations among object- and room-entities; (2) a simulated human assistant is introduced in the environment, who provides direct intervention assistance when required; (3) a memory-based Transformer architecture is adopted as the policy framework to make full use of the history clues stored in memory tokens for exploration. Extensive experiments demonstrate the effectiveness of our method compared with other baselines. omonsense knowledge; simulated human assistant; direct intervention; memory-based Transformer; cross-modality reasoning +
Footnote †: journal: _Sci_  
0000-0002-0001-999-2233-3  
in the visual content and infer their relationships for self-exploration and autonomous decision-making. Ref. [4] proposed a valuable task named Vision-based Navigation with Language-based Assistance (VNLA) to further facilitate the general embodied-AI field. The goal of VNLA is to enable an agent in a realistic 3D environment to find the target object according to the given high-level instruction and obtain additional language assistance when navigation is difficult. The language-based assistance from the advisor simulates actual human help, improving the agent's navigation performance. However, some challenges are going to be tackled in achieving the VNLA task: (1) VNLA towards practicability only annotates high-level instructions such as \"Find an armchair in the living room\". This is more natural and closer to the needs of human daily life, but also more challenging. The agent needs to conduct goal-oriented exploration in the environment, rather than strict instruction-following. (2) It may be hard for an agent in the task to find the exact location of a target object through using only the intuition formed by the visual and linguistic multimodal information it receives (since this is hard even for humans). In particular, applying internal-external correlations among rooms and objects learned from limited environments to previously unseen scenes is not trivial, since there is no specific regularity for target object placement. (3) When the agent makes mistakes in understanding the language assistance, i.e., the advisor's original intent, it will deviate from the trajectory suggested by the advisor, further increasing the possibility of failure. (4) In real life, a robot that asks humans for help too frequently or at inappropriate times may annoy them to the point that they stop helping. On the other hand, helping a robot too frequently will cause its autonomy to drop dramatically, although the task success rate will increase. Therefore, the robot needs to improve its autonomous decision-making performance and task success rate as much as possible with limited human assistance. Efficiently solving the above-mentioned challenges can result in many potential applications.