# Incorporating External Knowledge Reasoning for Vision-and-Language Navigation with Assistant's Help
## 3 Method
### Modeling Human Help  
We model human help by applying a simulated human assistant in the environment. The simulated human assistant guides an agent to take actions during both training and test time through direct intervention. When an agent sends the request signal, the simulated human assistant will overwrite the agent's decision-making with its own, making the agent take the action that the simulated human assistant wants, so direct intervention is always perfectly executed. The simulated human assistant has an oracle-like function that always chooses action \(a_{t}^{D}\) along the shortest path from the current location to the goal location at each time step \(t\):  
\[a_{t}^{D}=\pi_{\text{ass}}^{*}(s_{t})\]  
where \(\pi_{\text{ass}}^{*}\) is the simulated human assistant's policy and \(s_{t}\) is the current environment state.  
To fit real-world applications, an agent should be committed to adaptively decide whether to ask for help during navigation [57]. In our design, the agent is able to ask for help from the simulated human assistant when heuristic-based rules are satisfied and the query budget is greater than 0. Inspired by [4], we introduce 4 heuristic rules as query policy \(\pi_{\text{ask}}\) to decide when the agent can ask for help:  
* The agent deviates from the navigation teacher path by more than \(\delta\) meters. The distance is defined as the length from the agent's current viewpoint to the closest viewpoint on the path.
* The agent is \"confused\", defined as when the difference between the entropy of the uniform navigation distribution and the entropy of the navigation distribution calculated by the agent is less than a threshold \(\epsilon\).
* The agent has remained at the same viewpoint for the last \(\mu\) steps.
* The agent is at a goal viewpoint, but the highest-probability action of the navigation distribution is not _stop_.  
As the agent's navigation performance is improved, the number of help requests to the simulated human assistant should also be reduced to improve the agent's autonomy.  
The query budget \(B_{t}\) and the agent's probability of asking \(\rho_{ask,t}\) are two independent variables, which together determine a dependent variable, _assistant_help_:  
\[\rho_{ask,t}=\left\{\begin{array}{cl}1,&\text{if $a$ or $b$ or $c$ or $d$ $>0$}\\ 0,&\text{else}\end{array}\right.\]  
\[B_{t}=\left\{\begin{array}{cl}B_{t-1}-1,&\text{if $B_{t-1}>0$ and $\rho_{ask,t-1}=1$}\\ B_{t-1},&\text{if $B_{t-1}>0$ and $\rho_{ask,t-1}=0$}\\ 0,&\text{else}\end{array}\right.\]  
\[assistant\_help=\left\{\begin{array}{cl}1,&\text{if $B_{t}>0$ and $\rho_{ask,t}>0$}\\ 0,&\text{else}\end{array}\right.\]  
We set the maximum query budget \(B_{0}=10\) as a study design consideration to balance human participation while having a controllable condition in the experiments.  
At time step \(t\), it \(assistant\_help\) is equal to 1, i.e., the simulated human assistant can receive a help request from the agent, it will use the teacher action to overwrite the agent's decision-making:  
\[a_{t}=a_{t}^{D}\]  
as shown in the upper right part of Figure 1.  
In the experiments section, the results demonstrate that the agent's navigation performance will be improved by indirectly implementing the \"Human-on-the-loop\" function.