Self-Monitoring Navigation Agent via Auxiliary Progress Estimation

1 INTRODUCTION . Recently , the Vision-and-Language ( VLN ) navigation task ( Anderson et al. , 2018b ) , which requires the agent to follow natural language instructions to navigate through a photo-realistic unknown environment , has received significant attention ( Wang et al. , 2018b ; Fried et al. , 2018 ) . In the VLN task , an agent is placed in an unknown realistic environment and is required to follow natural language instructions to navigate from its starting location to a target location . In contrast to some existing navigation tasks ( Kempka et al. , 2016 ; Zhu et al. , 2017 ; Mirowski et al. , 2017 ; 2018 ) , we address the class of tasks where the agent does not have an explicit representation of the target ( e.g. , location in a map or image representation of the goal ) to know if the goal has been reached or not ( Matuszek et al. , 2013 ; Hemachandra et al. , 2015 ; Duvallet et al. , 2016 ; Arkin et al. , 2017 ) . Instead , the agent needs to be aware of its navigation status through the association between the sequence of observed visual inputs to instructions . Consider an example as shown in Fig . 1 , given the instruction ” Exit the bedroom and go towards the table . Go to the stairs on the left of the couch . Wait on the third step. ” , the agent first needs to locate which instruction is needed for the next movement , which in turn requires the agent to be aware of ( i.e. , to explicitly represent or have an attentional focus on ) which instructions were completed or ongoing in the previous steps . For instance , the action ” Go to the stairs ” should be carried out once the agent has exited the room and moved towards the table . However , there exists inherent ambiguity for ” go towards the table ” . Intuitively , the agent is expected to ” Go to the stairs ” after completing ” go towards the table ” . But , it is not clear what defines the completeness of ” Go towards the table ” . The completeness of an ongoing action often depends on the availability of the next action . Since the transition between past and next part of the instructions is a soft ∗Work done while the authors were research interns at Salesforce Research . boundary , in order to determine when to transit and to follow the instruction correctly the agent is required to keep track of both grounded instructions . On the other hand , assessing the progress made towards the goal has indeed been shown to be important for goal-directed tasks in humans decision-making ( Benn et al. , 2014 ; Chatham et al. , 2012 ; Berkman & Lieberman , 2009 ) . While a number of approaches have been proposed for VLN ( Anderson et al. , 2018b ; Wang et al. , 2018b ; Fried et al. , 2018 ) , previous approaches generally are not aware of which instruction is next nor progress towards the goal ; indeed , we qualitatively show that even the attentional mechanism of the baseline does not successfully track this information through time . In this paper , we propose an agent endowed with the following abilities : ( 1 ) identify which direction to go by finding the part of the instruction that corresponds to the observed images—visual grounding , ( 2 ) identify which part of the instruction has been completed or ongoing and which part is potentially needed for the next action selection—textual grounding , and ( 3 ) ensure that the grounded instruction can correctly be used to estimate the progress made towards the goal , and apply regularization to ensure this —progress monitoring . Therefore , we introduce the self-monitoring agent consisting of two complementary modules : visual-textual co-grounding and progress monitor . More specifically , we achieve both visual and textual grounding simultaneously by incorporating the full history of grounded instruction , observed images , and selected actions into the agent . We leverage the structural bias between the words in instructions used for action selection and progress made towards the goal and propose a new objective function for the agent to measure how well it can estimate the completeness of instruction-following . We then demonstrate that by conditioning on the positions and weights of grounded instruction as input , the agent can be self-monitoring of its progress and further ensure that the textual grounding accurately reflects the progress made . Overall , we propose a novel self-monitoring agent for VLN and make the following contributions : ( 1 ) We introduce the visual-textual co-grounding module , which performs grounding interdependently across both visual and textual modalities . We show that it can outperform the baseline method by a large margin . ( 2 ) We propose to equip the self-monitoring agent with a progress monitor , and for navigation tasks involving instructions instantiate this by introducing a new objective function for training . We demonstrate that , unlike the baseline method , the position of grounded instruction can follow both past and future instructions , thereby tracking progress to the goal . ( 3 ) With the proposed self-monitoring agent , we set the new state-of-the-art performance on both seen and unseen environments on the standard benchmark . With 8 % absolute improvement in success rate on the unseen test set , we are ranked # 1 on the challenge leaderboard . 2 SELF-MONITORING NAVIGATION AGENT . 2.1 NOTATION . Given a natural language instruction with L words , its representation is denoted by X = { x1 , x2 , . . . , xL } , where xl is the feature vector for the l-th word encoded by an LSTM language encoder . Following Fried et al . ( 2018 ) , we enable the agent with panoramic view . At each time step , the agent perceives a set of images at each viewpoint vt = { vt,1 , vt,2 , ... , vt , K } , where K is the maximum number of navigable directions1 , and vt , k represents the image feature of direction k. The co-grounding feature of instruction and image are denoted as x̂t and v̂t respectively . The selected action is denoted as at . The learnable weights are denoted with W , with appropriate sub/super-scripts as necessary . We omit the bias term b to avoid notational clutter in the exposition . 2.2 VISUAL AND TEXTUAL CO-GROUNDING . First , we propose a visual and textual co-grounding model for the vision and language navigation task , as illustrated in Fig . 2 . We model the agent with a sequence-to-sequence architecture with attention by using a recurrent neural network . More specifically , we use Long Short Term Memory ( LSTM ) to carry the flow of information effectively . At each step t , the decoder observes representations of the current attended panoramic image feature v̂t , previous selected action at−1 and current grounded instruction feature x̂t as input , and outputs an encoder context ht : ht = LSTM ( [ x̂t , v̂t , at−1 ] ) ( 1 ) where [ , ] denotes concatenation . The previous encoder context ht−1 is used to obtain the textual grounding feature x̂t and visual grounding feature v̂t , whereas we use current encoder context ht to obtain next action at , all of which will be illustrated in the rest of the section . Textual grounding . When the agent moves from one viewpoint to another , it is required to identify which direction to go by relying on a grounded instruction , i.e . which parts of the instruction should be used . This can either be the instruction matched with the past ( ongoing action ) or predicted for the future ( next action ) . To capture the relative position between words within an instruction , we incorporate the positional encoding PE ( · ) ( Vaswani et al. , 2017 ) into the instruction features . We then perform soft-attention on the instruction features X , as shown on the left side of Fig . 2 . The attention distribution over L words of the instructions is computed as : ztextualt , l = ( Wxht−1 ) > PE ( xl ) , and αt = softmax ( ztextualt ) , ( 2 ) where Wx are parameters to be learnt . ztextualt , l is a scalar value computed as the correlation between word l of the instruction and previous hidden state ht−1 , andαt is the attention weight over features in X at time t. Based on the textual attention distribution , the grounded textual feature x̂t can be obtained by the weighted sum over the textual features x̂t = αTt X . 1Empirically , we found that using only the images on navigable directions to be slightly better than using all 36 surrounding images ( 12 headings × 3 elevations with 30 degree intervals ) . Visual grounding . In order to locate the completed or ongoing instruction , the agent needs to keep track of the sequence of images observed along the navigation trajectory . We thus perform visual attention over the surrounding views based on its previous hidden vector ht−1 . The visual attention weight βt can be obtained as : zvisualt , k = ( Wvht−1 ) > g ( vt , k ) , and βt = softmax ( zvisualt ) , ( 3 ) where g is a one-layer Multi-Layer Perceptron ( MLP ) , Wv are parameters to be learnt . Similar to Eq . 2 , the grounded visual feature v̂t can be obtained by the weighted sum over the visual features v̂t = β T t V . Action selection . To make a decision on which direction to go , the agent finds the image features on navigable directions with the highest correlation with the grounded navigation instruction x̂t and the current hidden state ht . We use the inner-product to compute the correlation , and the probability of each navigable direction is then computed as : ot , k = ( Wa [ ht , x̂t ] ) > g ( vt , k ) and pt = softmax ( ot ) , ( 4 ) where Wa are the learnt parameters , g ( · ) is the same MLP as in Eq . 3 , and pt is the probability of each navigable direction at time t. We use categorical sampling during training to select the next action at . Unlike the previous method with the panoramic view ( Fried et al. , 2018 ) , which attends to instructions only based on the history of observed images , we achieve both textual and visual grounding using the shared hidden state output containing grounded information from both textual and visual modalities . During action selection , we rely on both hidden state output and grounded instruction , instead of only relying on grounded instruction . 2.3 PROGRESS MONITOR . It is imperative that the textual-grounding correctly reflects the progress towards the goal , since the agent can then implicitly know where it is now and what the next instruction to be completed will be . In the visual-textual co-grounding module , we ensure that the grounded instruction reasonably informs decision making when selecting a navigable direction . This is necessary but not sufficient for ensuring that the notion of progress to the goal is encoded . Thus , we propose to equip the agent with a progress monitor that serves as regularizer during training and prunes unfinished trajectories during inference . Since the positions of localized instruction can be a strong indication of the navigation progress due to the structural alignment bias between navigation steps and instruction , the progress monitor can estimate how close the current viewpoint is to the final goal by conditioning on the positions and weights of grounded instruction . This can further enforce the result of textual-grounding to align with the progress made towards the goal and to ensure the correctness of the textual-grounding . The progress monitor aims to estimate the navigation progress by conditioning on three inputs : the history of grounded images and instructions , the current observation of the surrounding images , and the positions of grounded instructions . We therefore represent these inputs by using ( 1 ) the previous hidden state ht−1 and the current cell state ct of the LSTM , ( 2 ) the grounded surrounding images v̂t , and ( 3 ) the distribution of attention weights of textual-grounding αt , as shown at the bottom of Fig . 2 represented by dotted lines . Our proposed progress monitor first computes an additional hidden state output hpmt by using grounded image representations v̂t as input , similar to how a regular LSTM computes hidden states except we use concatenation over element-wise addition for empirical reasons2 . The hidden state output is then concatenated with the attention weightsαt on textual-grounding to estimate how close the agent is to the goal3 . The output of the progress monitor ppmt , which represents the completeness of instruction-following , is computed as : hpmt = σ ( Wh ( [ ht−1 , v̂t ] ) ⊗ tanh ( ct ) ) , p pm t = tanh ( Wpm ( [ αt , h pm t ] ) ) ( 5 ) where Wh and Wpm are the learnt parameters , ct is the cell state of the LSTM , ⊗ denotes the element-wise product , and σ is the sigmoid function . 2We found that using concatenation provides slightly better performance and stable training . 3We use zero-padding to handle instructions with various lengths .

Summary: The paper considers the problem of following natural language route instructions in an unknown environment given only images. Integral to the proposed ("self-aware") approach is its ability to reason over which aspects of the instruction have been completed, which are to be followed next, which direction to go in next, as well as the agents current progress. This involves two primary components of the architecture. The first is a visual-textual module that grounds to the completed instruction, the next instruction, and the next direction based upon the visual input. The second is a "progress monitor" that takes the grounded instruction as input and captures the agent's progress towards completing the instruction.