Landmark-RxR: Solving Vision-and-Language Navigation with Fine-Grained Alignment Supervision

1 Introduction . Vision-and-language navigation ( VLN ) is an important task about cross-modal intelligence and can be applied to service and rescue robots . Different from other cross-modal tasks like image/video captioning [ 1 , 2 ] and visual question answering [ 3 ] , where the agent only needs to understand fixed images or videos , VLN agent has to learn and reason by dynamically interacting with the real environment guided by human instructions . Since Anderson et al . [ 4 ] first introduced the VLN task with the coarse-grained dataset Room-to-Room ( R2R ) , great progresses in this direction have been made , ranging from the sequence-to-sequence models [ 4 ] combined with cross-modal grounding modules [ 5–8 ] to Transformer-based models [ 7 , 9–12 ] . Cross-modal alignment is one of the most critical challenges in VLN as the predicted trajectory needs to match the given instruction accurately . However , it is difficult for the agent to learn very ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . accurate visual and textual modality alignment from coarse-grained data , i.e. , only coarse-level correspondences between global instructions and trajectories are annotated , without finer ones between sub-instructions and sub-trajectories . Works mentioned above [ 4–8 , 10–12 ] tried attending on specific parts of the instructions during navigation to help the agent better align salient visual contents with the textual modality . But only coarse-grained data like R2R are not capable of providing enough supervising signals of accurate cross-modal alignment . Some recent works such as FineGrained R2R [ 13 ] and BabyWalk [ 14 ] also found this same issue and proposed fine-grained datasets based on R2R . However , their datasets are generated by heuristic rules and are not precise enough , which inevitably limit the navigation performance . Although Room-across-Room ( RxR ) dataset [ 15 ] includes word-level alignment between each word and point in the trajectory , it lacks segmentation tags to split an instruction into sub-instructions with independent meanings and to figure out the landmarks in a trajectory , namely the marked end points of sub-trajectories . Since words in the same sub-instruction are highly correlated , these important correlations are lost because no sub-instruction exists in RxR . It is the same for the sub-trajectories . As a result , there is also an absence of the correlation between the words in a sub-instruction and the points in corresponding sub-trajectories , which should be an important cross-modal alignment supervision signal . In addition to the dataset annotation , designing better rewards during reinforcement learning is another important issue on cross-modal alignment learning . Currently , reward shaping has been well investigated in VLN based on coarse-grained data to align instructions and trajectories globally , but pays little attention to enhance local cross-modal alignment . The goal-oriented reward simply [ 6 ] uses the arrival signal and the reduced distance to the goal point as the reward , guiding agent to find the global goal points . The fidelity-oriented rewards [ 16 , 17 ] help agent improve the similarity between the predicted trajectory and demonstration trajectory globally by taking intermediate points as external supervision . However , some of the points in the trajectory have more detailed descriptions and are more helpful to navigation process , i.e. , they can assist agent in knowing the visual scenes around these specific points in the trajectories could be better aligned with certain parts of the instructions than the trivial points , where the alignment is in a local to local manner . These points are called critical points in this paper . Thus , an emphasis on these critical points in the reward shaping for better local cross-modal alignment is essential . In this paper , we address the challenges above from the perspective of fine-grain . Firstly , based on the English Guide part of RxR ( en-RxR ) , we introduce a fine-grained dataset Landmark-RxR , which is human-annotated , landmark-based , fine-grained and currently the largest scale . With the groundtruth fine-grained data , experiments demonstrate that agent generalizes better to unseen environments and instructions with domain gap . This indicates that the fine-grained data help agent to align textual and visual modalities better . Secondly , we propose two kinds of focal-oriented rewards that encourage local alignment between instructions and critical points . Since the landmarks in Landmark-RxR naturally meet the requirements of critical points , we just sampled critical points from the landmark set in Landmark-RxR . The focal-oriented rewards outperform the commonly used goal-oriented reward and fidelity-oriented reward . We also propose the re-initialization mechanism to fully evaluate the navigation process in a way that is insensitive to difficult points , which can cause the deviation from the correct trajectory . With the fine-grained data and focal-oriented rewards , our agent shows superior navigation performance on Landmark-RxR , en-RxR and R2R . 2 Related Work . Vision-and-Language Navigation Anderson et al . [ 4 ] first introduced the simulated , photorealistic VLN task with a benchmark Room-to-Room ( R2R ) and several sequence-to-sequence baselines . Fried et al . [ 5 ] utilized the speaker-driven data and panoramic action space to improve navigation performance . Wang et al . [ 18 ] proposed a planned-ahead hybrid reinforcement learning model with model-free and model-based reinforcement learning for better generalization . Tan et al . [ 19 ] first trained agent with mixed imitation learning and reinforcement learning and used unseen triplets generated by environment dropout method for fine-tuning then . Jain et al . [ 16 ] and Ilharco et al . [ 17 ] proposed the fidelity-oriented rewards to ensure the similarity between predicted trajectories and demonstration trajectories . Kurita et al . [ 20 ] combined generative and discriminative policy to enhance the navigation ability from different aspects . An et al . [ 21 ] fused neighbor visual contexts both globally and locally to get visual features with richer semantics . Huang et al . [ 22 ] defined two in-domain auxiliary tasks for representation learning which benefit the downstream navigation task . Li et al . [ 7 ] leveraged the large-scale pretrained language models BERT [ 23 ] and GPT [ 24 ] for powerful text representation . Hao et al . [ 10 ] pretrained and finetuned generic transformer-based model in VLN environment and the pretrained model generalizes well to other two navigation tasks . Majumdar et al . [ 11 ] demonstrated that pretrained on image-text pairs from the web , the transformerbased model can perform on VLN task well after fine-tuning . Hong et al . [ 12 ] proposed a Recurrent Vision-and-Language BERT to process time-dependent tasks like VLN efficiently . Fine-Grained VLN Wang et al . [ 6 ] designed a cross-modal grounding module to infer which part of the instructions to focus on . Ma et al . [ 8 ] introduced a self-monitoring agent with a visual-textual co-grounding module to locate the required part in instructions and a progress monitor to reflect navigation progress . Qi et al . [ 25 ] decoupled instructions into object-aware and action-aware parts for more accurate action prediction . Zhu et al . [ 14 ] adopted a two-stage method for navigation . The given instruction is first split into several sub-instructions named baby-steps according to heuristic rules and then the baby-steps which are much easier to navigate are fed to the agent one by one during validation . Similar to [ 14 ] , Hong et al . [ 13 ] first segmented instruction into sub-instructions using a heuristic method . Then the agent navigates following these easy sub-instructions sequentially with a shifting module inferring whether current sub-instruction has been completed . Ku et al . [ 15 ] explored utilizing spatially-temporally aligned annotations to supervise the textual and visual attention weights but got very few performance improvements . Different from these works , we propose a human-annotated , fine-grained dataset Landmark-RxR to alleviate the weak cross-modal alignment supervision from the coarse-grained data . Our experiments is designed to demonstrate the supervision from fine-grained and coarse-grained data can complement each other to improve the cross-modal alignment ability of the model itself . In addition , two kinds of focal-oriented rewards using fine-grained supervision signals are proposed to enhance the local cross-modal alignment by paying more attention to critical points which benefit the navigation process more . 3 Landmark-RxR Dataset . Our Landmark-RxR is built based on the English Guide split of RxR ( en-RxR ) . It contains subinstruction and sub-trajectory pairs ( sub pairs ) split from instructions in en-RxR . To facilitate the collection of sub pairs , we develop a 3D web-based collection tool . It allows the annotator to mark the landmarks , namely the end points of sub-trajectories , in trajectories and split out the corresponding sub-instructions . During annotation , the annotators can move between the discrete points or change heading and evaluation continuously by mouse click , like what the agent will do in the VLN task . In the following , we will describe the data collection and dataset analysis of Landmark-RxR . 3.1 Data Collection . The annotators are asked to read the complete instructions and then explore the 3D environments following corresponding trajectories . If landmarks are found , annotators need to mark them and split out the corresponding sub-instructions by interacting with the collection tool . As illustrated in Figure 1 ( a ) , the complete instruction and complete trajectory pair ( complete pair ) are sampled from en-RxR . In Figure 1 ( b ) , three landmarks in the complete trajectory are marked and the corresponding sub-instructions are also split out by annotators . These three sub pairs will be included in our Landmark-RxR dataset . Five principles are proposed to ensure the annotation process more standardized . Matching Principle : The sub-trajectory should match the sub-instruction accurately . Independent Meaning Principle : Each sub-instruction expresses an independent meaning . Position Change Principle : The end point of each sub-trajectory should differ from the start point . Clear End Point Principle : The end point should be described clearly in a sub-instruction . For example , the sub-instruction go toward the door has a clear description of end point door , but no end point is described in the sub-instruction go forward . Minimum Granularity Principle : The sub-instruction and sub-trajectory pair should be the minimum granularity which means it can not be split anymore . After an annotator finishes the annotations , another annotator will verify the annotations again and modify the inaccurate part to ensure the annotations satisfy the five principles . For some mistakes in RxR , we also correct them in Landmark-RxR . Totally 30 annotators participated in the annotation task , contributing about 2,700 hours . 3.2 Dataset Analysis . Table 1 gives statistics on R2R , RxR , en-RxR and Landmark-RxR . The total number of subinstructions from Landmark-RxR is 166,740 , which contains 133,602 sub-instructions in train split , 13,591 sub-instructions in validation seen split , and 19,547 sub-instructions in validation unseen split . The average number of edges in a sub-trajectory is 1.6 with 21 words in the corresponding sub-instruction on average . The number of sub-trajectories contained in Landmark-RxR is 46,645 . Landmark-RxR has the largest scale of instructions and trajectories and the minimum granularity among current human-annotated datasets . The fine-grained annotations are essential to effective cross-modal alignment learning and more accurate evaluation . In addition , because of the minimum granularity principle , the dataset can be easily expanded to a larger scale with different granularities by recombining the sub pairs . 4 Re-initialization Mechanism and Evaluation Metrics . 4.1 Re-initialization Mechanism . An agent needs to navigate following the instructions strictly . So it is essential to evaluate the fidelity between predicted trajectories and the instructions exactly . Current metrics like Coverage weighted by Length Score ( CLS ) [ 16 ] and the normalized Dynamic Time Warping ( nDTW ) [ 17 ] try to obtain the fidelity by computing the similarity between predicted trajectory and the demonstration 1 Note that the number of public instructions in RxR test split is 12,469. trajectory . However , the nDTW and CLS scores are sensitive to the positions of difficult points , and have a deviation as a result when evaluating the navigation ability . As illustrated in Figure 2 , the agent is initialized at point A and ready to follow the given instruction navigating to point B . Then during navigation , the agent deviates from demonstration trajectory seriously after encountering the difficult point and can only get a low fidelity score . The difficult point indicates the point where next action is hard to choose because of the difficult visual scene , e.g. , cluttered and similar objects in Figure 2 , or ambiguous expressions in an instruction . However , for the same agent navigating from point B to point A , it is easy to finish the majority of the trajectory and get a higher fidelity score because the difficult point is just near the end . The distributions of difficult points in trajectory significantly impact current fidelity metrics , especially for the long trajectories . Based on the landmark annotations in Landmark-RxR , we introduce the re-initialization mechanism to alleviate the influence of difficult points sensitivity of the fidelity metrics . We define a trajectory as T = [ ST1 , ST2 , ... , ST|T | ] , where STi = [ Pi1 , Pi2 , ... , Li ] , and corresponding instruction is defined as I = [ SI1 , SI2 , ... , SI|I| ] . Pij means the jth point in the ith sub-trajectory STi . Li means the ith landmark in the trajectory , in other words , the end point of the ith sub-trajectory and SIi means the ith sub-instruction . As illustrated in Figure 3 , if the agent reaches within 3 meters , which is a fixed threshold for success metric defined in [ 4 ] , of the landmark Li ( i < |T | ) , or the agent can stop within 3 meters of the landmark L|T | in π ( π is set to 10 in this paper ) steps from the last landmark , the navigation to landmark Li is successful . The failure is defined that the agent fails to get within 3 meters of the landmark Li ( i < |T | ) in π steps from the last landmark , or stops 3 meters away from the landmark L|T | . If the agent fails in navigating to landmark Li ( i < |T | ) , we re-initialize the agent at landmark Li as the new start point , and reset the given instruction as Ireset = [ SIi+1 , ... , SI|I| ] . Since the re-initialization mechanism can correct serious deviations in time , for a complete trajectory , the averaged fidelity score of its sub-trajectories will be much less insensitive to the difficult points and can better reflect the navigation ability no matter where the difficult points locate .

Summary: The paper proposes to improve vision and language navigation by augmenting the RxR dataset with corresponded subinstructions and subtrajectories.  Using the notion of critical points (points that are more important/helpful in the navigation process), two kinds of focal-oriented rewards (soft and hard) are proposed to encourage the predicted trajectory to be close to the demonstration trajectories at critical points.  The authors conduct experiments on R2R and RxR and show that using the collected subinstructions and subtrajectories and proposed losses, the trained agent is able to outperform the baseline RCM model.  For the RxR dataset, the training data consisted both of original RxR data as well as an augmented dataset consisting of concatenating the collected subinstruction and subtrajectories together.  Main contributions - Landmark-RxR: Additional annotations on top of the RxR dataset providing sub-instructions paired with sub-trajectories - Proposed to use modified rewards for improved cross-modal alignment - Ablation experiments of proposed method on the R2R and RxR dataset