Landmark-RxR: Solving Vision-and-Language Navigation with Fine-Grained Alignment Supervision

1 Introduction . Vision-and-language navigation ( VLN ) is an important task about cross-modal intelligence and can be applied to service and rescue robots . Different from other cross-modal tasks like image/video captioning [ 1 , 2 ] and visual question answering [ 3 ] , where the agent only needs to understand fixed images or videos , VLN agent has to learn and reason by dynamically interacting with the real environment guided by human instructions . Since Anderson et al . [ 4 ] first introduced the VLN task with the coarse-grained dataset Room-to-Room ( R2R ) , great progresses in this direction have been made , ranging from the sequence-to-sequence models [ 4 ] combined with cross-modal grounding modules [ 5–8 ] to Transformer-based models [ 7 , 9–12 ] . Cross-modal alignment is one of the most critical challenges in VLN as the predicted trajectory needs to match the given instruction accurately . However , it is difficult for the agent to learn very ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . accurate visual and textual modality alignment from coarse-grained data , i.e. , only coarse-level correspondences between global instructions and trajectories are annotated , without finer ones between sub-instructions and sub-trajectories . Works mentioned above [ 4–8 , 10–12 ] tried attending on specific parts of the instructions during navigation to help the agent better align salient visual contents with the textual modality . But only coarse-grained data like R2R are not capable of providing enough supervising signals of accurate cross-modal alignment . Some recent works such as FineGrained R2R [ 13 ] and BabyWalk [ 14 ] also found this same issue and proposed fine-grained datasets based on R2R . However , their datasets are generated by heuristic rules and are not precise enough , which inevitably limit the navigation performance . Although Room-across-Room ( RxR ) dataset [ 15 ] includes word-level alignment between each word and point in the trajectory , it lacks segmentation tags to split an instruction into sub-instructions with independent meanings and to figure out the landmarks in a trajectory , namely the marked end points of sub-trajectories . Since words in the same sub-instruction are highly correlated , these important correlations are lost because no sub-instruction exists in RxR . It is the same for the sub-trajectories . As a result , there is also an absence of the correlation between the words in a sub-instruction and the points in corresponding sub-trajectories , which should be an important cross-modal alignment supervision signal . In addition to the dataset annotation , designing better rewards during reinforcement learning is another important issue on cross-modal alignment learning . Currently , reward shaping has been well investigated in VLN based on coarse-grained data to align instructions and trajectories globally , but pays little attention to enhance local cross-modal alignment . The goal-oriented reward simply [ 6 ] uses the arrival signal and the reduced distance to the goal point as the reward , guiding agent to find the global goal points . The fidelity-oriented rewards [ 16 , 17 ] help agent improve the similarity between the predicted trajectory and demonstration trajectory globally by taking intermediate points as external supervision . However , some of the points in the trajectory have more detailed descriptions and are more helpful to navigation process , i.e. , they can assist agent in knowing the visual scenes around these specific points in the trajectories could be better aligned with certain parts of the instructions than the trivial points , where the alignment is in a local to local manner . These points are called critical points in this paper . Thus , an emphasis on these critical points in the reward shaping for better local cross-modal alignment is essential . In this paper , we address the challenges above from the perspective of fine-grain . Firstly , based on the English Guide part of RxR ( en-RxR ) , we introduce a fine-grained dataset Landmark-RxR , which is human-annotated , landmark-based , fine-grained and currently the largest scale . With the groundtruth fine-grained data , experiments demonstrate that agent generalizes better to unseen environments and instructions with domain gap . This indicates that the fine-grained data help agent to align textual and visual modalities better . Secondly , we propose two kinds of focal-oriented rewards that encourage local alignment between instructions and critical points . Since the landmarks in Landmark-RxR naturally meet the requirements of critical points , we just sampled critical points from the landmark set in Landmark-RxR . The focal-oriented rewards outperform the commonly used goal-oriented reward and fidelity-oriented reward . We also propose the re-initialization mechanism to fully evaluate the navigation process in a way that is insensitive to difficult points , which can cause the deviation from the correct trajectory . With the fine-grained data and focal-oriented rewards , our agent shows superior navigation performance on Landmark-RxR , en-RxR and R2R . 2 Related Work . Vision-and-Language Navigation Anderson et al . [ 4 ] first introduced the simulated , photorealistic VLN task with a benchmark Room-to-Room ( R2R ) and several sequence-to-sequence baselines . Fried et al . [ 5 ] utilized the speaker-driven data and panoramic action space to improve navigation performance . Wang et al . [ 18 ] proposed a planned-ahead hybrid reinforcement learning model with model-free and model-based reinforcement learning for better generalization . Tan et al . [ 19 ] first trained agent with mixed imitation learning and reinforcement learning and used unseen triplets generated by environment dropout method for fine-tuning then . Jain et al . [ 16 ] and Ilharco et al . [ 17 ] proposed the fidelity-oriented rewards to ensure the similarity between predicted trajectories and demonstration trajectories . Kurita et al . [ 20 ] combined generative and discriminative policy to enhance the navigation ability from different aspects . An et al . [ 21 ] fused neighbor visual contexts both globally and locally to get visual features with richer semantics . Huang et al . [ 22 ] defined two in-domain auxiliary tasks for representation learning which benefit the downstream navigation task . Li et al . [ 7 ] leveraged the large-scale pretrained language models BERT [ 23 ] and GPT [ 24 ] for powerful text representation . Hao et al . [ 10 ] pretrained and finetuned generic transformer-based model in VLN environment and the pretrained model generalizes well to other two navigation tasks . Majumdar et al . [ 11 ] demonstrated that pretrained on image-text pairs from the web , the transformerbased model can perform on VLN task well after fine-tuning . Hong et al . [ 12 ] proposed a Recurrent Vision-and-Language BERT to process time-dependent tasks like VLN efficiently . Fine-Grained VLN Wang et al . [ 6 ] designed a cross-modal grounding module to infer which part of the instructions to focus on . Ma et al . [ 8 ] introduced a self-monitoring agent with a visual-textual co-grounding module to locate the required part in instructions and a progress monitor to reflect navigation progress . Qi et al . [ 25 ] decoupled instructions into object-aware and action-aware parts for more accurate action prediction . Zhu et al . [ 14 ] adopted a two-stage method for navigation . The given instruction is first split into several sub-instructions named baby-steps according to heuristic rules and then the baby-steps which are much easier to navigate are fed to the agent one by one during validation . Similar to [ 14 ] , Hong et al . [ 13 ] first segmented instruction into sub-instructions using a heuristic method . Then the agent navigates following these easy sub-instructions sequentially with a shifting module inferring whether current sub-instruction has been completed . Ku et al . [ 15 ] explored utilizing spatially-temporally aligned annotations to supervise the textual and visual attention weights but got very few performance improvements . Different from these works , we propose a human-annotated , fine-grained dataset Landmark-RxR to alleviate the weak cross-modal alignment supervision from the coarse-grained data . Our experiments is designed to demonstrate the supervision from fine-grained and coarse-grained data can complement each other to improve the cross-modal alignment ability of the model itself . In addition , two kinds of focal-oriented rewards using fine-grained supervision signals are proposed to enhance the local cross-modal alignment by paying more attention to critical points which benefit the navigation process more . 3 Landmark-RxR Dataset . Our Landmark-RxR is built based on the English Guide split of RxR ( en-RxR ) . It contains subinstruction and sub-trajectory pairs ( sub pairs ) split from instructions in en-RxR . To facilitate the collection of sub pairs , we develop a 3D web-based collection tool . It allows the annotator to mark the landmarks , namely the end points of sub-trajectories , in trajectories and split out the corresponding sub-instructions . During annotation , the annotators can move between the discrete points or change heading and evaluation continuously by mouse click , like what the agent will do in the VLN task . In the following , we will describe the data collection and dataset analysis of Landmark-RxR . 3.1 Data Collection . The annotators are asked to read the complete instructions and then explore the 3D environments following corresponding trajectories . If landmarks are found , annotators need to mark them and split out the corresponding sub-instructions by interacting with the collection tool . As illustrated in Figure 1 ( a ) , the complete instruction and complete trajectory pair ( complete pair ) are sampled from en-RxR . In Figure 1 ( b ) , three landmarks in the complete trajectory are marked and the corresponding sub-instructions are also split out by annotators . These three sub pairs will be included in our Landmark-RxR dataset . Five principles are proposed to ensure the annotation process more standardized . Matching Principle : The sub-trajectory should match the sub-instruction accurately . Independent Meaning Principle : Each sub-instruction expresses an independent meaning . Position Change Principle : The end point of each sub-trajectory should differ from the start point . Clear End Point Principle : The end point should be described clearly in a sub-instruction . For example , the sub-instruction go toward the door has a clear description of end point door , but no end point is described in the sub-instruction go forward . Minimum Granularity Principle : The sub-instruction and sub-trajectory pair should be the minimum granularity which means it can not be split anymore . After an annotator finishes the annotations , another annotator will verify the annotations again and modify the inaccurate part to ensure the annotations satisfy the five principles . For some mistakes in RxR , we also correct them in Landmark-RxR . Totally 30 annotators participated in the annotation task , contributing about 2,700 hours . 3.2 Dataset Analysis . Table 1 gives statistics on R2R , RxR , en-RxR and Landmark-RxR . The total number of subinstructions from Landmark-RxR is 166,740 , which contains 133,602 sub-instructions in train split , 13,591 sub-instructions in validation seen split , and 19,547 sub-instructions in validation unseen split . The average number of edges in a sub-trajectory is 1.6 with 21 words in the corresponding sub-instruction on average . The number of sub-trajectories contained in Landmark-RxR is 46,645 . Landmark-RxR has the largest scale of instructions and trajectories and the minimum granularity among current human-annotated datasets . The fine-grained annotations are essential to effective cross-modal alignment learning and more accurate evaluation . In addition , because of the minimum granularity principle , the dataset can be easily expanded to a larger scale with different granularities by recombining the sub pairs . 4 Re-initialization Mechanism and Evaluation Metrics . 4.1 Re-initialization Mechanism . An agent needs to navigate following the instructions strictly . So it is essential to evaluate the fidelity between predicted trajectories and the instructions exactly . Current metrics like Coverage weighted by Length Score ( CLS ) [ 16 ] and the normalized Dynamic Time Warping ( nDTW ) [ 17 ] try to obtain the fidelity by computing the similarity between predicted trajectory and the demonstration 1 Note that the number of public instructions in RxR test split is 12,469. trajectory . However , the nDTW and CLS scores are sensitive to the positions of difficult points , and have a deviation as a result when evaluating the navigation ability . As illustrated in Figure 2 , the agent is initialized at point A and ready to follow the given instruction navigating to point B . Then during navigation , the agent deviates from demonstration trajectory seriously after encountering the difficult point and can only get a low fidelity score . The difficult point indicates the point where next action is hard to choose because of the difficult visual scene , e.g. , cluttered and similar objects in Figure 2 , or ambiguous expressions in an instruction . However , for the same agent navigating from point B to point A , it is easy to finish the majority of the trajectory and get a higher fidelity score because the difficult point is just near the end . The distributions of difficult points in trajectory significantly impact current fidelity metrics , especially for the long trajectories . Based on the landmark annotations in Landmark-RxR , we introduce the re-initialization mechanism to alleviate the influence of difficult points sensitivity of the fidelity metrics . We define a trajectory as T = [ ST1 , ST2 , ... , ST|T | ] , where STi = [ Pi1 , Pi2 , ... , Li ] , and corresponding instruction is defined as I = [ SI1 , SI2 , ... , SI|I| ] . Pij means the jth point in the ith sub-trajectory STi . Li means the ith landmark in the trajectory , in other words , the end point of the ith sub-trajectory and SIi means the ith sub-instruction . As illustrated in Figure 3 , if the agent reaches within 3 meters , which is a fixed threshold for success metric defined in [ 4 ] , of the landmark Li ( i < |T | ) , or the agent can stop within 3 meters of the landmark L|T | in π ( π is set to 10 in this paper ) steps from the last landmark , the navigation to landmark Li is successful . The failure is defined that the agent fails to get within 3 meters of the landmark Li ( i < |T | ) in π steps from the last landmark , or stops 3 meters away from the landmark L|T | . If the agent fails in navigating to landmark Li ( i < |T | ) , we re-initialize the agent at landmark Li as the new start point , and reset the given instruction as Ireset = [ SIi+1 , ... , SI|I| ] . Since the re-initialization mechanism can correct serious deviations in time , for a complete trajectory , the averaged fidelity score of its sub-trajectories will be much less insensitive to the difficult points and can better reflect the navigation ability no matter where the difficult points locate .

Summary: The paper tackles the problem of Vision and Language Navigation (VLN). The main contribution of the paper is a dataset called Landmark RxR. The instructions in the RxR dataset have been broken down into smaller segments by human annotators, where each segment corresponds to navigation to a landmark. A landmark is a point at which it is difficult to make a decision (due to ambiguity, clutter, etc). The paper proposes a "re-initialization" strategy to better evaluate VLN agents. Two types of reward functions based on the deviation from the landmarks are defined. The paper reports results on R2R, en-RxR and Landmark RxR datasets.