Loc4Plan: Locating Before Planning for Outdoor Vision and Language Navigation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 13:20:02

Abstract: Vision and Language Navigation (VLN) is a challenging task that requires agents to understand instructions and navigate to the destination in a visual environment. One of the key challenges in outdoor VLN is keeping track of which part of the instruction was completed. To alleviate this problem, previous works mainly focus on grounding the natural language to the visual input, but neglecting the crucial role of the agent’s spatial position information in the grounding process. In this work, we first explore the substantial effect of spatial position locating on the grounding of outdoor VLN, drawing inspiration from human navigation. In real-world navigation scenarios, before planning a path to the destination, humans typically need to figure out their current location. This observation underscores the pivotal role of spatial localization in the navigation process. In this work, we introduce a novel framework, Locating before Planning (Loc4Plan), designed to incorporate spatial perception for action planning in outdoor VLN tasks. The main idea behind Loc4Plan is to perform the spatial localization before planning a decision action based on corresponding guidance, which comprises a block-aware spatial locating (BAL) module and a spatial-aware action planning (SAP) module. Specifically, to help the agent perceive its spatial location in the environment, we propose to learn a position predictor that measures how far the agent is from the next intersection for reflecting its position, which is achieved by the BAL module. After the locating process, we propose the SAP module to incorporate spatial information to ground the corresponding guidance and enhance the precision of action planning. Extensive experiments on the Touchdown and map2seq datasets show that the proposed Loc4Plan outperforms the SOTA methods.

Main paper: 
 related works 2.1 vision-and-language navigation: 
 The Vision-and-Language Navigation (VLN) task requires the agent to navigate in a 3D simulated environment toward the goal location based on instructions and egocentric observations. Various work has conducted in the indoor VLN task, including exploring feature representation [6, 11, 20, 22] , modal interaction [1, 8, 30, 33] , designing diverse mechanisms of reinforcement learning [1, 35, 36] , scenic map building [5, 7, 18, 26] , etc. Recently, research in the outdoor scenario has also begun to emerge and develop, which is based on real-world urban environments comprising actual street layouts and panoramic images [4] . Compared with indoor VLN, outdoor VLN contains a larger vocabulary and longer navigation instructions than indoor corpora [4] , posing a greater challenge for agents to make cross-modal alignment between navigation state and longspan instruction. To solve this challenge, numerous methods have been proposed. Most of prior works [3, 4, 21, 25, 37] directly encode the observation, trajectory, and instructions in an LSTM-based model. The L2STOP [37] method differentiates STOP and other actions to boost the localization of stop action. GA [3] uses gated attention to compute a fused representation of instructions and images to predict actions. ORAR [25] adds junction-type embedding and a heading delta to improve the generalization of the agent in unseen scenarios. Additionally, many works [2, 15, 39] adopt the Transformer architecture for navigation, where VLN-Transformer [39] is the first transformer-based model of outdoor VLN. PM-VLN [2] method introduces the pre-training of priority map to achieve the temporal sequence alignment. However, previous studies have overlooked the crucial importance of spatial positioning in navigation which significantly affects the agent's ability to interpret and execute navigation instructions accurately. In this study, we underscore the importance of initial spatial localization prior to planning decision actions. We introduce a novel framework called Locating before Planning (Loc4Plan) to tackle challenges in outdoor VLN tasks.In vision-and-language navigation (VLN) tasks where the agent is provided with instructions detailing the entire navigation route, it is crucial to ground the linguistic guidance to the relevant segments pertaining to the next action. The overall framework of our proposed Loc4Plan. The image and text encoder extract features of visual observation and instructions, respectively. Initially, the block-aware spatial locating (BAL) serves to leverage the visual representation and spatial information (i.e. junction-type embedding, heading delta) of trajectory to locate the agent's position relative to the current block. Then we identify the corresponding guidance that the agent needs to follow by associating spatial-aware state representation with provided instructions in a hierarchical manner, ranging from sentence-level to token-level granularity. Finally, the agent further incorporates spatial locating information into action decision planning.has emerged as a core challenge in VLN research. This challenge is not unique to VLN and represents an important issue that needs to be addressed for other multimodal tasks as well, as explored by various works [14, 16, 17, 27] . To achieve textual grounding, a main line of research [25, 28, 37, 39] employs cross-modal grounding over individual words between the natural language instruction and the environmental scene. And some of works [31, 39] implicitly make multi-modal alignment in an attention-based component of a transformer. Other works [2, 13, 15, 20, 33] focus on improving the representations of vision and language modalities, proposing auxiliary tasks to enhance grounding. However, we find that relying solely on word-level localization for planning is insufficient, particularly when instructions are extensive. In this work, we align the observation and the instruction in a hierarchical manner, ranging from coarse to fine granularity. 
locating before planning 3.1 priliminary: 
 Problem Formulation. Given a natural language instruction, a Vision-and-Language Navigation (VLN) agent is tasked with navigating from the starting position to the destination by following the guidance provided in the instruction. All navigation paths are instantiated on the directed graph environment G = (𝑉 , 𝐸) with nodes 𝑣 ∈ 𝑉 and labeled edges (𝑢, 𝑣) ∈ 𝐸. Each node is associated with a 360 • panorama image and each edge is labeled with an angle 𝛼 𝑢,𝑣 . At each timestep 𝑡, the agent's state 𝑠 𝑡 ∈ S is defined by 𝑠 𝑡 = (𝑣 𝑡 , 𝛼 (𝑣 𝑡 -1 ,𝑣 𝑡 ) ), where 𝑣 𝑡 is the node at timestep 𝑡, and 𝛼 (𝑣 𝑡 -1 ,𝑣 𝑡 ) is the heading angle from previous state's node 𝑣 𝑡 -1 to the current state's node 𝑣 𝑡 . Given the current navigation state 𝑠 𝑡 , the agent receives the corresponding visual observation from the environment. Based on the instruction information I and observed visual information x 𝑡 , the agent infers the action 𝑎 𝑡 from the candidate actions list of [𝐹𝑂𝑅𝑊 𝐴𝑅𝐷, 𝐿𝐸𝐹𝑇 , 𝑅𝐼𝐺𝐻𝑇 , 𝑆𝑇𝑂𝑃], and then executes the chosen action to update the next state 𝑠 𝑡 +1 . The agent must produce a sequence of state-action pairs [(𝑠 1 , 𝑎 1 ), (𝑠 2 , 𝑎 2 ), ..., (𝑠 𝑛 , 𝑎 𝑛 )], where 𝑎 𝑛 = 𝑆𝑇𝑂𝑃, to reach the goal location. Block Definition. We define a "block" as the region within the environment graph bounded by adjacent intersections, where traversal is constrained to a single path without the possibility of intersection crossings. An example of a block is shown in Figure .1. block-aware spatial locating (BAL) module and a spatial-aware action planning (SAP) module. Specifically, the BAL module localizes the spatial position on the block level, which enables the agent to be aware of its relative location position within the current block. After this locating process, the SAP module leverages the spatial information (obtained in the BAL module) to identify the corresponding guidance that the agent needs to follow and make the action prediction. Further details of our Loc4Plan framework are presented below.In human navigation, the initial stage entails identifying the user's current location before providing a routed path to the desired endpoint, which requires prior knowledge of the navigated region's topology. Drawing inspiration from human navigation, we integrate spatial localization into outdoor VLN tasks. However, navigating in outdoor VLN scenarios presents the challenge of maneuvering through unknown terrains where the complete environmental layout remains undisclosed. To address this, we introduce the blockaware spatial locating (BAL) module tailored for outdoor VLN, which establishes spatial positioning at a block-level granularity. We define blocks using intersections as demarcations, guaranteeing that the space between adjacent intersections pertains to the same block. Through the learning process facilitated by the BAL module, the agent gains an awareness of its current location, indicative of its relative position within the observation field at the block level. Formally, consistent with the prior work [25] , at timestep 𝑡 on node 𝑣 𝑡 , we incorporate the action embedding a 𝑡 -1 at timestep 𝑡 -1, visual representation x 𝑡 of current observation, the turning angle of the current node 𝑔 𝑐 𝑡 , and the junction type embedding n 𝑡 into a sequence modeling function to obtain the representation of the current state. Besides, we introduce the long-term turning angle 𝑔 𝑙 𝑡 as a novel input feature to model the long-range direction information. Then the state representation of the current node can be obtained by:EQUATIONwhere 𝜙 1 is a feature encoder, ⊕ denotes the concatenation operation, o 𝑡 -1 is the previous state of time step 𝑡 -1, n 𝑡 is the embedding indicating the number of outgoing edges of the node at timestep 𝑡, and𝑔 𝑐 𝑡 = 𝑁𝑜𝑟𝑚(𝛼 (𝑣 𝑡 -1 ,𝑣 𝑡 )) is a value in (-1, 1] that encodes the turning angle relative to the previous timestep. The long-term turning angle 𝑔 𝑙 𝑡 is computed by:EQUATIONindicates the turning angle across consecutive steps, where 𝐾 denotes the number of steps involved in this calculation. When the special case that 𝑡 -𝑘 < 0 occurs, we set 𝑔 𝑐 𝑡 -𝑘 = 0. The concept of the long-term turning angle is introduced to aggregate turning angles across multiple consecutive steps, recognizing the process of turning left or right typically spans several individual actions.The obtained node state presentation o 𝑡 has encoded information of previous states and observation visual and topological (e.g. junction-type embedding and heading delta) information of the current timestep. Further, we expect the agent can be aware of the spatial position where they have achieved. To address this, we locate the relative position of the agent with the observation field under the block level. Specifically, we first obtain the spatial-aware state representation of the node 𝑣 𝑡 by:EQUATIONwhere W 𝑏 ∈ R 𝑑 ×𝑑 is a linear layer, and ⊤ indicates the transpose operation. We assume that the o 𝑝 𝑡 accumulates relevant information beneficial for spatial localization. Optimization of BAL. To ensure that spatial perception representation o 𝑝 𝑡 can be aware of the information related to navigation progress within a block, we feed forward o 𝑝 𝑡 to the spatial predictor to predict a navigation process score within current block by:EQUATIONwhereW 𝑝 ∈ R 𝑑 ×1 is a linear layer. 𝑒 𝑝 𝑡 ∈ [0, 1)indicates the agent's spatial position relative to the current block, where 0 indicates the start node of the block and 1 indicates the end. Block process score 𝑒 𝑝 𝑡 is supervised by the MSE loss:EQUATION)where 𝑇 is the number of time steps to complete the entire navigation. And ẽ𝑝 𝑡 is the ground-truth block progress score at timestep 𝑡, which is calculated based on the number of nodes within the current block:EQUATIONwhere 𝑁indicates the number of steps to "forward" to the next intersection node. And 𝑁 𝑎𝑙𝑙 𝑡 indicates the number of nodes in the current block (excluding the starting node). For example, when the agent is at the position shown in Figure 1 , 𝑁 𝑠𝑡𝑒𝑝 𝑡 = 3 and 𝑁 𝑎𝑙𝑙 𝑡 = 5. Discussion. With the learning of the BAL module, we assume the spatial-aware state representation has accumulated the information related to the spatial position. This suggests that the agent develops a self-awareness of location ability, just like humans.The self-awareness of location ability developed in the BAL is beneficial for textual grounding, thereby facilitating further action plannin. Therefore, we introduce the spatial-aware action planning (SAP) module, which incorporates spatial locating information (i.e., the spatial-aware state representation obtained in the BAL module) to associate the corresponding guidance and enhance the precision of action planning. In detail, we first propose a hierarchical semantic association (HSA) submodule to identify the corresponding guidance that the agent needs to follow by associating spatial-aware state representation with provided instructions in a hierarchical manner, ranging from sentence-level to token-level granularity. We have found that relying solely on word-level localization is inadequate, especially when instructions are intricate and extensive. Consequently, we initially align spatial perception observations with instructions at the sentence level, leveraging the broader contextual understanding and richer semantics afforded by sentences. Subsequently, we devise a fine-grained mask derived from this sentence-level alignment to selectively filter out irrelevant information within the instructions, thereby identifying the corresponding guidance for the current step. Based on this, the agent further incorporates spatial locating information into action decision planning. Hierarchical Semantic Association. We first identify the corresponding guidance that the agent needs to follow by associating spatial-aware state representation with provided instructions in a hierarchical manner, ranging from coarse to fine granularity.We start with associating the visual and textual in a sentence level. Specifically, given natural-language instructions I = {w 1 , ..., w 𝐿 } with 𝐿 words, we use a period delimiter to split the instruction into multiple sentences and obtain the sentence-level embeddings I 𝑠 = {s 0 , .., s 𝑁 𝑠 } ∈ R 𝑁 𝑠 ×𝑑 𝑡 by average pooling the token embedding of each sentence, where 𝑁 𝑠 is the number of sentences. The sentence-level embedding I 𝑠 is then queried by the spatial-aware state representation o 𝑝 𝑡 to model the sentence-level contextual feature Î𝑠 ∈ R 𝑑 by adopting multi-head cross-attention [34] , which can be denoted as follows:EQUATIONwhere o 𝑝 𝑡 serves as the query and I 𝑠 is utilized to calculate the keys and values. The sentence-level contextual feature contains coarselygrained guidance relevant to the current state. Then, we utilize the sentence-level contextual feature Î𝑠 as long as the sentence features I 𝑠 to calculate the relevance scores between the sentences and the current state in the relevance predictor:EQUATIONwhere Subsequently, we devise a fine-grained mask derived from this coarsely-grained relevance scores, which is utilized to filter out irrelevant information in the token embedding for fine-grained instruction attention. Specifically, for the 𝑖-th sentence, we obtain its corresponding token level mask m 𝑡,𝑖 = [𝑟 𝑡,𝑖 , ..., 𝑟 𝑡,𝑖 ] by repeating the sentence-level relevance score 𝑟 𝑡,𝑖 with 𝐿 𝑠 𝑖 times, where 𝐿 𝑠 𝑖 denotes the length of the 𝑖-th sentence. Then the complete mask of the instruction can be denoted as M 𝑡 = [m 𝑡,𝑖 , ..., m 𝑡,𝑁 𝑠 ]. We utilize the token mask M 𝑡 to filter out the irrelevant information in the instruction by:W ∈ R 𝑑 ×1 , W ŝ ∈ R 𝑑 𝑡 ×𝑑 , W 𝑠 ∈ R 𝑑 𝑡 ×𝑑 ,EQUATIONThen the masked instruction I 𝑚 𝑡 is used to obtain the attended instruction feature with the spatial-aware state representation o 𝑝 𝑡 as query:EQUATIONThen we utilize the I 𝑎𝑡𝑡𝑛 𝑡 as query to obtain the attended visual feature:x EQUATIONEQUATION)where t is the embedded timestep 𝑡. z 𝑡 contains cross-modal information for action planning, which is then integrated with turning angles 𝑔 𝑎 𝑡 of each action to predict action scores for the next step:EQUATIONwhere W 𝑎 ∈ R 𝑑 ×1 is a linear layer. We define 𝑔 𝑎 𝑡 to represent the turning angles obtained if the agent executes action 𝑎𝑐𝑡𝑖𝑜𝑛 ∈ [𝐹𝑂𝑅𝑊 𝐴𝑅𝐷, 𝐿𝐸𝐹𝑇 , 𝑅𝐼𝐺𝐻𝑇 , 𝑆𝑇𝑂𝑃] at timestep 𝑡. After the calculation of action scores for all candidate actions, the agent will execute the corresponding action of the maximum score and update the navigation state. Optimization of SAP. The optimization of SAP comprises the learning of the hierarchical semantic association and the action planning loss. For the hierarchical semantic association, the sentence level relevance score r 𝑡 (Eq. ( 8)) is learned under the supervision of the sentence-level corresponding labels, which indicates the alignment of the sentences and nodes along the trajectory. Specifically, by referring to the multi-label classification method [32] , a binary cross-entropy loss is used to supervise each sentence: (14) where r𝑡,𝑖 is a binary label indicating whether the agent needs to attend to the 𝑖-th sentence when navigating to the node 𝑣 𝑡 . This label is generated from an economy-efficient method of template matching [12, 38] . 𝛾 𝑏 is the weight related to the confidence of the corresponding pseudo-label, which is used to control the influence of 𝐿 HSA on training. More details can be found in Supplementary.𝐿 𝐻𝑆𝐴 = -𝛾 𝑏 𝑇 ∑︁ 𝑡 𝑁 𝑠 ∑︁ 𝑖 ( r𝑡,𝑖 𝑙𝑜𝑔(𝑟 𝑡,𝑖 ) + (1 -r𝑡,𝑖 )𝑙𝑜𝑔(1 -𝑟 𝑡,𝑖 )),The action planning loss 𝐿 𝐴𝑃 is a regular cross-entropy loss:EQUATIONwhere c𝑎 𝑖 𝑡 denotes the binary ground-truth label indicating whether execute action 𝑎 𝑖 at time step 𝑡, and 𝑖 indicates the action index of [𝐹𝑂𝑅𝑊 𝐴𝑅𝐷, 𝐿𝐸𝐹𝑇 , 𝑅𝐼𝐺𝐻𝑇 , 𝑆𝑇𝑂𝑃]. Discussion. By involving the spatial location information into the learning of the SAP module, we can ground the instruction guidance that the agent should adhere to in the current stage and facilitate action prediction.As mentioned above, our proposed Loc4Plan is optimized with three losses, i.e., 𝐿 𝐴𝑃 , 𝐿 𝐵𝐴𝐿 , and 𝐿 𝐻𝑆𝐴 . The overall loss function can be written as:EQUATION)During training, we use Teacher-Forcing [19] strategy to achieve stable optimization. where the samples in train and test sets are mixed in terms of their geographic locations (i.e. seen scenarios). The unseen scenarios are then conducted by [25] that makes geographic separation of the training and testing area. The map2seq dataset [24] was created for the task of navigation instructions generation and introduced in outdoor VLN [25] . It is also constructed in New York environment, containing 7,672 samples with both seen and unseen scenarios. Evaluation Metrics. Following [2, 4, 15] , we adopt Task Completion (TC), Shortest-path Distance (SPD), and Success weighted by Edit Distance (SED) to evaluate the models. Specifically, TC represents the percentage of successful agent navigation. SPD measures the shortest path distance from the node the agent stopped to the goal node within the environment graph. SED represents the task completion weighted by the Levenshtein edit distance between prediction and ground-truth trajectories. Implementation Details. We use a bidirectional LSTM [9] as the text encoder to extract token-level embedding. A ResNet pretrained on ImageNet [23] is utilized as image encoder to extract visual representation 𝑥 𝑡 . Following [25, 29, 37] , we implement 𝜙 1 (.) and 𝜙 2 (.) as LSTMs [10] . We set the dimension of token embedding 𝑑 𝑡 = 512 and the hidden dimension of model 𝑑 = 256. The dimension of timestep embedding is 32, and the size of action embedding and junction type embedding is 16. For the specific process of generating sentence-level label r𝑡,𝑖 (Eq. ( 14)) and other training details, please refer to the Supplementary. 
comparisons with sota vln methods: 
 In this section, we compare our proposed locating before planning (Loc4Plan) framework with previous state-of-the-art (SOTA) approaches [2-4, 21, 25, 39] . Except for ORAR [25] , these methods only report results for the seen scenario on Touchdown. To enable a comprehensive comparison, we evaluate methods with published code, supplementing results on the map2seq dataset and the unseen scenario of Touchdown. The results, as illustrated in Table 1 , indicate that our method exhibits superior performance on both Touchdown and map2seq datasets. Remarkably, on the challenging test unseen split, our model showcases significant improvements over ORAR [25] by 3.3% on the Touchdown dataset and 4.8% on the map2seq dataset. This demonstrates the effectiveness of our Loc4Plan framework. In the test seen split of Touchdown, our Loc4Plan achieves performance comparable to that of PM-VLN, despite PM-VLN relying on additional ground-truth trajectory information for trajectory planning, which is not necessary for our Loc4Plan. Furthermore, our agent significantly enhances the SPD score by 12.3% compared to PM-VLN, underscoring its superior ability to adhere to instructions while navigating accurately. Furthermore, we investigate the performance of different agents under varying trajectory complexities and instruction lengths, as illustrated in Figure 3 . Specifically, we utilize the SED score to evaluate the navigation capabilities of these agents on the test set of the Touchdown dataset. Trajectory complexity is quantified by the number of intersections, while instruction length serves as a proxy for the level of difficulty in comprehension. As depicted, the Loc4Plan agent exhibits significant improvements, particularly in scenarios involving longer instructions and complex trajectories, demonstrating our ability to handle challenging long-term navigation tasks.In this section, we conduct detailed ablation experiments to evaluate the effectiveness of each component proposed in our Loc4Plan, including the block-aware spatial locating (BAL) module and spatialaware action planning (SAP) module.Effectiveness of the BAL and SAP module. We first investigate the effectiveness of our overall design in Table 2 . Row #1 presents the performance derived from our baseline approach. In this configuration, the agent's state representation relies exclusively on factors such as junction type, current heading angle, and visual cues. Instruction representation is attained through the average pooling of token embeddings. The results presented in Table 2 indicate that both the BAL and SAP modules contribute significantly to performance improvement. Specifically, the results delineated in rows #1 and #2 exhibit a promising enhancement in performance with the integration of the BAL module compared to the baseline. Furthermore, the inclusion of the SAP module, as evidenced in rows #1 and #3, yields notable improvements, enhancing the TC metric by 6.3% and 2.5% on map2seq and Touchdown, respectively. Investigation of the BAL module. The BAL module is proposed to establish spatial positioning, which indicates its relative position within the observation field at the block level. Here, we investigate the components within the BAL module in Table 3 . In addition to incorporating the current turning angle 𝑔 𝑐 𝑡 , we innovatively introduced the long-term turning angle 𝑔 𝑙 𝑡 (Eq. ( 1)) for spatial perception. By separately comparing #1, #2 with the full model (#5), we can find that adding both the current and long-term turning angles results in improvements, validating their effectiveness. By comparing the results without supervision of the block process score (Eq. ( 5)) (#3) and with supervision (#5), we find that the inclusion of this supervision leads to a significant improvement of 4.3% and 2.6% in the TC metric on the map2seq and Touchdown datasets, respectively. 2)) in our model. The corresponding results are presented in Table4. Specifically, 𝐾 = 0 implies that the model only utilizes the turning angle of the current node, while increasing values of 𝐾 expand the model's perception range to encompass a longer history of turning angles. The result shows that the agent achieves optimal performance when 𝐾 = 3. This phenomenon indicates that a proper perception length of long-term turning angle is mandatory to achieve better performance. Investigation of the SAP module. To investigate the impact of spatial information obtained from the BAL module on the action planning stage, we first show the results without using spatial information in the SAP module in table 5. Specifically, "W/o spatial info" means using the state representation o 𝑡 to replace the spatial-aware feature o 𝑝 𝑡 . Clearly, with the absence of spatial locating information, the agent's performance decreases on both datasets. Moreover, to investigate the impact of the components involved in hierarchical semantic association, we conducted a series of ablation experiments, with the results presented in Table 6 . Specifically, the label "token" and "sentence" refer to the multi-head attention on token level (Eq. ( 14)) and sentence level (Eq. ( 7)). "L HSA " denotes the supervision on the sentence relevance score (Eq. ( 14)). The results reveal that using both sentence and token-level attention yields significantly better results than using either one alone, demonstrating their complementary roles in facilitating semantic association. Interestingly, the introduction of explicit supervision on sentence attention did not have as significant an impact as anticipated. Even without such supervision (#3), the model was able to achieve competitive navigation performance.To intuitively show the effect of our method in spatial locating, a qualitative result is shown in Figure 4 that visualizes an example of the block progress score. It can be seen that our agent can track the progress of navigation across different blocks, which proves the ability in spatial localization. Moreover, Figure 5 presents a visualization of the agent's attention dynamics on the instruction sentences progress over multiple time steps. As shown in the heatmap of sentence relevance scores, our agent is able to focus on the relevant sentences within the instruction at different steps, especially exhibiting precise decision-making at several key steps. 
conclusion: 
 In this work, drawing the inspiration from human navigation, we propose a "Locating before Planning (Loc4Plan)" framework. For the first time, this framework enables agents to locate their spatial position before planning a decision action based on corresponding guidance. The Loc4Plan framework comprises a block-aware spatial locating (BAL) module and a spatial-aware action planning (SAP) module. The BAL module is proposed to localize the spatial position on the block level, which enables the agent to be aware of its relative location position within the current block. The self-awareness of location ability developed in the BAL is beneficial for textual grounding, thereby facilitating further action planning. Therefore, we propose the SAP module for the planning process, which associates spatial-aware state representation with provided instructions in a hierarchical manner, promoting a comprehensive understanding of the provided instructions and make the action prediction.Benefiting from the ahead localization to the agent's position and comprehensive understanding of the provided instructions, our Loc4Plan achieves the new state-of-the-art for Touchdown and map2seq dataset on both seen and unseen scenarios. 
: 
 Where am I now? The illustration of navigation process of our locating before planning approach. During the locating phase, the agent locates its relative spatial position in the current block. In the planning phrase, the agent associates the corresponding guidance to follow and makes an action decision to take (i.e., FORWARD).navigate to the destination in a visual environment. The agent is embodied in the environment and receives complete navigation instructions consisting of multiple sub-instructions to describe how to reach the destination step-by-step. Based on the instructions, the observed surroundings, and the current trajectory, the agent decides its next action. Executing this action changes the position and/or heading of the agent within the environment, and eventually, the agent follows the described route and stops at the desired goal location. Arguably, one of the key challenges in outdoor VLN is keeping track of which part of the instruction was completed. To alleviate this problem, various methods [25, 28, 37, 39] have been proposed. These methods mainly focus on grounding the natural language to the visual input, while neglecting the crucial role of the agent's spatial position information in the grounding process.In this work, we argue that it is crucial for an agent to first determine its spatial position within the visual environment before grounding the appropriate guidance to follow in the outdoor VLN task. Consider a real-world navigation scenario for human, such as when a tourist seeks directions from a native in an unfamiliar area. The native typically begins by ascertaining the tourist's current spatial position before offering a route to the desired destination. This observation underscores the pivotal role of spatial localization in the navigation process. Unfortunately, previous studies have overlooked the crucial importance of this spatial positioning stage, which significantly affects the agent's ability to interpret and execute navigation instructions accurately. Although the work [25] encodes some topological information, such as junction-type embedding and heading delta, to enhance agent generalization, it does not extensively explore the crucial role of spatial localization.Drawing inspiration from human navigation, we first explore the effect of spatial position locating on the textual grounding of the outdoor VLN task. Generally, in human navigation, spatial positioning relies on prior knowledge of the navigated region's topology. However, outdoor VLN tasks usually require agents to navigate unseen environments, where the comprehensive environmental topology is unavailable during inference. Meanwhile, the visual observation perception of the agent is limited to a local region. To help the agent perceive its spatial location in the environment, we propose to learn a spatial predictor that measures how far the agent is from the next intersection for reflecting its position, which is achieved by a block-aware spatial locating (BAL) module. In our modeling, a "block" is defined as the area between adjacent intersections, as shown in Figure 1 . In other words, each block represents a straight street segmented by two adjacent intersections. The BAL module enables the agent to determine its position at a finer granularity (block-level) rather than an intricate global-level, thereby facilitating the subsequent planning process.The self-awareness of location ability developed in the BAL is beneficial for textual grounding, thereby facilitating further action planning. Therefore, we introduce the spatial-aware action planning (SAP) module, which incorporates the spatial locating information to associate the corresponding guidance and enhance the precision of action planning. In detail, we first identify the corresponding guidance that the agent needs to follow by associating spatial-aware state representation (obtained in BAL) with provided instructions in a hierarchical manner, ranging from sentence-level to token-level granularity. Specifically, the sentence-level association leverages the broader contextual understanding and richer semantics afforded by sentences. Subsequently, we devise a fine-grained mask derived from this sentence-level alignment to selectively filter out irrelevant information embedded in the token sequence. Compared to relying solely on word-level localization, our hierarchical semantic association provides a comprehensive understanding of the instructions, especially the extensive and intricate ones. Based on this identified corresponding guidance, the agent further incorporates spatial locating information into action decision planning.Based on the above two modules, we construct a novel learning framework for addressing outdoor VLN tasks, named Locating before Planning (Loc4Plan), which enables agents to develop an ability of location-awareness like humans by first identifying the initial spatial localization before deciding where to go. Benefiting from the ahead localization to the agent's position and comprehensive understanding of the provided instructions, our Loc4Plan achieves the new state-of-the-art for Touchdown and map2seq dataset on seen and unseen scenarios, which outperforms ORAR framework [25] 3.3% and 4.8% of TC in test unseen scenario on the Touchdown [4] and map2seq [25] datasets, respectively.In summary, the main contributions of this paper are as follows:• We introduce a Locating before Planning (Loc4Plan) learning framework to address outdoor VLN tasks. Loc4Plan mimics the human navigation process by first determining the current location of the agent before making the next planning decision. • To seek the location-awareness ability, we introduce the block concept in VLN tasks and propose a block-aware spatial locating (BAL) module to determine the agent's position within the given block, forming positional modeling. • We introduce a spatial-aware action planning (SAP) module, which incorporates the spatial locating information to associate the corresponding guidance and enhance the precision of action planning. 
 
 Appendix:  


Decision: There is consensus among reviewers that the paper is well-motivated, well-organized, and clearly written. The results presented in the paper are convincing and through ablations are presented as well. The author response addressed majority of the reviewers concern. I am recommending to accept the paper. However, the reviewers are advised to add the missing citations and discuss them in the paper.

Reviews: review: paper_summary: paper_summary: This work introduced a novel framework Loc4Plan for enhancing agent performance in outdoor vision and language navigation. The Loc4Plan framework contains block-aware spatial locating module and a spatial-aware action planning module for improving agent's grounding ability and action planning precision.
strength_weakness: strength_weakness: The paper is well-organized: The backgrond and motivation is clear. The function of each module is introduced in detail. The framework is clear to understand. 

The experiment is convincing. The experiment is sufficient and analysized from quanititive and qualitative aspect.
limitations: limitations: Real-time performance of the system is not evaluated. Since this work focuses on outdoor vision and language navigation, computation efficiency and corner cases should be considered.
paper_summary: paper_summary: The paper presents a method for outdoor vision-and-language navigation that is composed of two main modules block-aware spatial locating (BAL) and spatial-aware action planning (SAP). BAL identifies how far is the agent from the next intersection (where a turn can be made), while SAP grounds the instruction to the atomic actions exploiting the prediction returned by BAL.
strength_weakness: strength_weakness: The paper is engaging and well-written, presenting an interesting method for outdoor Vision-and-Language Navigation (VLN). The method is thoroughly described, ensuring the reproducibility of the code. The figures are well-integrated with the text, enhancing the comprehension of the method and providing valuable insights during the qualitative evaluation.

The method is thoughtfully designed, particularly with the introduction of two key modules for spatial localization and action planning. These modules significantly contribute to the overall experimental results.

The experimental evaluation is comprehensive, thoroughly examining all proposed modules. This extensive evaluation clearly demonstrates the improvement provided by each component of the architecture.
limitations: limitations: Regarding the limitations of the paper, my main concerns are related to doubts about the behavior of the model in some specific cases, in order to have an idea of the reasoning capabilities of the method.

How does it behave when the directions contained in the initial text instruction are not in order, e.g. "Go right at the second intersection after crossing the crossroad with the red building"?

I've seen that using the BAL module gives an improvement in the performance, but could the author explain better why predicting the distance from the next intersection is useful for action planning? The agent would need to reach the intersection anyway before making an eventual turn, and the information given by the observation captured at the intersection could be enough to make a decision. Why is using BAL better than forcing the agent to move directly at the next available intersection?

Could the authors elaborate on how the BAL output is beneficial for textual grounding?

What is the computational complexity of the model compared to the competitors?

The related work section on VLN could be expanded, for example, some recent work is missing:
[1]: Chen, S., Guhur, P. L., Tapaswi, M., Schmid, C., & Laptev, I., Learning from unlabeled 3d environments for vision-and-language navigation. ECCV 2022
[2]: Wang, X., Wang, W., Shao, J., & Yang, Y., Lana: A language-capable navigator for instruction following and generation. CVPR 2023
[3]: Rawal, N., Bigazzi, R., Baraldi, L., & Cucchiara, R., AIGeN: An Adversarial Approach for Instruction Generation in VLN. CVPRW 2024
[4]: Zhou, G., Hong, Y., & Wu, Q., Navgpt: Explicit reasoning in vision-and-language navigation with large language models. AAAI 2024

Some minor typos need to be fixed:
line 88 phrase-> phase
line 284 Priliminary -> Preliminary
line 449 plannin -> planning
paper_summary: paper_summary: This paper proposes a new framework named Loc4Plan consisting of a block-aware spatial locating (BAL) module and a spatial-aware action planning (SAP) module. The BAL module is used for localizing the spatial position on the block level and the SAP module can leverage this spatial information to make the action prediction.
strength_weakness: strength_weakness: - This design of the BAL module and the SAP module is creative.
- Numerous ablation studies conducted by the authors evaluate the effectiveness of various modules.
- This framework achieves the new State-Of-The-Art performance on two different datasets.
limitations: limitations: - The Vision-and-Language Navigation task is a sequence prediction task. However, the overview picture depicted in Figure 2 does not show this well. The illustration may lead to a misunderstanding that this task is a one-time prediction classification task.
paper_summary: paper_summary: They introduce a novel framework, Loc4Plan, designed to incorporate spatial perception for action planning in outdoor VLN tasks. The main idea behind Loc4Plan is to perform the spatial localization before planning a decision action based on corresponding guidance.
strength_weakness: strength_weakness: 1. They introduced the agent location information into the action decision-making process.

2. The use of BAL and SAP modules significantly improves navigation performance.
limitations: limitations: 1. They use LSTM and resnet as text and visual encoders respectively, which are too old. Why not use a transformer to extract features?

2. At line 414, they assume that $o_t^p$  accumulates the relevant information about spatial localization. I think this is unreasonable.