Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environments

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 04:48:00

Abstract: Real-world navigation often involves dealing with unexpected obstructions such as closed doors, moved objects, and unpredictable entities. However, mainstream Vision-and-Language Navigation (VLN) tasks typically assume instructions perfectly align with the fixed and predefined navigation graphs without any obstructions. This assumption overlooks potential discrepancies in actual navigation graphs and given instructions, which can cause major failures for both indoor and outdoor agents. To address this issue, we integrate diverse obstructions into the R2R dataset by modifying both the navigation graphs and visual observations, introducing an innovative dataset and task, R2R with UNexpected Obstructions (R2R-UNO). R2R-UNO contains various types and numbers of path obstructions to generate instruction-reality mismatches for VLN research. Experiments on R2R-UNO reveal that state-of-the-art VLN methods inevitably encounter significant challenges when facing such mismatches, indicating that they rigidly follow instructions rather than navigate adaptively. Therefore, we propose a novel method called ObVLN (Obstructed VLN), which includes a curriculum training strategy and virtual graph construction to help agents effectively adapt to obstructed environments. Empirical results show that ObVLN not only maintains robust performance in unobstructed scenarios but also achieves a substantial performance advantage with unexpected obstructions. The source code is available at \url{https://anonymous.4open.science/r/ObstructedVLN-D579}.

Main paper: 
 introduction: 
 Vision-and-Language Navigation (VLN) [5] requires agents to follow natural language instructions to reach a specified destination. Recently, there has been a growing interest in this area due to its great potential for real-world applications such as household robots, especially with the booming of Large Language Models (LLMs) [9, 63] . However, current VLN tasks are limited by several unrealistic assumptions, making their application remain in simulators with few real-world robotic deployments [18, 70] .One significant constraint is what we call the "perfect instruction assumption", which implies that instructions are always perfectly aligned with the environment, neglecting real-time dynamics like unexpected obstructions. Agents trained under this assumption excel in following instructions but lack adaptability for actual navigation where discrepancies between instructions and reality are commonly seen. For example, as shown in Fig. 1 , the human instructs the agent to "Walk straight down the hall" based on prior knowledge of the house. However, real-world environments have changed, with unforeseen obstacles like a suitcase blocking the hallway. While humans can quickly adapt and find detours, current VLN agents struggle with these instruction-reality mismatches, often leading to navigation failures.To address this issue, it is essential to introduce these discrepancies into VLN. While various factors can lead to such discrepancies, this work focuses on one of the most representative and prevalent causes: obstructions. We propose to integrate obstructions into existing discrete VLN environments to block the path described by the instruction, resulting in an instruction-reality mismatch. Notably, although both approaches involve obstructions, our work differs from previous work on obstacle avoidance [3, 71] in terms of problem setup, navigation focus, and potential solutions (See Sec. 2), leading us to choose the discrete setting.Therefore, we make various modifications to the navigation graphs and visual observations of the R2R dataset [5] , proposing the R2R with UNexpected Obstructions (R2R-UNO) dataset as the first VLN task to emphasize the instruction-reality mismatches. At the graph level, we selectively block edges whose removal does not impact the overall graph connectivity in existing paths, ensuring agents can still reach the destination. To keep visual coherence with graphs, we design an object insertion module that leverages text-to-image inpainting techniques [58] to integrate diverse objects into different scenes seamlessly. Due to the instability of inpainting results, a filtering module is employed to select the high-quality ones from multiple candidates to ensure obstruction generation. Our study using R2R-UNO reveals that state-of-the-art VLN agents [13, 68] perform poorly in obstructed environments, limiting their practical applications. The overall framework of our method. We first generate obstructed environments based on existing datasets, and then train agents with our proposed curriculum strategy and graph construction mechanism on both data types.Obstructed EnvironmentsAlthough we expect agents to perform well with mismatches, it is crucial to maintain their performance in original environments simultaneously, which represent most situations. However, directly training with these two types of data poses challenges for agents to learn beyond following instructions and to differentiate between mismatched and aligned scenarios. Empirical results show that discrepancies between these two environments can lead to imbalanced optimization, favoring one type. To address this, we develop the Obstructed VLN (ObVLN) method, including a curriculum training strategy [7] to organize the training and a novel graph construction mechanism to introduce virtual nodes for blocked edges to facilitate efficient exploration. The overall framework including data generation and agent training is presented in Fig. 2 .We conduct comprehensive experiments on R2R, REVERIE [52] , and R2R-UNO datasets to demonstrate the significance of incorporating instruction-reality mismatches in VLN. Established methods, such as DUET [12] , struggle in obstructed settings, with a significant 30% drop in Success Rate (SR). The proposed object insertion and filtering modules provide crucial visual feedback aligned with navigation graph changes, proved by the improved performance of agents using inpainting images. By employing ObVLN, agents not only maintain robust performance in original scenarios but also effectively adapt to mismatches, achieving an impressive 67% SR, marking a significant advancement.We summarize our contributions in this paper as follows:• We address the underexplored issue of instruction-reality mismatches in VLN by proposing R2R-UNO, the first VLN dataset that includes such mismatches through graph changes and diverse obstruction generation, offering a unique challenge that reflects real-world navigation complexities.  
related work: 
 Vision-and-Language Navigation. In VLN, agents need to navigate within simulated environments like Matterport3D [10] following natural language instructions. Numerous methods [19, 47, 53, 61, 74] and datasets [33, 43, 52, 62, 73] have been developed to address various challenges in this domain. Some works address the data scarcity problem by introducing extra sources, including synthesized instructions [66, 67] , additional environments [29, 41] , and predicted scenes [31, 37] . Recently, ScaleVLN [68] synthesized enormous high-quality instruction-trajectory pairs for HM3D [55] and Gibson [69] environments to make agent performance approach human results. Other works [24, 40, 46] apply diverse network structures such as LSTM [23] , Transformer [64] , and Graph Neural Networks [21] to enhance cross-model alignment and decisionmaking ability. For example, HAMT [11] encodes the full history information through transformers and integrates it with instructions and observations for better action prediction. DUET [12] further maintains a topological map to enable agents aware of global visual representations to make global decisions instead of adjacent viewpoints. We adopt HAMT and DUET as the evaluation models due to their superior performance and difference in whether map-based.Environment Changes in VLN. Many works [16, 34, 51] have proposed to modify VLN environments for different purposes, which can generally be categorized based on the changing level. One category involves changing the visual observations to augment data for improving agent generalization. For example, Li et al. [17] utilized image captioning to obtain language descriptions of panoramas and employed generative models to produce novel views based on these descriptions. Conversely, another category focuses on graphlevel adjustments rather than the visual level to go beyond the established navigation graphs by mixing up different graphs [42] or substituting a fixed viewpoint with proximate locations [31] . Notably, VLN-CE [32] abandons the graph-based navigation paradigm and allows agents to traverse continuous environments freely to enhance task realism. Differently, our task makes changes at both visual and graph levels to block edges in the graph and generate obstacles in the views to generate instruction-reality mismatches.Obstacle Avoidance. Obstacle avoidance has been a long-standing research challenge in visual navigation [1, 8, 49, 59] . In VLN, ETP-Nav [3] applies an obstacle-avoiding controller with a trial-anderror heuristic to explicitly escape from deadlocks. SafeVLN [71] employs a LiDAR-based waypoint predictor and a re-selection strategy to avoid non-navigable waypoints and obstacles. However, our work diverges from these works under the continual setting [32] in three aspects, leading us to choose the discrete setting. Firstly, we target a broader range of situations with instruction-reality mismatches, including avoidable obstacles, closed doors, rearranged furniture, instruction errors, etc. These scenarios can be effectively captured by graph changes but are hard to model as continuous signals. Secondly, while obstruction avoidance focuses on assessing the properties of obstructions to avoid them, our work emphasizes changes in navigation graphs that render the instructions temporarily invalid, ignoring the nature of the obstructions. The discrete setting decouples path planning from technicalities, such as the shape and size of obstacles, thereby focusing on high-level adaptability when instructions fail and enhancing its generality. Finally, addressing instruction-reality mismatches involves more than just navigating around obstructions. It requires a complex strategy to find detours and the ability to navigate without instruction guidance, posing a significant adaptability challenge which is too hard for current continuous agents.Object Insertion. Inserting novel objects into target images has always been a research challenge in Computer Vision. Early works [28, 30, 35] employ the cut-and-paste strategy to merge two pictures, which is straightforward but lacks photorealism. With the development of neural networks, object insertion can be achieved in various ways and higher quality, including image synthesis [36] , neural radiance fields editing [65] , image generation [6] , etc. In navigation, THDA [48] inserts 3D scans of household objects into random locations as navigation goals to augment the training data for ObjectGoal Navigation [4] . In VLN, Envedit [16] leverages semantic image synthesis [50] to create objects based on modified environment semantics as augmented environments. In this work, we uniquely take advantage of the text-to-image inpainting models [58] to seamlessly embed desired objects within specific view locations without influencing visual surroundings. 
obstructed environments: 
 In this section, we first introduce the problem formulation of VLN on vanilla and obstructed environments, and then describe our proposed R2R-UNO dataset for introducing instruction-reality mismatches into VLN.In VLN, the agent must follow a natural language instruction x = (𝑥 1 , 𝑥 2 , ..., 𝑥 𝐿 ) with 𝐿 words to navigate a simulated environment. This environment is usually discrete with a predefined undirected navigation graph G = {V, E} with navigable nodes V and connectivity edges E. At each timestep 𝑡, the agent perceives a panoramic representation, comprised of 𝑁 = 36 views 𝑂 𝑡 = 𝑜 𝑖 𝑡 𝑁 𝑖=1and an orientation feature that encodes heading 𝜃 and elevation 𝜙 information of its current node 𝑣 𝑡 . It then determines an action 𝑎 𝑡 to transition to one of the neighboring nodes N (𝑣 𝑡 ) by selecting the view that aligns best with the target node from the current panorama 𝑂 𝑡 . An additional "STOP" action is available to conclude the navigation process. For each instruction x, there exists a corresponding ground truth path𝑃 =< 𝑣 1 , 𝑒 1 , 𝑣 2 , 𝑒 2 , • • • , 𝑣 𝑛 > represent-ing the intended trajectory with 𝑛 nodes for the agent. However, previous work presumes perfect alignment between instruction and reality, assuming that all edges < 𝑒 1 , 𝑒 2 , • • • , 𝑒 𝑛-1 > are consistently accessible. This assumption overlooks the dynamics of real-world navigation, where the graph G may have changed. Among various reasons for graph changes, we focus on the most representative one: obstructions. In our obstructed environments, some specific edges 𝐸 𝑠 ⊆ 𝑃 might be obstructed, making parts of the instruction inapplicable to the current scenario. This obstruction also leads to corresponding visual changes in the panoramic views 𝑂 𝑡 of nodes linked by these edges. When meeting obstructed edges 𝐸 𝑠 , the agent must find a substitute path 𝑃 ′ to reach the final destination 𝑣 𝑛 . In this section, we introduce the proposed R2R-UNO dataset in detail, including the graph changes and visual modifications.3.2.1 Graph Changes. For a path 𝑃 =< 𝑣 1 , 𝑒 1 , 𝑣 2 , 𝑒 2 , • • • , 𝑣 𝑛 > in R2R, we first identify all redundant edges within this path, denoted as 𝐸 𝑟 ⊆ 𝑃. A redundant edge 𝑒 𝑖 = (𝑣 𝑖 , 𝑣 𝑖+1 ) ∈ 𝐸 𝑟 is defined by the property that its removal does not affect the overall connectivity of the graph, ensuring an alternate path 𝑃 ′ between 𝑣 𝑖 and 𝑣 𝑖+1 .To account for scenarios where multiple edges are concurrently obstructed, we assess the collective redundancy of combinations of these redundant edges 𝐸 𝑟 and categorize these combinations by the number of obstructed edges 𝑥 into Block-𝑥 set. Since R2R paths have 4 to 7 nodes in length, we set 𝑁 𝑚𝑎𝑥 = 3 as the maximum number of obstructed edges, generating three sets Block-1, 2, 3, each containing individual training and evaluation splits. With a Block-𝑥 set, consider a combination of redundant edges𝐶 =< 𝑒 𝑖 1 , 𝑒 𝑖 2 , • • • , 𝑒 𝑖 𝑥 >. For each edge, we identify the shortest path < 𝑃 ′ 1 , 𝑃 ′ 2 , • • • , 𝑃 ′ 𝑥 >between the corresponding nodes in the modified graph G = {V, E -𝐶} to replace 𝐶 in the original path 𝑃, generating the path P as the new intended path for agents:EQUATIONWe refer to P as the "real path" to be traversed by agents, in contrast to the "instructional path" 𝑃 based on the navigation instructions x. Therefore, the instruction-reality mismatch is defined as P ≠ 𝑃, which demands agents actively seek alternate paths. To exclude excessively lengthy and impractical new routes, each Block-𝑥 set possesses a "real path" length restriction as 𝐿( P) ≤ 10 + 5 • 𝑥. As a result, 98.8% R2R paths are modified to be one or more new paths in R2R-UNO. As indicated in Tab. 1, the R2R-UNO sets contain a greater number and length of paths than R2R, with the average path length increasing with 𝑥, denoting greater navigation difficulty. Notably, when constructing P, we deliberately avoid shortening the path when 𝑃 ′ contains future viewpoints from 𝑃, even if this leads to the loop formation in P. This design aligns with practical navigation scenarios since when deviating from the instruction, it is challenging for agents to align the current viewpoint with the described path due to the absence of contextual clues. Therefore, it is logical and efficient for agents to seek a detour around the obstruction before attempting to re-align with the instructions.To align with the graph changes, we introduce two novel modules to infuse various objects into panoramic views of nodes along redundant edges, as shown in Fig. 3 .The first one is the object insertion module, which employs a stable diffusion inpainting model [58] to approach the issue from an inpainting perspective. Consider a redundant edge 𝑒 linking nodes 𝑣 𝑎 and 𝑣 𝑏 . We elaborate the process of modifying the panoramic view𝑂 𝑎 = 𝑜 𝑖 𝑎 36𝑖=1 at node 𝑣 𝑎 as follows, which is similarly applied to 𝑂 𝑏 . First, we localize the other node 𝑣 𝑏 within 𝑂 𝑎 to find the corresponding discrete view 𝑜 𝑏 𝑎 and calculate the pixel coordinates (𝑥 𝑏 , 𝑦 𝑏 ) in 𝑜 𝑏 𝑎 as follows:EQUATIONEQUATIONEQUATION𝑊 , 𝐻 is the width and height of 𝑜 𝑏 𝑎 , while 𝜃 𝑟 and 𝜙 𝑟 is the relative heading and elevation from 𝑣 𝑎 to 𝑣 𝑏 , and 𝛿 is the vertical Field of View (FoV) to calculate the focal length 𝑓 . A right trapezoid mask 𝑀 𝑏 𝑎 is then generated, expanding from the bottom edge to around the point 𝑣 𝑏 with a specified width to cover potential areas of paths from 𝑣 𝑎 to 𝑣 𝑏 . This is based on the observation that paths in these views either stretch from the bottom middle to the target point (𝑥 𝑏 , 𝑦 𝑏 ) or form a vertical line from the bottom to the target. Finally, we combine an object name 𝑜𝑏 𝑗 with a predefined prompt template to formulate the final prompts 𝑝𝑟 , which, along with the view image 𝑜 𝑏 𝑎 and mask 𝑀 𝑏 𝑎 , serve as the input to the inpainting model to generate the inpainted view ô𝑏EQUATIONWhile inpainting models can generate visually coherent and photo-realistic images, their instability is notable, with a certain probability of failing to incorporate the desired object. This instability often arises when the area outside the mask already contains elements similar to the intended object, such as a "chair" or "table", which can mislead the inpainting model even with intricate prompts. Therefore, we propose a filtering module to improve the quality of inpainted images, which evaluates multiple generated candidates with different objects to filter out higher-quality ones. Specifically, to enrich the generated scenarios, we first carefully select ten object categories as obstructions based on their frequencies in both the Matterport3D dataset and real-world environments, including chair, table, sofa, potted plant, basket, exercise equipment, vacuum cleaner, suitcase, toy, and dog. We then utilize the object insertion module to produce a set of novel views as candidates, each with a different kind of object. To reflect the inpainting quality, we employ the CLIP [54] model to assess the compatibility score 𝑠 𝑐 of each view-object pair. Note that this assessment considers only the modified part of the view by combining the mask 𝑀 𝑏 𝑎 :EQUATIONThe • operator represents pixel-wise multiplication. These scores are aggregated to form a dataset including all redundant edges within R2R. Visual analyses of this dataset indicate that the score distribution for each object category often takes the shape of a bimodal Gaussian distribution, with one peak N (𝑠 𝑐 |𝜇 1 , 𝜎 2 1 ) for successful incorporation and another N (𝑠 𝑐 |𝜇 2 , 𝜎 2 2 ) for failure:EQUATION𝜋 is the mixing coefficient, N (•|𝜇, 𝜎 2 ) represents a Gaussian distribution with mean 𝜇 and variance 𝜎. Based on this insight, we train a bimodal Gaussian Mixture Model (GMM) [57] for each category and use them to decide which images are qualified:EQUATIONFinally, we randomly select one of those qualified candidates as the modified view, or, in the absence of suitable options, the choice is narrowed to the top three scorers. This module not only largely enhances the probability of incorporating visible obstructions but also ensures the selection of contextually fitting objects. We present the distributions of object categories and their compatibility scores in the appendix to prove the diversity of R2R-UNO. All steps above only concern one discrete view 𝑜 𝑏 𝑎 . The final step is propagating the updated view ô𝑏 𝑎 to adjacent views 𝑜 𝑗 1:8 𝑎 containing overlapped areas with 𝑜 𝑏 𝑎 to maintain consistency across the panorama. We first calculate the transformation matrix 𝐾 𝑖 for view 𝑜 𝑗 𝑖 𝑎 based on SIFT features [45] matching. We then apply 𝐾 𝑖 to project the inpainted part of ô𝑏 𝑎 onto the adjacent view and merge it with the original view to construct a coherent novel panorama:EQUATIONEQUATIONHere, * denotes the projection operation by pixel-wise matrix multiplication applied to image coordinates.It is important to note that we only perform 2D inpainting for nodes linked by redundant edges, which may lead to multiview inconsistency. Although 3D object insertion can alleviate this problem, we stick to the 2D method for two reasons. Firstly, current 3D techniques [20, 56] mainly rely on limited pre-built object datasets [14, 39] , which lack diversity and can cause visual inconsistencies in aspects like lighting and style. Other generative 3D methods [38, 60] struggle to generate high-resolution photo-realistic images and often produce noticeable artifacts. Secondly, the impact of multi-view inconsistency is minimal in our task, as instructionreality mismatches are primarily defined by graph changes. Agents can only detect graph changes when moving to nodes alongside redundant edges, making visual modifications at other nodes irrelevant since agents should still follow instructions without detecting the mismatch. Moreover, various environment augmentation methods [16, 17] have proved that this inconsistency would not affect agent performance in real-world scenarios. 
instruction-reality mismatches and solution: 
 In this section, we show that current VLN methods perform poorly when encountering instruction-reality mismatches and thus propose the ObVLN method to solve this problem.To assess the impact of instruction-reality mismatches, we conduct zero-shot evaluations on the R2R-UNO validation unseen splits using five advanced VLN methods, known for their excellent performance under the perfect instruction assumption: 1. RecBERT [25] employs a recurrent BERT [15] model to preserve cross-modal information throughout navigation. 2. HAMT [11] utilizes a transformerbased model to encode instructions, observations, and historical context. 3. DUET [12] constructs a real-time topological map to enable global action decisions with a graph transformer. 4. GELA [13] enhances the cross-modal alignment between visual landmarks and corresponding phrases with new annotations and pretraining objectives. 5. ScaleVLN [68] integrates extensive high-quality environments and instruction-trajectory pairs from diverse datasets to enhance performance. Fig. 4 illustrates how their success rates vary with different numbers of blocked edges. While most methods achieve a 60%-70% success rate in original environments, the curves sharply decline to around 40% when a single edge is obstructed, with further deterioration observed in the Block-2 and Block-3 scenarios. ScaleVLN outperforms other methods in R2R-UNO owing to its strong generalization ability brought by extensive training data, but it still suffers from a nearly 20% performance reduction in the block-1 set of R2R-UNO. We conclude that current VLN models are overly focused on instruction-following capabilities and lack essential basic navigation functionality to adapt to graph changes.5HF%(57>&935@ +$07>1HXU,36@ '8(7>&935@ *(/$>,&&9@ 6FDOH9/1>,&&9@ Figure 4 : The large performance drop of current VLN methods in the validation unseen splits of R2R-UNO.These findings emphasize the severity of using the perfect instruction assumption in VLN research and the urgency for its resolution.With R2R-UNO, agents can be trained in obstructed environments to better adapt to instruction-reality mismatches. However, significant gaps exist between original and obstructed environments, such as navigation requirements, reliance on instructions, and trajectory numbers, making direct training on these two types of data compromise the performance in both environments (See Tab. 2). Therefore, we propose Obstructed VLN (ObVLN), including a novel training strategy and a graph construction mechanism to organize the training of these two environments and help agents deal with obstructions. We advocate a curriculum learning strategy to treat original and obstructed environments as distinct yet complementary tasks to facilitate the training. This strategy leverages the instruction-following skills gained in original environments as a foundation to tackle obstructed settings. Initially, agents are trained in purely unobstructed environments. As training progresses, the sample ratio of obstructed environments 𝛼 on the training data gradually increases until it reaches a predefined maximum 𝛼 𝑚𝑎𝑥 at step 𝑐, defined as follows:EQUATION)where 𝑡 is the current training step. All training batches include instances from both settings to ensure smooth optimization.To facilitate exploration for detours, we introduce a graph construction mechanism for graph-based methods to incorporate "virtual nodes" to represent currently inaccessible nodes due to obstructions. This mechanism consists of three steps, with the pseudo-code in the appendix. Firstly, when encountering an obstacle at node 𝑣 𝑎 , we estimate the location of the obstructed node 𝑣 𝑏 based on the direction of the obstructed view and a predefined distance 𝑑 (3 meters in R2R-UNO). This estimation allows us to conceptualize a virtual node 𝑣 * 𝑏 within the topological graph as a placeholder for 𝑣 𝑏 . Next, as the agent moves, it consistently computes its distances to 𝑣 * 𝑏 and 𝑣 𝑎 , denoted as 𝐷 (𝑣 * 𝑏 ) and 𝐷 (𝑣 𝑎 ), respectively. This distance computation is crucial for determining the proximity of the agent to the virtual and obstructed nodes. Finally, if the distances satisfy 𝐷 (𝑣 * 𝑏 ) < 𝜃 and 𝐷 (𝑣 𝑎 ) < 𝜃 for a given threshold 𝜃 , and the current view indicates obstruction, we infer that the agent  
experiments 5.1 datasets and evaluation metrics: 
 We mainly evaluate our methods on the widely used VLN benchmark R2R [5] with step-by-step instructions and our proposed R2R-UNO to focus on the challenge of instruction-reality mismatches.We present results on the goal-oriented benchmark REVERIE [52] without such mismatches in the appendix. R2R is built on the Mat-terport3D dataset [10] , including 10,800 panoramic views from 90 building-scale scenes. Each R2R path has three or four natural language instructions from human annotators. R2R has four splits: train, validation seen (val seen), validation unseen (val unseen), and test unseen. We utilize the train split for training and the val seen and val unseen splits for evaluation to align with R2R-UNO.For evaluation, we follow previous works [2, 72] to use four primary metrics in VLN: (1) Trajectory Length (TL): the total navigation length in meters; (2) Navigation Error (NE): the distance between the stop location and the target; (3) Success Rate (SR): the ratio of agents stopping within 3 meters of the target; (4) Success rate weighted by Path Length (SPL) [4] : SR normalized by the ratio between the length of the shortest path and the predicted path. Metrics related to instruction fidelity like CLS [27] or nDTW [26] are not included due to the modified trajectory with obstacles.For R2R-UNO creation, we use the stable-diffusion-v1.5-inpainting model for object insertion and the CLIP ViT-L/14 to evaluate textimage pairs. We use brute force and K-Nearest Neighbors matching to align the SIFT features of two adjacent views. For methods in Fig. 4 , we use their best models on the validation unseen split and extract the features of the obstructed environments according to their settings. We employ HAMT [11] and DUET [12] for navigation training and follow the implementation details in their official repositories. The obstructed environments are only used in the fine-tuning stage. The maximum sample ratio 𝛼 𝑚𝑎𝑥 is set to 0.5 with the increasing step 𝑐 as 20,000. We increase the maximum action length to 30 for longer ground truth paths. PREVALENT [22] is used as the augmented data to stabilize the training. We utilize the AdamW optimizer [44] and select the best model based on the average SPL in the val unseen splits of R2R and R2R-UNO. All models are fine-tuned for 100K iterations with a learning rate of 1e-5 and a batch size of 8 on a single NVIDIA A6000 GPU.We first demonstrate that our proposed obstructed environments and ObVLN can facilitate agent adaptation to instruction-reality mismatches. Therefore, we apply HAMT and DUET to three distinct training settings: (1) Basic: training with R2R; (2) +OE: incorporating data from both the R2R and R2R-UNO; (3) +ObVLN: using the ObVLN method on R2R and R2R-UNO. Note that both settings 2 and 3 include all three sets of R2R-UNO. Tab. 2 presents the navigation performance of different models on the val seen and unseen splits of R2R and R2R-UNO. The results show that models trained with two types of data significantly outperform those trained only on R2R in navigating obstructed scenes within the R2R-UNO dataset. However, this improvement is accompanied by a noticeable degradation in R2R performance, like the 6% and 7% SR drop of DUET in the seen and unseen splits, respectively. This phenomenon can be explained by the considerable TL increase in R2R, suggesting that agents are over-optimized for obstructed environments and tend to take detours even without obstructions. Our ObVLN method effectively addresses this issue. For HAMT, ObVLN surpasses the basic setting in R2R, achieving the best results across all four sets of R2R and R2R-UNO. For DUET, the topological map and global action space lead agents to return to the node with the blocked edge which aligns best with the instruction. This results in a slight loss of efficiency in terms of SPL in R2R for ObVLN. However, it still achieves a comparable SR to the basic setting and significantly outperforms setting 2 in R2R. Additionally, it achieves state-of-theart results in all three sets of R2R-UNO. The advantage of HAMT with ObVLN is less significant because HAMT is not a map-based approach, making our graph construction mechanism inapplicable. While DUET outperforms HAMT by a 7% SR increment on R2R, this superiority becomes much more significant in obstructed environments, in which DUET achieves around 25% SR lead in the most challenging Block-3 set. We attribute this to the topological map design in DUET, which significantly enhances exploration efficiency and serves as a critical component in finding detours. This advantage is consistent with the large lead of DUET on explorationintensive, object-oriented datasets like REVERIE [52] .In this section, we conduct an ablation study on several important components to show their effectiveness.Modules for R2R-UNO. We first evaluate the impact of the Object insertion Module (OM) and the Filtering Module (FM) by generating different Block-1 sets for training and evaluation. Without these two modules, the obstructed environments only contain graph changes without visual modifications. Tab. 3 presents the performance of HAMT and DUET on the val unseen splits using different modules. The R2R performance is consistent across all experiments due to ObVLN. For obstructed environments, using the object insertion module to modify the panoramic views can enhance agent navigation by providing critical visual feedback on graph changes, evidenced by performance gains. Moreover, including the filtering module further improves the inpainting quality, leading to more successful obstruction generations and the best performance.Strategy. We explore four sampling strategies for DUET to utilize both original and obstructed environments: (1) Path-wise, where a path is randomly selected from the combined shows their performances on the the val unseen splits of R2R and R2R-UNO. We set 𝛼 𝑡 = 0.5 to match 𝛼 𝑚𝑎𝑥 and provide experiments on different 𝛼 𝑡 and 𝛼 𝑚𝑎𝑥 in the appendix. Among all strategies, our curriculum sampling achieves dual superiority, achieving the best performance in both R2R and R2R-UNO. Other methods experience performance drops on R2R due to the over-optimization problem. Path-wise sampling obtains the worst performance due to path imbalance, with more training on paths with many redundant edges while ignoring those with none or less.Mechanism. We compare our graph construction mechanism with two baselines: the Vanilla setting that overlooks the obstruction in the graph and an idealized Oracle setting that receives the ground truth information about nodes occluded by obstructions as a performance ceiling. Tab. 5 shows the performance of DUET with different graph construction methods on the val unseen splits of R2R-UNO. Our approach, incorporating virtual nodes into the graph, significantly outperforms the vanilla setting on all three sets, especially in more challenging scenarios with three obstructed edges. As expected, the Oracle graph achieves the highest performance due to its access to accurate, unobstructed We present some obstructed environments from R2R-UNO in Fig. 5 comparing with corresponding original views and those generated without the filtering module. Our method successfully integrates various objects into specific locations within the original views, creating realistic and contextually harmonious obstructions. In contrast, results without the filtering module often fail to include the objects, demonstrating the critical role of this module in enhancing the inpainting reliability. Additionally, we evaluate the generation quality using the CLIP [54] score and present the scores for each category in Fig. 6 . The filtering module consistently improves and stabilizes the generation quality, achieving higher scores and lower variance across all categories, aligned with the examples. 
conclusion: 
 This work introduces obstructed environments into VLN to address the prevalent issue of instruction-reality mismatches in real-world navigation. We present R2R-UNO, the first VLN dataset to incorporate such mismatches by integrating diverse obstructions into R2R at both the graph and visual levels through a novel object insertion and filtering module. Using R2R-UNO, we demonstrate that current VLN methods struggle in obstructed settings and further propose the ObVLN method to help agents effectively adapt to obstructed environments. Through these innovations, our agents achieve state-ofthe-art results in R2R-UNO while maintaining robust performance in R2R. We believe addressing the perfect instruction assumption is crucial for the practical application of VLN agents and for assessing their adaptive capabilities to navigate beyond only following instructions. Future work should improve the agent performance in both settings and extend this work to continuous 3D environments.  
 
 Appendix:  


Decision: The work addresses instruction-reality mismatches in VLN, an important topic in real-world applications that has previously been overlooked. The reviewers agree on the paper's technical novelty and convincing results. Some concerns were raised in regard to additional baselines, multi view consistency, real world application etc. and most of them were addressed in the author response. Consequently, I am recommending acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This work shows the discrepancy between instructions and reality in real-world navigation. To solve this problem, this work proposed a novel method called ObVLN, including a curriculum training strategy and virtual graph construction for agent to adapt to obstructed environments.
strength_weakness: strength_weakness: Very novel: The work focuses on the instruction-reality mismatches in VLN, which is a important topic in real application and didn't get attention before.

Well-organized: The work is well-organized and easy to follow. It first introduced the R2R-UNO dataset for exploring instruction-reality mismatch. After that, it introduced the proposed ObVLN structure and make performance comparison between proposed method and current VLN methods in obstructed environment.

Convincing. The work made a throughout analysis on performance comparison results and gave explainations for phenomenon. More experiments and explainations are provided in supplementary materials.
limitations: limitations: No obvious limitations in this work.
paper_summary: paper_summary: They introduce R2R-UNO, which contains various types and numbers of path obstructions to generate instruction-reality mismatches for VLN research. To  help agents effectively adapt to obstructed environments, They propose a novel method called ObVLN (Obstructed VLN), which includes a curriculum training strategy and virtual graph construction.
strength_weakness: strength_weakness: 1. The R2R-UNO which provided a foundation for subsequent research，is meaningful. They integrate obstructions into existing discrete VLN environments to block the path described by the instruction, resulting in an instruction-reality mismatch.

2. The result of ObVLN method is significant.
limitations: limitations: 1. The R2R dataset is based on step-by-step instructions, which differs from practical applications. I would prefer to see their results on the REVERIE dataset. This dataset uses high-level instructions, which is more realistic.

2. The baseline they use is relatively outdated, DUET is already the method for 2021. Currently, many methods[a,b,c] use semantic maps, large models, etc. for navigation, and I believe these methods can solve the problem of environmental changes.
[a] BEVBert: Multimodal Map Pre-training for Language-guided Navigation
[b] March in Chat: Interactive Prompting for Remote Embodied Referring Expression
[c] GridMM: Grid Memory Map for Vision-and-Language Navigation
paper_summary: paper_summary: The paper introduces the R2R with Unexpected Obstructions (R2R-UNO) dataset and task to address the issue of instruction-reality mismatches in Vision-and-Language Navigation (VLN).  The authors propose the ObVLN (Obstructed VLN) method which includes a curriculum training strategy to organize training on both original and obstructed environments, and a virtual graph construction mechanism to help agents explore efficiently when facing obstructions.
strength_weakness: strength_weakness: (1)The paper is well-structured and clearly written. The motivation, problem formulation, proposed solutions, and experimental results are presented in a logical and easy-to-follow manner. 
(2)The proposed R2R-UNO dataset and task are the first to explicitly incorporate such instruction-reality mismatches into VLN benchmarks.
(3)The paper provides comprehensive experiments to validate the significance of the proposed dataset and method.
limitations: limitations: (1)The experiments are limited to simulated environments.  While this is common in VLN research, real-world validation is crucial for demonstrating practical applicability. To fully demonstrate the practicality of the proposed method, it would be beneficial to include real-world experiments or discuss the challenges and potential solutions for deploying ObVLN agents in physical environments.
(2)The object insertion and filtering modules, while innovative, are complex and computationally expensive. This complexity may limit practical implementation due to the reliance on high computational resources and specific inpainting models, which could be a barrier for broader adoption.
(3)The paper mentions potential issues with multi-view inconsistency due to the use of 2D inpainting methods but does not thoroughly discuss the impact this may have on agent performance or possible solutions.