General Scene Adaptation for Vision-and-Language Navigation
Authors: Haodong Hong
Yanyuan Qiao
Sen Wang
Jiajun Liu
Qi Wu
Venue: ICLR 2025 Poster (2025)
Keywords: vision-and-language navigation; scene adaptation; multi-modal learning
Primary area: datasets and benchmarks
TL;DR: We propose a new dataset, GSA-R2R, which significantly expands the diversity and quantity of environments and instructions for the Room-to-Room (R2R) dataset to evaluate agent adaptability in both ID and OOD contexts.

Abstract: Vision-and-Language Navigation (VLN) tasks mainly evaluate agents based on one-time execution of individual instructions across multiple environments, aiming to develop agents capable of functioning in any environment in a zero-shot manner. However, real-world navigation robots often operate in persistent environments with relatively consistent physical layouts, visual observations, and language styles from instructors. Such a gap in the task setting presents an opportunity to improve VLN agents by incorporating continuous adaptation to specific environments. To better reflect these real-world conditions, we introduce GSA-VLN (General Scene Adaptation for VLN), a novel task requiring agents to execute navigation instructions within a specific scene and simultaneously adapt to it for improved performance over time. To evaluate the proposed task, one has to address two challenges in existing VLN datasets: the lack of out-of-distribution (OOD) data, and the limited number and style diversity of instructions for each scene. Therefore, we propose a new dataset, GSA-R2R, which significantly expands the diversity and quantity of environments and instructions for the Room-to-Room (R2R) dataset to evaluate agent adaptability in both ID and OOD contexts. Furthermore, we design a three-stage instruction orchestration pipeline that leverages large language models (LLMs) to refine speaker-generated instructions and apply role-playing techniques to rephrase instructions into different speaking styles. This is motivated by the observation that each individual user often has consistent signatures or preferences in their instructions,  taking the use case of home robotic assistants as an example. We conducted extensive experiments on GSA-R2R to thoroughly evaluate our dataset and benchmark various methods, revealing key factors enabling agents to adapt to specific environments. Based on our findings, we propose a novel method, Graph-Retained DUET (GR-DUET), which incorporates memory-based navigation graphs with an environment-specific training strategy, achieving state-of-the-art results on all GSA-R2R splits.

Published as a conference paper at ICLR 2025
GENERAL SCENE ADAPTATION FOR VISION-AND- LANGUAGE NAVIGATION
Haodong !i Yanyuan Qiao", Sen Wang', Jiajun Liu W
'The University of Queensland, CCSIRO Dataó1, "The University of Adelaide {haodong. .hong sen, wen.wangJeug, edu, au, jiajun. www.cciro...... (yanyuan. giao, gg.wwol]eadelalde edu.au
ABSTRACT
aibio--nnd-Languaaa Navigation (VLN) tasks mainly evaluate agents based on one-time execution of individual instructions across multiple environments, aim- ing to develop agents capable of functioning in any environment in a zero-shot manner. However, real-world navigation robots often operate in persistent envi- ronments with relatively consistent physical layouts, visual observations, and lan- guage styles from instructors. Such gap in the task setting presents an opportu- nity to improve VLN agents by incorporating continuous adaptation specific en- vironments. To better reflect these real-world conditions, we introduce GSA-VLN (General Scene Adaptation for VLN), novel task requiring agents execute nav- igation instructions within a specific scene and simultaneously adapt to it for im- proved performance over time. To evaluate the proposed task, one has to address two challenges in existing VLN datasets; the lack of outoof-distribution (OOD) data, and the limited number and style diversity of instructions for each scene. Therefore, we propose new dataset, GSA-R2R, which significantly expands the diversity and quantity of environments and instructions for the Room-to-Room (R2R) dataset to evaluate agent adaptability in both ID and OOD contexts. Fur- thermore, we design a three-stage instruction orchestration pipeline that leverages large language models (LLMs) to refine speaker-generated ply role-playing techniques to rephrase instructions into different speaking styles. This is motivated by the observation that each individual user often has consistent signatures or preferences in their instructions, taking the use case of home robotic assistants an example. We conducted extensive experiments on GSA-R2R thoroughly evaluate our dataset and benchmark various methods, revealing key factors enabling agents to adapt to specific environments. Based on our findings. we propose a novel method, porates memory-based navigation graphs with an environment-specine training strategy, achieving ttate-of-the-art results all GSA-R2R splits. The dataset and code are available at /tttps://gitiii
generated instructions and ap-
Retained DUET (GR-DUET)) which incor-
GSGA-VIN
INTRODUCTION
-nd-Languagee Navigation (VLN) (Anderson et al., 2018b) aims to enable agents to navigate to specific destination following language instructions. Traditional VLN researches (Qi et al, 2020 Thomason et al. 2020) mainly focus on evaluating agents using strictly unseen instructions and environments to assess their generalization capabilities. This "unseen" criterion applies not only to the data between training and evaluation phases, but also to individual evaluation instances, where agents tested on each instrection-trajectory pair without prior knowledge of the environment.
However, this setting diverges from practical navigation situations. In real-world applications, such as indoor houschold robots, agents actually operate in a consistent environment over time. As they execute more instructions, the working environment generally transitions from "unseen" to "seen" and agents become increasingly familiar with the environment. This leaves room for agents to adapt and improve their performance through environmental adaptation, which can significantly
"Corresponding author. Work done during internship at Australian Institute for Machine Learning (AIML)..

---

Published as a conference paper at ICLR 2025
Lab
I'm as clueless as the first time I was here.
Walk past the table and turn right. Walk past the doorway and turn right. Stop in front of the sink.
Traditional VLN
Praceed forward, droomnvnting workestation with the arimost care, and stsoete a 4D degree rightwaard your passago through
grow more familiar with the Lab and Dr. Cooper's communication style each day!!!
rotation. Then doorway, maintaliing A steadiist riglitword trajectory. Upon completing this, predisety in front of the stainless sinic.
halt
UPDATE
SAVE
Frozen
Trainable
SAVE
LOAD
Memory
Bank
GSA-VLN
Figure 1: Comparison between the traditional VLN task and the proposed GSA-VLN task. Tra- ditional VLN agents can only execute fixed-style instructions with frozen parameters, remaining unfamiliar with the environment like the laboratory even after extended use. In contrast, GSA-VLN enables agents to dynamically update parameters, leverage long-term history from the memory bank, and quickly adapt to both the environment and varying instruction styles from different users.
enhance the agent performance of existing VIN methods. IVI.N Krantz et al. 2023 emphasizes the persistent environment problem by linking paths of each scan into long- monghooizonn tours, but the limited number of paths (6-100) restricts its applicability to optimization- time Adaptation (TTA) (Liang al. 2024) and fails to replicate real-world usage scenarios.
methods like Test-
Therefore, we propose a novel task called General Scene Adaptation for VLN (GSA-VLN). As shown in Fig. [ in this task, agents required to maintain long-ter memory and continuously update their model parameters to improve performance over time while executing navigation in- structions in specific scene throughout their lifetime. Externally, the agent behaves the same as traditional VINN settings. However, internally, evolves during this process by adapting to the working environment without additional feedback or assistance, process known as unsupervised learning, making our task both practical and realistic.
Although GSA-VLN focuses on environment-specitn adaptation, the agent must also be general enough to adapt to diverse range of scenes as its target environment, given the wide variety of real- world settings. However, the limited number and diversity of environments and instructions in ex- isting VIN datasets raise concerns about whether the trained agents can be deployed across various real-world situations. Currently, all indoor VI.N tasks are built on the Matterport31) dataset (Chang et al. 2017) with fewer than 30 evaluation buildings, most of which are residential. While some works expand the scope by incorporating additional datasets (Chen et al. 2022b Wang et al. 2023b) or generating synthetic ones (Li et.. 2022; Li & Bansal 2024), they only used for training rather than evaluation. Moreover, instructions in these tasks are typically concise and simplistic, lacking personal characteristies distinctive language habits that reflect real-world user interactions. Since both the environment and instructions are in-distribution (ID) with the training data, it is essential to evaluate agents on -iitt---iitiiittion (OOD) data to ensure broader generalization.
To this end, we introduce the GSA-R2R dataset, which provides a comprehensive collection of en- vironments and instructions for evaluating agent performance in both ID and OOD contexts within a single scene. We incorporate buildings from the Habitat-Mattepoor333 (HM3D) dataset (Ramakr- ishnan et al. 202T), which offers a broader range and greater number of photorealistic environments compared to the MP3D dataset, supporting more general evaluation. Since most training environ- ments are residential, we categorize residential buildings as ID data and conduct building-scan scale clas- sification to identify non-residential structures, such as cinemas and shops, as OOD data, resulting in diverse set of 150 evaluation buildings with various layouts and structures.
Besides visual environments, instruction variety is a crucial factor in scene adaptation, as instruc- tions for a specific environment are often consistent in style, reflecting the unique speaking patterns of the instructors. For example, household agents are typically instructed by homeowners, while school- -ased agents are primarily used by teachers and students. These instructions are delivered in consistent style, reflecting either personal speaking habits or the professional language asso- ciated with the roles in the building. However, this aspect has been largely overlooked in current VLN research. To address this, develop a three-stage instruction orchestration pipeline to pro-
2

---

Published as a conference paper at ICLR 2025
duce abundant and diverse instruction ttrjonctorrrrrrryyooooooooooorte pairs for each environment. Initially, use trained speaker model (Tan et al.) 2019) to produce noisy instructions, which are then refined by large Vision-Language Models (AChiaa et al. 2023) with path visualizations, and finally rephrased by Large Language Models (LL.Ms)(Achaan et al. (023) to simulate diverse instructional styles of specific characters typical users of the building type. As result, each environment is equipped with 600 instructions across multiple speaking styles, providing a more realistic and
valuable resource for future VLN research.
We conduct a comprehensive evaluation of our generated data using both automatic metrics and human studies to validate its quality and diversity. We also test various existing adaptation meth- ods, revealing that unsupervised training approaches are ineffective in our setting, while explicitly maintaining the memory of previous visual and textual information shows promise in improving agent performance in GSA-R2R. Based on these findings, we propose novel method called Graph- Retained DUET (GR-DUET), which continuously updates an overall topological graph for each en- vironment to preserve historical information in both training and evaluation. This method achieves an 8% improvement in Success Rate (SR) compared to the vanilla DUET (Chen et al. 2022c and produces state-of-the-ee results across all splits.
RELATED WORK
Vision-and- Ssion-and-Langgaag Language Navigation (VLN). VLN tasks involve agents following natural language instructions to reach a specified target (Gu et al. 2022 Zhang et al., 2024), a challenge first intro- duced with the Room-to-Root (R2R) dataset (Anderson al. Bb). Since then, numerous indoor
VLN datasets emerged (Krantz et al., 2020: Jain et al. 2019) which mainly focus on varying the textual inputs, such as high-level object- oriiettd instructions QQi et al. 22200 Zhu et al. (22T)) mul- tilingual instructions (Ku et al. 2020), and multi-modal instructions (Hong et al.] (024). However, they are all based the same scenarios from the Matterport3D dataset (Chang al.] (017). Many works propose to introduce novel scenarios by incorporating other datasets (Chen et al. 2022b)
Wang et al. 2023b) or utilizing web data (Lin al. 2023). However, all these methods are only used as additional training data with evaluations still being conducted on the original splits with lim- ited diversity. In contrast, we include diverse environments and instructions in the evaluation splits to fully evaluate agent adaptability in both ID and OOD contexts.
Adaptation Methods in VLN. Although no prior works in VLN have addressed the problem of single-scene adaptation, some studies offer potential solutions, such as the pre-explore setting (Wang et al. 2019). We categorize them into two kinds. The first is optimization--assed methods, where the parameters of the navigation model are updated within the target environment. TTA methods like TENT (Wang et al., (02I) and SAR (Niu al. 2023) further optimize the model parame- ters with the objective of entropy minimization, while Tran-Trannlation (Wang et al., 2020 uses a trained speaker to generate instructions for imitation learning. The second is memory- baased meth- ods, which explicitly store information about seen places and instructions to help decision decision-making, as seen methods of IVLN (Zhao et al., (024) and RREx-BoT (Sigurdsson et al. (023). For example, TourHAMT (Krantz et | 2023) incorporates previous episodes as additional history em- beddings, while OVE (Zhao et al. 2024) detects keywords from instructions within observa- tions and stores the results in an Omnigraph to assist navigation. Similarly, SG-Nav (Yin et al. (024)) constructs consistent 3D scene graphs to represent environments, offering a powerful approach for recording and utilizing historical information. Recently, LLM-based VLN methods, such as In- structNav (Long et al.] 2024) and NavCoT (Lin et al. 2024). have demonstrated strong zero-shot navigation performance, highlighting their potential to address the scene adaptation problem.
Persistent Environments in VLN. The concept of agents operating in persistent environments has led to several popular tasks in embodied AI, such as multi-object navigation (Wani et al. 2020) and multi-target embodied QA (Yu et al., 2019). Iterative VLN (IVLN) (Krantz et al. (023) introduces this idea to VIN by organizing all instructions in sequence to form a long-horizon tour and enable memory throughout the tour. Our task differs from IVLN in two key aspects. First, while both tasks focus on enhancing agent performance within single environment, IVLN emphasizes long-horizon navigation with a single tour per environment, whereas our GSA-VLN focuses on enabling agents to adapt to each environment from a diverse range of visual buildings and instruction types. Second, IVLN only includes a limited number of trajectory-instruccion pairs for each environment, making it not suitable for optimization-basee ones.
3

---

Published as a conference paper at ICLR 2025
3 TASKS AND DATASETS
3.1 PRELIMINARIES
In VLN task, the agent is required follow given natural language instruction X (I1, #2. ... TI) consisting of L words to navigate to the target destination. Specifically, the agent is initially placed at the start node vo. At each time step t, the agent predicts an ac- tion dr to move to one of the neighboring nodes in the connectivity graph G of the environment, guided by the visual observation O₁, the language instruction X, and the history of previous steps H, {On: ag: O1, 01, O2-1, a.-1). Typically, the observation O, is represented as panorama composed of 36 discrete views O { {
N 1=1 This process continues until the agent either reaches the predefined step limit or selects the special [STCP] ] action.
3.2 THE GSA-VLN TASK
Most VLN tasks evaluate each instruction independently, initializing the history as empty (Ho 0) and keeping the model parameters fixed (# 0g). While this setting effectively assesses the agent's ability to interpret and follow isolated instructions, it fails to capture the continuity and adaptation required in real-world navigation scenarios. In practical applications, both environments and in- struction styles remain consistent over time, enabling agents to accumulate and leverage contextual knowledge enhance performance. To address this limitation, we propose the GSA-VLN task to introduce the challenge of single-scene adaptation, enabling agents to continuously improve as they execute instructions in previously unseen environments.
Specifically, GSA-VLN introduces an environment-speciffc memory bank Mg. which stores his- torical information from all executed episodes within a given environment E. This memory bank dynamically expands as the agent executes instructions, capturing four key components: visual ob- servations (O), the instructions (X), selected actions (A), and trajectory paths (P). For example, after executing l instructions in environment E, the memory bank is updated as follows:
Mg = (x1* O 1sk A 1:k p1:k
Externally, agents in GSA-VLN behave similarly to those in standard VLN tasks when executing instructions. However, internally, the agent can leverage the memory bank adapt the cur- rent working environment for better performance, depending on the method employed. The stored memories in the memory bank represent the execution history of agents, although there may exist misalignment between instructions and paths due to navigation errors, all the memories are treated as unlabeled data and are primarily used for unsupervised learning techniques.
There are two primary distinctions between GSA-VLN and standard VLN tasks. First, the agent can access the memory bank retrieve long-term history, Ho M' S MM, instead of beginning each navigation episode without prior knowledge:
ag x0,, X, Ho:
Second, the parameters of agents can be updated during the navigation process by employing unsu- pervised learning techniques on data from the memory bank:
0' = - a VLLLa 0),
where 0 denotes the parameters of the navigation model. Notably, While GLL aims to develop environment-speciin agents 6', the initial model 9 should be general enough to be applied to various environments, maintaining level of environment-agnosicn
max RE (P(F; 0' (0o))
where E represents the environment distribution, and P 0 (0o)) denotes the agent performance in environment E' with updated parameters 0', which are adapted from the initial parameters 0g-
We address differences between GSA-VLN and the related areas. Unlike lifelong learning (Liu 2017). which focuses on acquiring multiple skills over time, our work emphasizes repeated mastery of the same navigation skill within scene-speeific context. While TTA (Gao al.) 2024) adapt an agent during inference without supervision, our approach extends TTA by integrating a dynamically updating memory bank, enabling Exed-parameter adaptation with varying inputs.
4

---

Published as a conference paper at ICLR 2025
R2R
Tran
RZR Eval Soones
GSA GSA-R2R Scenes
office
GSA-R2R
hiorntr
Figure 2: Left: Building type counts in R2R and GSA-R2R. Right: Comparison of buildings in R2R and GSA-R2R. Unlike R2R, where evaluation scenes similar to the training set, GSA-R2R includes a more diverse mix of both in-distribution (ID) and nutro--distribution (OOD) data.
3.3 THE GSA-R2R DATASET
The limited number and diversity of existing VLN datasets make them unsuitable for the GSA-VLN task. For instance, the most widely used VLN dataset, R2R (Anderson et al. 2018b), includes 90 building-scale scenes from the Matterport3D (MP3D) dataset (Chang al. 2017). with only 29 scenes for evaluation and most of them are residential houses. Each building contains limited number of paths, ranging from a maximum of 100 to as few as 6, with each path paired with three natural language instructions that share similar concise and plain style. Therefore, we propose the GSA-R2R dataset, which not only provides sufficient data to allow continuous agent optimization to test their specialization, but also includes diverse range of building types and instruction types to evaluate the agent generalization to various application scenarios.
3.3. 1 ENVIRONMENT
The Habitat- MMrtterprrttttt (HM3D) dataset (Ramakrishnan et al. 2021) has 800 photorealistic build- ings with more diverse building types and structures compared to MP3D. Therefore, we choose to incorporate HM3D environments into agent evaluation assess whether agents can adapt more diverse settings. We categorize the buildings into two types: residential and non non-residential, with the former as ID data and the latter as OOD data since the training data are most residential ones.
Specifically, we collect and manually refine 187 building types from the OpenStreetMap website | which provides crowd-source building tags worldwide. For each environ- ment, we use GPT-4 (Achiam et al., (023) to predict the building type from these cate- gories based three types of image prompts: an overview the building, top-down views of each floor, and panoramas selected using spectral clustering (Ng et al., (001)) on the graph nodes to capture representative visual observations. For non-residential results, we manually verify and correct the predictions and apply two filtering rules. First, we ex- clude environments in the R2R training set to avoid data leakage, ensuring that existing VLN models could be directly evaluated us- ing our dataset. Second, we remove scans with fewer than 600 potential paths to ensure sufficient number of paths per scan. In to- tal, we identify 75 residenidentaal buildings across 19 different types with 70 from HM3D and 5 from MP3D. To balance the dataset, sample
Environments
Ovandon
Pisnark
HMOD & NP3D Enviromenns
VLM
Suilding 7ypes
Fabding
Serraing
Spectral Clusdering
Repeovetador Narks
Instructions
Enstronmnnte
GSA-CaR
Greeph
Spaaker
IeHtal Insrunfoas
Navigation Model
Testull
Boci Intractions
Path
Rasibrrta
Charactor Profiees Dháloge
Intracioas
ScAte
LLM
Ever
Exsruntiass
Figure 3: The generation procedure of the GSA- R2R dataset. Up: environment selection. Bottom: the three-stage instruction orchestration pipeline.
5

---

Published as a conference paper at ICLR 2025
Table 1: Compared to the evaluation part of existing datasets in embodied navigation tasks. f:
Town is an outdoor dataset in New York City, which makes it hard to define scene types.
Scenes
Path Num
Instructions Type
Vocab Size
Datasct
Source
Num
Type
Num
All
Unseen
R2R (Anderson et al
2018b
MP3D
29
2,174
6,522
1,946
545
R4R Dain el all
MP3I>
29
5,016
45,234
1,230
RxR-en [Ku
2020
MP3D
29
4,201
8,636
4,789
1,387
CVDN
[Thomason et al.
2020
MP3D
29
6
2,741
2,741
1
1,559
490
TouchDown? (Chen et
2019
Google Street View
1
2,800
2,800
3,104
759
RobustNav Challopadiivay
2021
ROBOTHOR
15
1,800
PASTURE (Gidre el al. 2025
ROBOTHOR
15
2,520
2,520
123
111
GSA-R2R (Ours)
MP3D & HM3D
150
20
90,000
90,000
7
4,337
2,905
an equivalent number of residential buildings and classify them into three categories based on the number of levels. For each category, we use K-Means clustering select 25 scans based on meta- data such as room count and navigable areas to cover a diverse set of layouts and structures within the 75 residential buildings, with 65 from HM3D and 10 from MP3D, as shown in Fig.
3.3.2 INSTRUCTION
With the selected environments, we employ the navigation graphs from ScaleVLN (Wang et al, 2023b) and randomly sample 600 paths for each environment. To simulate the diverse speaking styles in real-world scenarios, we develop a three-stage instruction orchestration pipeline to produce three types of instructions for these sampled paths, as illustrated in Fig.
In the first stage, we employ the EnvDrop speaker (Tan et al. 2019) trained on R2R to generate initial instructions P However, some of these instructions contain noise and inaccuracies, requir- ing further refinement. Therefore, in the second stage, we utilize a navigation model trained on the unselected paths from the 150 environments identify incorrect instructions by using successful ex- ecution as an indicator of instruction feasibility. For failed instructions, we leverage GPT-4 to detect misalignments between trajectories and instructions,and correct them using specialized path visu- alization prompts, resulting in refined "Basic" instructions. Building on the "Basic" instructions, we further rephrase them in the third stage to reflect distinct speaking styles. For non-residential buildings, we use an LLM to select potential user identity based on the scene type and rephrase the instructions into "Scene" style. We further create "User" style for all scenes by simulating specific characters from TV series using their role profiles and dialogues (Wang et al. (2023)). We select five diverse characters from the SummScreen dataset (Chen et al.] (2022a) to generate instruc- tions with unique speaking styles, including generalized child-speaking style due to the absence of actual child characters. We provide a comparison of different instructions in the appendix.
3.3.3 SPLIT AND STATISTICS
Since our focus is on scene adaptation after the training phase, we include only evaluation splits and use the training split of R2R for the GSA-R2R dataset. Given the two general building types (resi- dential and non-residential) and three types of instructions, we design five splits for both validation and testing, with their details the appendix. The splits are named using the format "Val/Test- where "R" denotes residential and "N" represents non-residential scenes. We exclude the val seen split, as adapting to a previously trained environment is unnecessary; thus, all environments in GSA-R21 are considered unseen. We compare GSA-R2R with other embodied navigation datasets in Tab. I demonstrating that GSA-R2R has the greatest number and diversity of scenes, paths, and instructions. Notably, are the first to incorporate various speaking styles to address OOD instructions, as evidenced by the highest proportion of unseen vocabulary size.
MNN-Bascc/Scene/Ul
3.3.4 DATASET EVALUATION
Reliability, Since our goal is to generate instructions with distinct styles and OOD charac- teristies, which intentionally diverge from the training data, traditional linguistic metrics such as BLEU (Papineni et al.) 2002) or ROUGE (Lin 2004) are not appropriate for evaluation. These metrics assess level alignment with ground truth instructions, whereas our in- structions should align with the path observations. However, there is currently no automatic
We include the evaluation splits in the training of EnvDrop to improve quality.

---

Published as a conference paper at ICLR 2025
method to evaluate this alignment. Therefore, conducted user study to invite 15 par- ticipants to evaluate 20 randomly selected instructions from GSA-R2R. They judged whether the instructions accurately described the path and whether they exhibited distinct speaking style. The results in Tab. 2 demonstrate around 80% alignment and a high proportion of dis- tinct speaking styles, proving the reliability of our data. User instructions are judged as less styled, likely because their word-level changes are less noticeable when viewed individually.
Diversity. We provide a I-SNE analysis to com- pare the instructions from R2R and GSA-R2R, shown in Fig. 4 Specifically, we use BERT (De- vlin 2018) to obtain embeddings for all instruc- tions from these two datasets and then use t- SNE (Van der Maaten & Hinton 2008) to project For
Table 2: Human evaluation of various GSA- R2R instructions. Matching: percentage of instructions that accurately describe the path. Style: percentage of instructions with clear speaking style.
Type
Matchingt
Stylet
them into a 2D space for visualization. clarity, we only display the results for the User Sheldon here; the results for other characters are presented in the appendix. The left image of Fig. | shows that the evaluation instructions in R2R are all ID data, sharing the same distribu- tion the training instructions. In contrast, distinctly different distribution from the training data, demonstrating the diversity of our dataset. Since the Basic instructions consist of speaker-generated and LLM-refined compo- nents, their cluster overlaps with the training cluster, which we still consider as ID data.
EnyDrop
52.2
13.0
Basic
80.0
14.7
Scene
76.6
96.1
User
83.3
57.6
Scene and User instructions exhibit
4 EXPERIMENTS
4.1 GR-DUET
Although TourHAMT incorporates history information additional input, its perfor- mance degrades for two reasons. First, rep- resenting each step with single history em- bedding fails to capture the necessary spatial correlations for modeling visited nodes, espe- cially when the history is extensive. Second, only fine-tunes the model with new history embedding compositions, leading to a signifi- cant input distribution shift between pretrain- ing and fine-tuning. To address these issues, we propose a novel cemory-based method, graph-retained dual-scale graph transformer (GR-DUET), where the extended history em- beddings are replaced by a global topological graph that retains information across episodes, ensur- ing comprehensive awareness of visited nodes.
Tatn K23 (val
2a 1BAKCRC USMICO OSAIZE adics
Figure 4: The t-SNE analysis of instructions from R2R (left) and GSA-R2R (right). Our instructions demonstrate significantly greater diversity com- pared to R2R and include OOD data.
Specifically, during inference, instead of maintaining separate graphs 92, G.. for each episode {EP,, E'P,, ... E.P.. J. our agent maintains single global graph G, to continuously up- date the topological map while preserving observations at each node throughout the evaluation. At the start t = 0 of episode k, we utilize the data from the memory bank MM to build the topological graph with all previously visited nodes G2 (G1 ... Gx-1). Each node represents a visited location, with node information including positions {xr, y, 2} for spatial alignment and the visual ob- servations at that node. We reset the visited status of all nodes at the beginning of each episode to enable agents to only choose unvisited nodes the next step for efficiency. By utilizing this global graph across episodes, GR-DUET can more effectively leverage historical information, enabling a deeper understanding of the environment and facilitating longer-term action planning, particularly after executing numerous instructions.
However, directly applying this mechanism in evaluation leads to the same distribution shift issue. Therefore, we modify the pretraining and fine- fine-tuning stages to align the input distribution between training and inference. During pretraining, we provide the model with the complete ground truth
7

---

Published as a conference paper at ICLR 2025
Table 3: Navigation performance of different VLN models in the test splits of R2R and GSA-R2R.
t: Data leakage exists since Scale uses HM3D as training data.
Test-R- MeeeRRBBaa
Basic
Test-N-Busii SPL
Test-Re R-User
Test-N-Use
Test-N-Scene
R2R- Test
Methods
SRT
SPL1
SR
SR
SPLt
SRT
SPL1
SR1
SPL1
SRT
SPL
HAMT Chen et al. 2021
48
44
42
38
46
42
38
34
34
30
65
60
DUET (Chen
37
55
44
40
BEVBet [An et
2025
58
45
46
35
55
43
44
33
39
27
73
62
ScaleVL. Wang et ml. 2123b
79
67
71
58
71
59
61
48
55
43
77
68
NavGPT-2 Zhou
45
48
35
56
45
45
44
58
72
60
topological map of the current environment to enable them utilize the most abundant information. For fine-tuning, we adopt an -ppeeiffttt-pecffi training strategy to assign individual graphs to each training environment. We set threshold a as the maximum number of episodes to be included in the graph. Once a is reached, the graph is re-initialized to avoid constant prediction under a fully populated graph. To enhance graph coverage and path diversity, we further incorporate the PREVALENT dataset (Hao et al. 2020) as augmented data.
4.2 EXPERIMENTAL SETUP
Baseline Methods We include two types of baseline methods. One is adaptation- based methods, including TENT (Wang et al. 2021). SAR (Niu et al., (023), Back-Translation (BT), and proxy tasks (MLM (Devlin 2018) & MRC (Lu al.) 2019)). The other is memory-based methods,
including TouTHAMT (Krantz et al. 2023) and OVER-NAV (Zhao et al. 2024). More details are provided in the appendix.
Evaluation Metrics. We use the following metries to evaluate the navigation performance: (1) Trajectory Length (TL): the total navigation distance in meters; (2) Navigation Error (NE): the distance between the stop location and the target; (3) Success Rate (SR): the ratio of agents stopping within 3 meters of the target; (4) Success rate weighted by Path Length (SPL) (Anderson et al., 2018ap: SR normalized by the ratio between the shortest path length and the predicted path length. (5) Normalized Dynamic Time Warping (nDTW) (Ilharco al., 2019): measure of instruction fidelity by computing the similarity between the reference path and the predicted path.
Implementation Details. We use GPT-4o from OpenAl's official API as the LLM for generating Scene and User instructions. All prompt templates are provided in the appendix. For baseline meth- ods, we follow the implementation details in their official repositories, with two modifications. First, we use CLIP-ViT/B-16 (Radford et al., (021) the visual feature extractor for both the navigation and speaker models for fair comparison. Second, all models are evaluated using a batch size of 1 in an online manner during evaluation. In GR-DUET, we set the maximum number of episodes a = 50. The best model is selected based on the average SPL across all validation splits. For each adaptation method, conduct the evaluation three times with randomly sequenced instructions and report the mean and standard error for each metric.
4.3 MAIN RESULTS
In this section, we benchmark existing VLN methods and adaptation methods to show their perfor- mance for the GSA-VLA task.
4.3.1 HOW DO CURRENT VLN METHODS PERFORM IN GSA-R2R?
Various techniques have been incorporated into current VLN methods to achieve human-like perfor- mance in R2R, demonstrating their strong reasoning capabilities. We evaluate these methods without adaptation techniques determine whether they can maintain the same performance in our diverse environments and instructions, as shown in Tab. 3 Except for the data leakage in Scale 3 the performance of other baselines in GSA-R2R is significantly lower than their performance in R2R, highlighting the challenges of our task. When comparing different environments, agents are better residential scenes than residennntt ones due to the biased distribution of the training data. For different instruction types, all models perform best on Basic instructions, followed by User instruc-
SSaale uses all buildings from HM3D augmented data for agent training, which includes the envi- ronments in GSA-R2R and portion of the Basic instructions.
8

---

Published as a conference paper at ICLR 2025
Table 4: Comparison of different adaptation methods in GSA-R2R with basic instructions.
Test-R-Basce
Test-N-Bawe Basic
Methods
SR1
NEL
SR
SPL1
aDTW1
Seveline
DUET
Chenn al. 2022c
13.1
4.2
57.7
47.0
55.6
14.8
5.3
48.1
37.3
45,9
Optimizationnnn
Methods
+MLM Deviin 2018
13.1 201
4.1 a01
57.9 282
47.3 101
55.9 =02
13.1 tn2
5.3 =1.1
48.3 205
38.8 :95
48,4 an0
+MRC (1: RUT3
13.1 =01
4.2 401
57.7 =611
47.0
55.6
14.7
5.3
48.1
37.3 e0.
45,9 es:
61.3 266
57.2 103
70.1 103
1M1
5.2
59.4 1m4
*TENT Wang el m1 2021
14.6 200
4.2 300
52.2 244
44.2 104
52.9 :91
16.2 21
5.4 -611
46.3 an1
33.7 ::0
42.6 1n3
+SAR Niu 2045
13.8 =08
4.0
57.6 =82
44.6
53.0
16.5
5:4
44.6
=1.5
40.6 +1.
Momory-Barde Methods
TourHAMI (Krantz et al. 2024
11.6 201
149
211
12.2 inl
34.7 101
9.4 101
1.7 .41
11.0 an3
8.6 103
32.2 in
OVER-NAV Ziap
14.1 =01
6.7 400
22.3
m83
16,8 402
37.1
11.4
7.1
16.6 #02
13.0
35.0 +02
GR-DUET (OUIS)
250
200
69.3
202
64.3 203
200
4.4
au3
:9.1
Table 5: Comparison of different adaptation methods in GSA-R2R with User instructions.
Child
Keith
Moira
Sheldon
Methods
SRT
SPLT
SRT
SPLT
SR
SPL
SR
SPLT
SRT
SPLT
Buseline
DUET
54.3
44.1
56.0
46.3
52.3
43.3
56.3
46.4
54.0
44.4
Optimization
Baxed Methods
+MLM
54.5 402
44.7
56.4 ::0
46.8 1:0.3
53.8 2n3
43.6
56.8
46.6 106
54.5 =04
44.2 20.3
+MRC
54.4 102
44.2 1n.1
56.0 101
46.3 10.1
52.3
102
43.3 in1
56.0 jei
46.2 102
51.7 Li3
44.2 104
+BT
57.5
54.0
61.2
57.9 +0.1
57.3
205
54.0
61.6
58.1 107
57.6 =05
54.3
+TENT
54.3 102
41.7 10.1
$5.4 1nz
43.8 103
51.7 102
41.0 1011
55.0 162
43.2 102
51.0 gm3
41.9 1n1
4SAR
54,5 20.
41.5
54.9
43.1 +03
51.0
40.3
55.3
43.0
52.9
41.4
Memory-Based
Methods
TouHHAMT
#02
12.0
15.1 ::2
12.3 +0.1
=0.1
15.3
12.5 #0.
14.4 =0.
#0.1
OVER NAV
20.9 101
16.1
102
20.5 1.1
16.4 10.
19.5 102
15.4
102
20.6
16.3
16.2 102
20.5 1m1
16.2 1011
GR-DUET (ours)
59.7
66.7
2:0
+0.3
202
:02
67.1
62.2 50.1
63.9 =0.
20.1
tions, and finally Scene instructions. This aligns with the cluster distance from the training data observed in Fig. emphasizing the importance of studying the OOD problem in VLN.
4.3.2 HOW DO ADAPTATION METHODS PERFORM IN GSA-R2R?
Environment Adaptation. We first test the adaptation methods with different en- vironments using Basic instructions, as shown in Tab. 4 Among the optimization- based methods, TTA techniques like TENT and SAR perform worse than vanilla DUET. This is because entropy- based methods assume a positive cor- relation between confidence and accu- racy. However, in sequential decision- making processes like VLN, eπrоrs accu- mulate over time, making entropy mea- sures meaningless after an incorrect step. The Back-Trane TrcksTrannltiion (BT) method shows improvement as the Basic instructions
Table 6: Comparison of different adaptation methods in GSA-R2R with Scene instructions.
Test-N-Seeme
Methods
NEL
SR
SPL.1
nDTW
Raseline
DUET
14.9
39.6
30.1
40.9
Optimization Bussed Methods
6.5
39.8 in1
30.5 131
in1
+MR
14,9 au1
6.4 19.1
39.7 am
30,2 sau
40.9 10.1
BT
8.4
6.3
41.2 215
38.2
51.3
-TNTT
16.4 im
6.3 181
40.6 in2
28.9 122
38.9 102
SAAR
16.3 205
6.0 402
41.4
29.1
39.0 #03
Memory
Baree Based Methods
TOUTAMT OVER-NAV
7.3 =01
8.1
9.7
8.0 #0.1
32.3 #0.1
11.8 40.1
7.6
402
16.7 204
12.6 #02
34.6 #03
GR-DUET (ours)
10.1 200
5.5 300
48.1 a0
42.8 aM1
83.7 1n
closely resemble the authentic data from the speaker. For proxy tasks, MRC performs similarly to DUET, as this coarse-graine grained task is not beneficial once the agent already has robust representations through extensive training, which is also observed in ScaleVLN (Wang et al., 2023b). MLM pro- vides marginal improvement. While it helps learn better textual features, it not optimized together with action prediction, preventing the model from leveraging these improved language features. For memory-based methods, both TourHAMT and OVER-NAV performed significantly worse than even vanilla HAMT. This is due to the larger number of instructions GSA-R2R compared to their training data, resulting in excessively long history embeddings as input, which confuses the model. In contrast, GR-DUET achieves the best performance in both Residential (R) and Non-residential (N) splits, with an 11.6% and 8.5% SR increase, respectively. This demonstrates its effectiveness in helping agents adapt to both ID and OOD environments.
Instruction Adaptation. We also evaluate these methods across different instruction styles. Tab. 5 shows the results for User instructions from the five characters, while Tab. 6 presents their perfor- mance Scene instructions. The performance of DUET suggests that different speaking styles
9

---

Published as a conference paper at ICLR 2025
Table 7: Ablation study on the pretraining and augmented data in GR-DUET.
Test-R-Basic B-Basic
Dist-N-Basic
OsstN--cenne
Pretrain
Aug.
SRT
SPLT
SRT
SPLT
SRT
SPLT
X
x
56.8 10.1
47.5 10.1
45,4 10.1
35,3 10.1
38.2 =0,0
28.9 1000
X
V
54.0 10.1
41.4 1011
43.0 100
30.9 10.1
35.9 =04
27.4 1:01
59.9 20.1
48.2 101
47.9 #0.1
35.3 101
43.7 =02
33.6 2:00
69.3
64.3
56.6 #0.1
51.5 +0.1
48.1
42.8 +0.1
Table 8: Ablation study on different graph construction mechanisms in GR-DUET.
Test-R-Basic R-Basic
Test-N-Basic
Test-N-Scene
Method
a
SR1
SPL1
SR
SPL
SR1
SPL1
0.25
61.2 10.1
53.3 #0.1
50.9 =0.1
39.2 1
41.4 :02
30.6 10.1
0.50
64.8 10.1
56.8 10.1
50.8 =0.1
40.4 10.1
37.8 :00
29.4 10.0
Proportion
0.75
57.7 10.1
48.7 10.0
48.3 =0.
37.5 10.1
40.7 10.1
28.7 10.1
1.00
66.2 10.3
58.5 102
55.7 .0.
46.0 10.1
47.5 102
40,9 10.1
57.6 +0.1
35.1 #0.1
45.1 =0.1
24.9 :00
36.9 #0.1
20.6 20.1
50
69.3 102
64.3 =0.1
56.6 =0.1
51.5 10.1
48.1 #0.1
42.8 10.1
Buffer
100
69.7 10.1
63.2 :0.1
56.1 =0.1
48.5
10.1
47.9 102
37.1 10.1
150
67.5 10.0
59.3 1000
54.8 -0.
44.6
10.1
46.2 10.1
37.9
introduce varying levels of difficulty for VLN models in interpreting instructions. Most results are consistent with those from the environment adaptation results, with two notable exceptions. First, TTA methods achieve 1% SR increase Scene instructions, but this advantage disappears in User instructions. We attribute this to their different language patterns. Scene instructions tend to include conversational fillers, which a noticeable pattern that optimization-base contrast, User instructions focus word variations, which is much more challenging. Second, the improvement from evaluation instructions, indicating that this method is effective for environment adaptations but not for instruction adaptations. Lastly, Our GR-DUET again achieves significant performance improve- ments with maximum SR increase of 1 1% across all splits, demonstrating its general applicability to both types of adaptations.
methods can capture. In
diminishes due to the domain shift between the authentic and
4.4 ABLATION STUDY
We conduct two ablation studies on our GR-DUET. Tab. Lab.7prover proves that only adding PREVALENT is detrimental due to the introduced instruction noises and only incorporating full graphs during pre- training is limited by the lack of path diversity during fine-tuning. Combining both approaches leads to significant improvements, demonstrating the effectiveness this strategy. Tab. g we experi- ment with two strategies to simulate the graph construction process in fine-tuning. The "proportion" method randomly provides a specific proportion of ground-truth graphs to agents while the "mem- ory" method is the realization in GR-DUET with maximum capacity for memorizing episodes. For simplicity, we use the same symbol, a, to represent both the proportion and buffer size. The results prove that memory methods outperform proportion methods, as they more closely align with the gradual expansion of the global graph during inference. For memory methods, performance initially improves with increasing buffer size, then declines. This trend is expected as small buffer cannot cover graphs adequately, while an excessively large buffer leads to inefficiencies.
5 CONCLUSION
In this paper, we introduce the GSA-VLN task to highlight the challenges faced by VLN agents operating in persistent environments, where long-term memory and model updates are required to adapt to specific settings. To thoroughly evaluate agent adaptability, we create the GSA-R2R dataset, which significantly expands the quantity and diversity of environments and instructions, including both ID and OOD data for evaluation. We benchmark popular VLN models and adaptation methods GSA-R2R and propose GR-DUET, novel model that integrates global graphs with an unvironment-speciin training strategy, achieving state-of-the. results. In the future, we aim to explore more unsupervised learning approaches further enhance agent performance in GSA-R2R.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work supported by projects DP230101753 funded by the Australian Research Council, and CSIRO's s Science Leader project R-91559, This work is also supported by the Centre for Augmented Reasoning, an initiative by the Department of Education, Australian Government.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087744, 2023.
Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. Bevbert: Multimodal map pre-training for lanuage-guides navigation. In ICCV, pp. 2737-2748, 2023.
Peter Anderson, Angel Chang. Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, al. On evaluation of embodied navigation agents. arXiv preprint arXiv: / 1807.06757, 2018a.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Ausion-and-land lan-lang navigation: Interpreting visually-grounder navigation instructions in real environments. In CVPR, pp. 3674-3683 2018b.
Peter Anderson, Ayush Shrivastava, Joanne Truong, Arjun Majumdar, Devi Parikh, Dhruv Batra, and Stefan Lee. Sim-to-real transfer for 2021.
aan-lang navigation. In CoRL, pp. 671-681,
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In 3DV, pp. 667-676, 2017.
Prithvijit Chattopadhyay, Judy Hoffmman, Roozbeh Mottaghi, and Aniruddha Kembhavi. Robustnav:
Towards benchmarking robustness in embodied navigation. ICCV, pp. 15691-15700, 2021.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR, pp. 12538- 12547, 2019.
Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, and Kwan- Yee Wong. Mapgpt: Map-guided prompting with adaptive path planning for pp. 919999999998 9810, 2024a.
navigation. ACL,
Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. Summscreen: A dataset for abstrac- tive screenplay summarization. In ACL, pp. 8602-8615, 2022a.
Nuo Chen, Y Wang, Yang Deng, and Jia Li. The oscars of ai theater: A survey role-playing with language models. arXiv preprint 2xXv:2407.11484 2024b.
Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and lan-lan navigation. NeurIPS, 34:5834-5847, 2021.
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Learning from unlabeled 3d environments for vision-and-lane lnd-language navigation. In ECCV, pp. 638-655,
2022b.
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act local: Dual-scale graph transformer for vision-and and-language navigation. CVPR, pp. 16537-16547, 2022c.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255. leee, 2009.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18.0 1810.04805, 2018.
11

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192: 2020.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Kinnk-Kiikttrckk Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for aision-and-langune navigation. NeurIPS, 31, 2018.
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilhareo, Ludwig Schmidt, and Shuran Song. Cows on pasture: Baselines and benchmarks for driven zero-shot object navigation. In CVPR, pp. 23171-23181, 2023.
Junyu Gao, Xuan Yao, and Changsheng Xu. Fast-slow test-time adaptation for online vision-and- language navigation. In ICML, 2024.
Jing Gu, Eliana Stefani, Qi Wu, Jesse Thomason, and Xin Eric Wang. Assion-and-languuee naviga- tion: A survey of tasks, methods, and future directions. In ACL, 2022.
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning a generic agent for vision-ande aasoo-and-langae and-langu navigation via pre-training. In CVPR, pp. 13137-13146, 2020.
Haodong Hong, Sen Wang, Zi Huang, Qi Wu, and Jiajun Liu. Why only text: Empowering vision- and-language navigation with multi-modal prompts. IJCAI, 2024.
Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. General eval- uation for instruction conditioned navigation using dynamic time warping. arXiv preprint arXiv:1907.05446, 2019.
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. Stay on the path: Instruction fidelity in usiio-aad--lnguall navigation. In ACL, pp. 1862-1872, 2019.
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision-anddlanguagg navigation in continuous environments. In ECCV, pp. 104-120, 2020.
Jacob Krantz, Shurjo Banerjee, Wang Zhu, Jason Corso, Peter Anderson, Stefan Lee, and Jesse Thomason. Iterative vision-and-language navigation. In CVPR, pp. 14921-14930 2023.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-rowe
Multilingual vision-and-langune navigation with dense spatiotemporal grounding. In EMNLP, pp. -422, 2020.
Jialu Li and Mohit Bansal. Panogen: Text-conditioned panoramic environment generation for vision-and languuge navigation. NeurIPS, 36, 2024.
Jialu Li, Hao Tan, and Mohit Bansal. Envedit: Environment editing for vision-and-language navi-
gation. In CVPR, pp. 15407-15417, 2022.
Jian Liang, Ran He, and Tieniu Tan. A comprchensive survey on test-time adaptation under distri- bution shifts. IJC,, pp. 1-34, 2024.
Bingqian Lin, Yunshuang Nie, Ziming Wei, Jiaqi Chen, Shikui Ma, Jianhua Han, Hang Xu, Xiaojun Chang, and Xiaodan Liang. Navcot: Boosting Ilm-based vision-and- lad--uugee navigation via learning disentangled reasoning. arXiv preprint arXivv2403.07377, 2024.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 81, 2004.
Kunyang Lin, Peihao Chen, Diwei Huang, Thomas H Li, Mingkui Tan, and Chuang Gan. Learning vision-and languuue navigation from youtube videos. In ICCV, pp. 8317-8326, 2023.
Bing Liu. Lifelong machine learning: a paradigm for continuous learning. Frontiers of Computer Science, 11(3):359-361 361,, 2017.
12

---

Published as a conference paper at ICLR 2025
Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. In CoRL, 2024.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin-
guistic representations for iision-and-language tasks. NeurlPS, 32, 2019.
Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering; Analysis and an algorithm.
NeurlPS, 14, 2001.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang. Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui
Tan. Towards stable test-time adaptation in dynamic wild world. In ICLR, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, Bleu: a method for automatic evaluation of machine translation. In ACL, pp- 311-318, 2002.
Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang. William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environ- ments. In CVPR, pp. 9982-9991, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8763, 2021.
Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. matteat-matterport 3d dataset (HM3d): 1000 large- scale 3d environments for embodied AI. In NeurIPS Datasets and Benchmarks Track (Round 2), 2021.
Gunnar A Sigurdsson, Jesse Thomason, Gaurav S Sukhatme, and Robinson Piramuthu. Rrex-bot: Remote referring expressions with bag of tricks. IROS, pp. 2203-5210, 2023.
Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back transla-
tion with environmental dropout. In NAACL, pp. 2610-2621, 2019.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navi- gation. In CoRL, pp. 394 406, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008
A Vaswani. Attention is all you need. NeurIPS, 2017.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, and Jianbing Shen. Active visual infor- mation gathering for vision-language navigation. In ECCV, pp. 307-322, 2020.
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced mrosl matching and self-supervised imi- tation learning for vision-language navigation. In CVPR, pp. 6629-6638, 2019.
Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongeheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, et al. Rolellm: Benchmarking, eliciting, and enhancing pole-playing abilities of large language models. arXiv preprint arXiv:2310.00746, 2023a.
Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scaling data generation in vision-and assion-and-langeee lnn-lan navigation. In ICCV, pp. 12009-12020, 2023b.
Saim Wani, Shivansh Patel, Unnat Jain, Angel Chang, and Manolis Savva. Multion: Benchmarking semantic map memory using multi-object navigation. NeurIPS, 33:9700-9712, 2020.
13

---

Published as a conference paper at ICLR 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Denny Zhou, et al. Chain-of thought prompting elicits reasoning in large language models. NeurIPS, 35: 2022.
Chengguang Xu, Hieu Trung Nguyen, Christopher Amato, and Lawson Wong. Vision and language navigation in the real world via online visual language mapping. In 2nd Workshop on Language and Robot Learning: Language as Grounding, 2023.
Hang Yin, Xiuwei Xu, Zhenyu Wu, Jie Zhou, and Jiwen Lu. Sg-nav, Online 3d scene graph prompt- ing for LLM-based zero-shot object navigation. In NeurIPS, 2024.
Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L. Berg, and Dhruv Batra. Multi- target embodied question answering. CVPR, pp. 6309-6318 2019.
Yue Zhang, Ziqiao Ma, Jialu Li, Yanyuan Qiao, Zun Wang, Joyce Chai, Qi Wu, Mohit Bansal, and Parisa Kordjamshidi. Laiion-and-aanguage navigation today and tomorrow: survey in the era of foundation models. ArXiv, abs/2407.07053. 2024.
Ganlong Zhao, Guanbin Li, Weikai Chen, and Yizhou Yu. Over-nav: Elevating iterative vision-and- language navigation with open-vocabulary detection and structured representation. CVPR, pp- 16306, 2024.
Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, and Qi Wu. Navgpt-2: Unleashing naviga- tional reasoning capability for large vision-language models. In ECCV, 2024.
Fengda Zhu, Xiwen Liang, Yi Zhu, Qizhi Yu, Xiaojun Chang, and Xiaodan Liang. Soon: Scenario oriented object navigation with graph-base exploration. In CVPR, pp. 22689-12699 2021.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
This document provides additional method details, supplementary experiments, and further analysis complement the main paper, including:
Appendix A.1
detailed descriptions of the baseline methods.
Appendix
A
more explanations of the process for generating the GSA-R2R dataset.
Appendix
additional statistics and visualizations of the GSA-R2R dataset.
Appendix
discussion of the feasibility of deploying GR-DUET in real-time systems.
Appendix
detailed analysis of the scalability of GR-DUET.
Appendix
comparison with LLMM-based VLN methods.
Appendix
quantitative analysis of the adaptation speed of GR-DUET.
Appendix
more details about the human study.
Appendix
A.
Detailed justification for selecting five characters for User instructions.
Appendix
A. 0
prompt templates used for generating GSA-R2R.
Appendix
discussions the limitations and future directions of this work.
A.1 BASELINE METHODS
In this section, we provide detailed descriptions of the baseline adaptation methods used in our ex- periments, which are categorized into two types: optimization- based methods, which update model parameters during evaluation, and memory- -aased methods, which use data from the memory bank as additional input.
A.1.1 DUET DETAILS
We adopt DUET Chen et al. 2022ch as our baseline model, which is an enhanced version of HAMT (Chen et al. 2021). The HAMT model employs transformer-basee based network to encode instructions, visual observations, and navigation history. These components are first processed by in- dividual encoders and subsequently integrated through cross-modal transformer (Vaswani) (017). Building on the foundation of HAMT, DUET introduces a real-time topological map to track visited nodes and leverages graph transformers to enable global action decisions. Unlike HAMT, which is restricted to selecting actions only from neighboring nodes of the current location- navigation inefficiencies DUET expands its action space include all nodes the topological map. This allows DUET to efficiently navigate to distant nodes using a path planner, significantly improving navigation efficiency. Moreover, DUET employs a dual-level encoding architecture to en- code both fine-graines features of visual observations and coarse saanse-rriied features of the topological map. These representations are fused with instructions to capture cross-modal relationships, facil- itating more effective action predictions. For agent training, DUET adopts the two-stage training paradigm introduced by HAMT, consisting of pretraining on proxy tasks and fine-tuning on down- stream tasks. During fine-tuning, DUET employs a pseudo- interriitte demonstrator enhance exploration, thereby improving generalization performance.
to
A.1.2 OPTIMIZATION BASED ADAPTATION METHODS
Due to the lack of ground-trune paths, optimization- based adaptation methods rely on unsupervised and self-supervised training strategies to adapt the model to specific environments.
Entropy-Basse -Bssed Methods. Online Test-Time Adaptation (TTA) methods (Liang et al.) 2024) fo- cus on minimizing the entropy of agent predictions to make agents more confident in their decisions by exploiting the positive correlation between entropy and errors. Since the VLN task is formu- lated as a classification task on neighboring viewpoints, these methods are directly applicable to the GSA- VLN task. We employ two widely used methods in this field, TENT (Wang al., 2021) and SAR (Nuu et al. 2023). to evaluate whether these TTA techniques can help agents adapt to specific environments and speaking styles. TENT takes entropy minimization an optimization objective
15

---

Published as a conference paper at ICLR 2025
on all test samples while SAR further takes the sharpness of entropy into consideration to only consider those reliable test samples instead of all samples to adapt to more diverse settings.
Tronsarranll If we consider the visited nodes in previous episodes as exploring the envi- ronment, the scene adaptation problem in the current episode can be regarded as the pre-explore setting (Wang et al.] (209)) in the standard VLN task. A key method in this setting is back- translation (Wang et al. 2020), which involves using a trained speaker to generate instructions for randomly sampled paths consisting of visited nodes, followed by teach-forcing imitation learning on this synthetic data. While this method has proven effective in the pre-explore setring of R2R, its performance for scene adaptation in GSA-R2R is less promising. This is primarily due to the intro- duction of diverse instruction styles, which causes the synthetic instruction to diverge significantly from the actual evaluation instructions. Another key difference is that traditional back-translation operates on the full navigation graph, while in GSA-VLN, graph, which is only part of the full graph. This leads to two issues. First, the sampled paths may not always be the shortest between two viewpoints since the local optimal way may differ from the global one. Second, with the generated instruction-rraecttry pairs, the agent performs pure imita- tion learning without exploration, as the entire process occurs within the agent's 's internal model, not in the simulator, and the memory bank lacks data from unvisited nodes.
limited to the current constructed
Proxy Tasks, One important approach for unsupervised training in VLN is using proxy tasks to update the model, which is widely employed during the pretraining stage of VL.N models. We employ two common proxy tasks, Masked I Language Modelling (MLM) (Devlin) 2018 8) and Masked Region Classification (MRC) (Lu et al. 2019), to individually assess whether they can help agents adapt to specific visual environments or distinct instruction styles. MLM randomly masks 15% of instruction tokens and utilizes the output embeddings to predict the masked tokens. MRC similarly masks 15% of visual observations and uses the output embeddings to predict the semantic labels of the masked regions where the label is obtained from an image classification model (Dosovitskiy) 2020) pretrained on ImageNet (Deng et al. (200)). Since both MLM and MRC used in the pretraining stage of DUET, we directly reuse their prediction heads in our adaptation process.
A.1.3 MEMORY-BASER ADAPTATION METHODS
The memory-based adaptation methods explicitly maintain information from previous episodes as additional input to help decision-making without updating model parameters. We evaluate two IVLN methods representatives of this category: TourHAMT (Krantz et al. 2023) and OVER- NAV (Zhao et al. 2024). TourHAMT incorporates information from previous episodes as additional history embeddings for HAMT to enable reasoning across episodes. OVE performs real-time detection of keywords from instructions within observations and stores the detection results in an Omnigraph, which describes the object distribution facilitate navigation.
A.2 DATA GENERATION IN GSA-R2R
instruction orchestration pipeline. In the first stage, we employ the Envrrop speaker (Tan et al. (201)) trained on R2R to generate initial instructions for each sampled path. Unlike previous works, we include data from the validation splits in the training process of the speaker to enhance instruction quality, as these results are used for evaluation rather than training. We selected EnvDrop over recent speakers because of its widespread use and established performance in VLN tasks. However, the generated instructions still contain noise and inaccuracies that may not align with the path.
Here, we provide additional details
To address this, the second stage involves model-based selection, which uses a navigation model to identify incorrect instructions. Specifically, we use unselected paths from the 150 environments in GSA-R2R to train DUET (Chen et al., (022c) model, which is then evaluated on the selected instructions to determine whether the trained agent can successfully execute the instructions, serv- ing as indicator of instruction feasibility. For environments with a limited number of unselected paths, we apply two-fold cross-vall volosdvali to finetune the DUET on the specific environment using all available paths. The trained DUET achieves a 73.6% SR for instructions generated on the lected paths. For instructions that fail, we leverage the multi-modal reasoning capability of GPT-4o to identify misalignments between the trajectory and instruction, providing corrected versions using
16

---

Published as a conference paper at ICLR 2025
specifically designed path visualization prompts. These prompts consist of two parts: sequence of panoramas from the viewpoints and a top-down view of the path on the map. Various methods were tested for presenting the panoramas, it remains challenging for the language model to fully com- prehend the spatial relationships between panoramas. We finally chose to display the first-person view facing the direction of travel, with red arrows indicating the intended path, with examples in Appendix A...3] Additionally, we employ the Chain-of-Thougee (CoT) technique (Wei et al. 2022) require the LLM detect any issues in the speaker-generated instructions, list the errors, and finally provide a corrected version with an explanation of the modifications. We refer to these in- structions, which are either good or refined, as "Basic" instructions in GSA-R2R, denoting their concise and style-neutral nature commonly found in VLN datasets.
Building on the Basic instructions, the third stage rephrases them to reflect distinct speaking styles. Since real-world instructors can be specific individuals, such as homeowners or specialized users of a building type, we introduce two speaking styles: "Scene" and "User". For non-residential buildings, We use LLM identify potential users based on the building type, then randomly select one to serve as the instructor to rephrase each instruction. We make the LLM align with the speaking style of the selected speaker while keeping the directional information unchanged, thus generating the Scene instructions.
For the User instructions, inspired by recent studies role-playing capabilities in language mod- els (Chen et al. 2024b), we use GPT-4o to simulate specific characters and rephrase the Basic instructions for both residential and non-residential scenarios. Current role-playing studies (Chen) iet al.) 2024b) include two types: persona-based, which relies on persona descriptions, and character- based, which mimies specific characters' behaviors. We found that persona-based methods often in- clude irrelevant words, making the instructions less realistic. For example, a persona described as a "bookshop owner providing reading material from various historical periods" resulted in metaphors like "Stroll beyond the bed, akin to stepping through the pages of a medieval manuscript" " which are inappropriate for navigation tasks. Therefore, we adopted the character-based method.
from
Specifically, we first identify characters with distinct speaking styles from TV series with mainly daily scene dialogue in the SummScreen dataset (Chen et al.) 2022a). Following RoleLLM (Wang al., (023a),, we build role profile and retrieve the top-5 relevant dialogue for each character from the scripts as context for prompting. We then generate rephrased instructions using the same five Basic instructions for each character and calculate word overlap to identify which character has the most distinct style. Considering the diversity in speaking styles, age, and gender, we selected five fictional characters generate the User instructions: Rachel from FRIENDS, Moira from Schitt's 's Creek, Keith from Veronica Mars, and Sheldon from The Big Bang Theory. Since there were no child characters in these TV series, we simulated general child-speaking style rather than that of specific character.
Two notable features emerged in the rephrased instructions. First, the vocabulary size expanded to include more uncommon words, such as "sashaying" " and "meander". Second, additional non- navigation words, such as conversational fillers and character catchphrases, were incorporated. The comparison of different instructions is illustrated in Fig. 55) Adapting agents to these varied speaking patterns beneficial for improving performance the GSA-VL.N task. We include all the prompt templates used in Appendix A.10
A.3 DATASET STATISTICS
In this part, we provide more statistics and visualizations of our GSA-R2R dataset to provide a more comprehensive evaluation.
A.3.1 LINGUISTIC METRICS.
As stated in the main paper, our goal is to generate diverse instructions, making traditional linguistic metrics less suitable, as they compare generated instructions with ground truth instructions rather than with the path itself. Therefore, we provide linguistic metrics here for reference only.
Basic Instructions. To demonstrate the effectiveness of the second stage of our instruction chestration pipeline which utilizes the VLM for instruction refinement, we apply the same method to the paths from the validation unseen split of R2R, allowing for comparison with ground truth
17

---

Published as a conference paper at ICLR 2025
Train
R2R
Basic
Scenees
Usar Foo
waak. divie ooe dight: of Walk stuss Walt at top
Den the roun and tun nast
Alngat so what you'rr ezene do wilk staggt
Okaz, drs, you gata wale the plase when
sters and stop to Iw dair lending.
wodk donn fhe hellwey and
down the tellng, don!! The stull auda lell
gunntge dris stall Kony going dreight dan't hum
aandnng
stop tn moat of the tod
nmm. keep poep unti you see the octie and noce
Thoe, 20 iots the moe rigzt in front of you. Thene
wiste clain.
you itaal, you'll aum rilht upain. Faaly
be shine susida usen wilh mic lescin.
office and daop right
tan tSchool
way
trngg and dopp them! (Catlo)
Louas the noum, anel aaka
Tom dight left and walt
Walk through the doce and
(s, just turn orrend and beat our et the mom chey?
movere the hellwey and descrer the satrraee Italt
lett Take migin at the and
dosss the asso to the back
turn riuit wilk isto the first
TTho,, take lelt and soed divas dae hullway Yadil
the umsaaae of dacskan (Bkidood)
the
T'om night then go aroued
door the rig in strp.
cant keeg gring peot the plas door then
right
ooluenn near the
youtl toom
watte chair Thats when
you
Go us thi stars, and.
Go the kindan ustas
Walk, Suorahh the doorsay
Alisga iasas Fiss, youtu queca walk snuigtt Stot. past Sae bas continas seoenti und ottar the
right Veer right and mnrer
the reaening the jeft then
and tim eet wolk ivi the
aroee, akey?? Ther whees sen the mimors on
mom emsigh aisee "I and yourel good
the salee eu the nggt. Stop
tur Seat and hndd aoward
roou, oa the left and stap it
your aet, ciado left turn Keep mociing stragtt and watt you
rtable claraer with stane bosetes Stnd
orce your dosmany
the soeis and Swers.
mar mendenll bok "gyt tam. trapetiials la before water bour, (Motr)) Keep oxina straght again
ans,, skip riftt
Turn righ and walk:: through
Trm agg walk swarr. slars. right and
Walk oot of the lxundry
so, when you come out the betarces, just go
Stail aata the bedroons, paing blltt, orpiiti and
day clta bis. Timi lum lell
masss una walk
Then, you went andl (sa chil Thacadd
vest lamp before
the
botuut dae
you
vay doer you sau, the doonayy (hoeel)
then
um jome past
skep? smerd spother let the
andll
the ballear
left the stvins and
Juft
dovn
wilk the small stogs Wast thrm.
peot the kinchnn walk hee kitchen and take that A thdt
mon beile, and the kithhen dounwey (Kerh))
enter the vey first soees
the istzica and stop the of Uarorasse Gut (Skky))
see the gosnter just walk dight and stop fruzt
wilh a vielle witk bell
desnway to lae kixias.
and stop thom
Figure 5: Comparison of instructions between R2R and various speaking styles in GSA-R2R. Words that represent the speaking style are underlined. Our instructions demonstrate significantly greater diversity and distinctiveness in speaking styles.
Table 9: Linguistic evaluations of different instruction generation methods on val unseen split of R2R.
Methods
SPICET
BLEU-11
BLEU-4*
Meteort
Rouget
BT- BT-Speakeer (Fried et al. 2018)
18.8
52.2
14.2
22.8
34.6
EnvDrop (Tan et al. 2019)
18.1
68.4
23.7
22.5
45.8
Ours
21.4
69.9
24.0
23.2
45.3
instructions. The results are presented in Tab. 9 For most metrics, our method outperforms the original EnvDrop instructions, highlighting the effectiveness of our approach. However, for Rouge, which is based on word-level matching, the speakers with a fixed vocabulary perform better than our open- vocabullay approach. This outcome is reasonable but does not necessarily reflect the overall quality of the instructions.
Scene and User Instructions. We evaluate Scene and User instructions compared to the Basic instructions to show their differences in Tab. TO The results provide two insights. First, User in- structions are more similar to Basic instructions compared to Scene instructions. This is likely due to the rephrasing techniques used, where Scene instructions include more conversational fillers, lead- ing greater divergence from the Basic style. Second, the five characters exhibit varying degrees of deviation from the Basic instructions. Although they are generated using the same method, the use of role profiles and dialogue history enables the LLM to capture distinct patterns in each character's speaking style, resulting in diverse instructions. This demonstrates the effectiveness of our approach in producing character-specifie instructions.
Navigation Evaluation. We further apply a trained navigation model (DUET) to evaluate the gen- erated instructions in a zero-shot manner, demonstrating the effectiveness of our refinement and rephrasing methods from another perspective. The results are shown in Tab. [ Comparing the speaker-generated instructions in Stage I to the refined Basic instructions in Stage 2, we observe performance increase, indicating that our VLM- based instruction refinement successfully corrects noisy instructions. After introducing diverse speaking styles in Stage 3, both Scene and User in- structions show a performance drop. suggesting that the incorporation of different speaking styles introduces additional challenges for the navigation task.
18

---

Published as a conference paper at ICLR 2025
Table 10: Evaluation of instructions of different speaking styles in GSA-R2R.
Instructions
SPICET
BLEU-11
BLEU-41
Meteort
Rouget
Scene
37.6
39.5
16.2
31.8
51.1
Sheldon
44.1
61.5
33.3
32.1
63.0
Moira
40.6
59.3
30.4
31.3
61.3
User
Rachel
51.9
68.0
40.9
36.9
70.0
Keith
48.4
61.3
33.6
33.7
65.0
Child
42.2
49,4
21.3
35.7
57.6
Table 11: Navigation performance in different instructions on validation splits GSA-R2R.
Val-N-Scene SPL
Val-R-Usee SPL
Val-N-User
Instructions
SR
nDTW
SR
nDTW
SR
SPL
nDTW
Stage
39.2
30.8
43.2
56.4
46.9
56.0
43.9
32,9
42.4
+Stage
42.8
33.5
44.8
59.1
49.1
57.0
46.4
34.8
43.4
+Stage
39.3
31.1
43.6
55.6
46.5
55.6
44.1
33.6
43.6
A.3.2 DATASET SPLITS
In Tab. 12 provide detailed information the splits GSA-R2R. Since we have types of environments and three kinds of instructions, there are five splits for both validation and test sets, as residential houses do not have Scene instructions. Each split contains at least 10 buildings. with 600 instruction-tren instruction-trajeetoryy pairs for each building. In the User splits, each path has five instructions corresponding to five different characters.
A.3.3 DATASET VISUALIZATIONS
In this section, we present additional visualizations of the GSA-R2R dataset to further illustrate its characteristics.
Instruction Length. We show the statistics on instruction length for different instruction types from GSA-R2R the left side of Fig. 6 These results support our observation about the different speaking patterns between Scene and User instructions. Scene instructions, which include conver- sational fillers, tend be the longest, while User instructions make changes primarily at the word level, resulting in lengths similar to the Basic instructions.
Table 12: GSA-R2R splits: each cell shows the split name, the number of scans, and the number of instruction-trajcctory pairs.
Validation
Test
Split
Residential
Non-residential
Residential
Non-residential
Val-R-Basic
Val-N-Basic
Test-R-Basic
Test-N-Basic
Basic
15 scans
10 scans
24 scans
15 scans
9K pairs
6K pairs
14.4K pairs
9K pairs
Val-N-Scene
Test-N-Scene
Scene
10 scans
15 scans
6K pairs
9K pairs
Val-R-User
Val-N-User
Test-R-User
Test-N-User
User
15 scans
10 scans
21 scans
15 scans 45K pairs
45K pairs
30K pairs
12.6K pairs
19

---

Published as a conference paper at ICLR 2025
Instraction Lengeh
0.150
Busic (Avg: 23)
Scane (Aag 43)
0.125
User (wVg 22)
0.100
JM GARLER
G2A42 add
0.975
0.000
0.025
0.000
20
60
10
100
Words
Figure 6: Left: The distributions of the instruction length for different types of instructions in GSA- R2R. Right: The t-SNE analysis of User instructions from other characters in GSA-R2R.
SQUArAIn
SunFoom
HUSE
stroll
Le
g
peacy on
quarters
e
Bight
bunk
Workstat
FOOthS
okay
ney wanmiss
se
ExpOLiE
vehicle
di SSS cans aisles
aays
korkstations
folks
Lightt
sweetie
hi
S a
boaa
d
Aypasspoor
Hulia warry
strol Eleppnng portside
momer Basic
Soene
User (Sheldon)
Figure 7: Word clouds of nouns and verbs absent from the training data across various speaking styles in the GSA-R2R dataset.
I-SNE Visualization for Other Characters. We present the t-SNE visualizations of other char- acters in the User instructions of GSA-R2R on the right side of Fig. Each character's instructions form independent clusters, separate from the training distribution, confirming that our User instruc- tions represent OOD data. We do not combine all users into single figure, as they are based on the same Basic instructions, resulting in similar semantic meanings, which makes it difficult to distinguish between the clusters visualized together.
Word Cloud. In Fig. 7 we present the word cloud visualization showing the nouns and verbs not found in the training data across the three types of instructions in GSA-R2R. The words in the Basic instructions are from the refined results since the trained speaker uses the same vocabulary as the training data. Scene instructions include many conversational fillers such as "Alright" " and "Okay", while the User instructions of Sheldon include more complex and specialized terms like "starboard" and "traverse" These results highlight the significant diversity in GSA-RU --R2R instructions and the distinct differences between various speaking styles.
Trajectory Visualization. We provide qualitative examples of the environments, trajectories, and instructions from GSA-R2R in Fig. 8 The trajectories consist of sequence of panoramas which are the observations from the start point to the destination. Each panorama is numbered in the upper left corner indicate its sequence the path. The direction each image faces shows the origin of movement, while red arrows mark the direction of travel. These visualizations are also used as prompts for the VLM refine the instructions, as well as for participants in human evaluations. From these examples, it is clear that GSA-R2R incorporates diverse range of environments and instructions and provides valuable resources for future VLN research.
A.4 PRACTICAL DEPLOYMENT FEASIBILITY
In this section, we provide detailed analysis of the computational and memory overhead associated in real-world
with GR-DUET to prove that proposed method be effectively deployed systems, such as robotics or autonomous agents. We use the largest environment in GSA-R2R as an example demonstrate the resource requirements of GR-DUET during inference.
20

---

Published as a conference paper at ICLR 2025
User: Traverse straight through the culinary zone into the dining area, enter the hallway ahead, proceed through it, enter the living room, and halt at the fireplace.
Basic: walk straight down the hallway and turn right. walk past the kitchen and turn right. walk into the kitchen and stop.
Scene: Okay, what you need to do is walk straight down the hallway, and then you'll make right turn. Keep going straight past the desk and chairs-don't get distracted!-and then turn left just before you reach the door. Walk straight through the office room that's directly in front of you. Once you get there, just stop.
and head into the
Figure 8: Qualitative examples of the environments, trajectories, and instructions from GSA-R2R.
Memory Requirements GPU Memory: GR-DUET requires peak of 4.3 GB of GPU memory during inference, which well within the capacity modern GPUs. For instance, can be deployed terminal servers equipped with hardware like the NVIDIA Jetson AGX Orin similar devices. CPU Memory: The method requires at most 5.3 GB of RAM, which is easily manageable by most modern roboties platforms.
Computational Overhead Inference Latency: GR-DUET achieves an average inference la- tency of 67 milliseconds per frame, allowing efficient navigation in most real-world environments. Throughput: The system processes 15 frames per second, which is sufficient for environments where navigation speed is moderate and does not require high-frequency updates, such as indoor environ- ments with static obstacles.
The model includes 180 million parameters, occupying ap- proximately 2.1 GB of disk space. Computational Complexity: The model has computational cost of 1.63 GFLOPs (excluding visual feature extraction), making feasible for implementation on robots without imposing excessive computational demands.
Model Characteristics Model Size:
These metrics demonstrate that GR-DUET is highly practical for deployment in real-time systems. Its resource requirements align with the capabilities of roboties platforms such TurtleBot2 and LoCoBot, which have been commonly used in previous works (Anderson et al. 2021 Xu et al. 2023
21

---

Published as a conference paper at ICLR 2025
Table 13: Variations in computational costs of GR-DUET across different episodes.
Episode
100
200
300
400
500
600
Graph Coverage (%)
4.1
68.1
81.1
89.6
94.1
97.4
98.5
GPU memory (MB)
823
2917
3609
3609
3609
3609
3609
CPU memory (MB)
5174
5252
5278
5284
5290
5291
5291
Inference Time (ms)
12
56
63
65
66
66
67
Figure 9: The change curve GPU memory. CPU memory, inference time, and graph coverage in three environments with different sizes from GSA-R2R with episodes.
A.5 MODEL SCALABILITY
In this section, we demonstrate the scalability of GR-DUET larger and more complex environ- ments, focusing on both computational efficiency and navigation performance.
Computational Costs and Memory Usage Although the memory bank in GR-DUET up- dates continuously during navigation, it sta- bilizes once the agent has explored most of the environment. Updates after this point are minimal, involving only new instructions and actions, which require relatively little mem- ory. Moreover, GR-DUET employs coarse- grained embeddings neighbors of the current node, limiting GPU memory growth despite inputting the entire graph into the model. To illustrate this, we analyze key computational metrics across episodes for one of the largest environments in GSA-R2R, as shown in Tab. 13 As agents execute more instructions, we observe grad- ual increases in CPU memory usage, GPU memory usage, and inference time. However, when the graph coverage approaches 100%, indicating that the agent has explored most places, these metrics stabilize with minimal additional overhead. This demonstrates that GR-DUET is computationally scalable and does not sacrifice much memory efficiency for improved performance. Additionally, Fig. presents the trends of these metrics across three environments of varying sizes from GSA-R2R, further sup- porting the scalability of GR-DUET. These results confirm that GR-DUET maintains its efficiency without compromising performance, even in large and complex navigation tasks.
Navigable Areas vs SR Gaps
SR Gaps
nodes that are not
0-
50
100
150
200
250
300
Navigable
Figure 10: The SR gaps between DUET and our GR-DUET versus navigable areas in all 150 envi- ronments from GSA-R2R.
Navigation Performance In large environments, the memory bank grows, introducing more in- puts to the model. However, GR-DUET's pre-training stage on the entire ground- truth graph ensures that it effectively handles these inputs. By focusing surrounding viewpoints rather than the full graph, GR-DUES mitigates the impact of environment size on computational efficiency and perfor- mance. To validate this, compare the performance gap (Success Rate, SR) between the original
22

---

Published as a conference paper at ICLR 2025
Table 14: Mean SR gap between GR-DUET and original DUET by environment size.
Viewpoint Number
50-100
100-150
150-200
200-250
250-300
>300
Mean SR Gap (%)
9.07
10.17
7.57
9.14
8.02
10.74
Table 15: Comparison of LLM-based methods and GR-DUET on GSA-R2R.
Bast-N-Basic SRT
Test-N-Scene SRT
Test-N-User
Methods
SPI.1
SPL.1
SR1
SPL
MapGPT
34.17
29.72
24.67
22.62
23.17
20.80
NavCoT
36.67
34.46
29.00
25.93
26.33
24.47
NavGPT-2
63.50
47.26
56.67
43.34
47.00
36.86
GR-DUET (ours)
74.17
70.45
54.33
47.04
58.00
52.93
DUET and GR-DUET across all 150 environments in GSA-R2R with varying numbers of view- points Tab. [4 The results show that GR-DUET consistently outperforms the original DUET, even in environments with large number of viewpoints. This indicates that GR-DUET effectively learns to focus on relevant parts of the graph, ensuring scalability in large environments. Addition- ally, include scatter plot of SR gaps versus navigable areas in Fig, 10 which further confirms that GR- GR-DUET consistently performs better in all large environments.
A.6 PERFORMANCE OF LLM-BASED METHODS
Since today's 's LLMS have strong capability of reasoning, we incorporate two additional LLM- based methods, MapGPTiChenn al.] 2024a) and NavCoT((inn et al.) (024) to figure out whether they can adapt to different instruction styles without the need for an additional adaption process. Due to the scale of our GSA-R2R dataset, evaluating proprietary LLM-based methods on the entire dataset is computationally expensive. To address this, we sampled one environment for each type of instruction to conduct meaningful comparisons in Tab. 15 The results reveal that LLM-based methods perform poorly on the GSA-R2R task compared to GR-DUET. While LLMs handle instructions with different styles, they struggle with the environmental adaptation required for GSA- VLN, particularly in processing visual information and interacting with persistent environments. We believe one promising direction for future research would be adapting LLM-based methods to specific environments using the memory bank provided by GSA-VLN.
We also test an intuitive idea of whether an LLM could translate styled instructions back into a basic style to facilitate the understand- ing of these styles for navigation models. This is evaluated on the Val-R-Scene split us- ing three sets of instructions: 1. Basic: In- SR
Table 16: Performance comparison of instruction styles on the Val-R-Scene split.
Instructions
Basic
Scene
Translated
46.37
42.30
44.83
structions after Stage 2. 2. Scene: Instruc-
tions transformed from Basic after Stage 3.
3. Translated: Scene instructions translated back into Basic style by LLM. The performance of these instruction types is summarized in Tab. [6] The results reveal that LLM-basee translation improved performance over Scene instructions but does not fully close the gap with Basic instruc- tions. This limitation arises from the open- vecabbaayy nature of LLMs, which introduces noise and leads to information loss, thereby reducing the effectiveness of the approach. It represents a promis- ing direction for future work to solve the instruction style problem, like fine-tuning an LLM-based translator or adding the translated instructions into the training process of the navigation model.
A.7 ADAPTATION EFFICIENCY
In this section, we analyze the adaptation efficiency of GR-DUET. GR-DUET builds a global graph for adaptation, which stabilizes once the agent has explored most parts of the environment. Based on the graph coverage data (as seen in Tab. [3). we find that 90% coverage achieved with at most 400 trrjecttion-rrjjcttorry pairs in large environments and as few 100 pairs in small environments.
23

---

Published as a conference paper at ICLR 2025
To measure adaptation speed, we treat the first X instructions (the number required to reach 90% graph coverage) as the adaptation phase and divide them into groups of 50 instructions. Performance within each group is measured, and linear regression is applied to calculate the slope of performance improvement, serving as a proxy for adaptation speed. Results show that among the 150 scans, 94 achieved positive slope, with mean slope of 0.26, indicating rapid adaptation in most cases.
To understand slower or less effective adaptation, we analyzed adaptation speed across various en- vironmental characteristics. First, Tab. 17 shows the mean adaptation slopes in environments with different numbers of floors. Adaptation becomes less effective as the number of floors increases, except for few cases with four floors. This is intuitive. as distinct floor layouts and styles make prior memory from other floors less relevant to the current navigation. Conversely. Tab. 18 shows that adaptation efficiency improves as the number of rooms increases, particularly in buildings with more than 15 rooms. After viewing specific buildings, we find that environments with many rooms (e.g., hotels or student accommodations) often have repetitive layouts and identical rooms, allowing the agent to leverage memory from similar spaces effectively. These findings suggest that GR-DUET performs well environments with repetitive structures but struggles in environments with dissim- ilar memory (e.g., multi-floor buildings with distinct layouts). We also calculate mean adaptation slopes for different scene and instruction types, as shown in Tab. 19 From the results, we can see that GR-DUET adapts faster in residential environments than in non-residential environments. This is likely due to the training environments being predominantly residential (from R2R), introducing bias that favors residential scenarios. For different instructions, agents adapt fastest to Scene structions and least effectively to User instructions, Scene instructions often include conversational fillers, which provide more distinct language patterns than the word variations User instructions, making them easier to adapt to. These analyses highlight both the strengths and limitations of GR- DUET's adaptation mechanism. While it excels in environments with repetitive layouts, it struggles in multi-floor floor or irregular environments. Future improvements could focus on leveraging dissimilar memories (e.g., between floors) and reducing training biases to enhance adaptation more diverse scenarios.
Table 17: The adaptation speeds of GR-DUET in environments with different number of floors.
Floor Number
1
3
4
Mean Slopes
0.24
0.19
0.05
0.40
Table 18: The adaptation speeds of GR-DUET in environments with different numbers of rooms.
Room Number
1-5
5-10
10-15
15-20
20-25
20-25
>25
Mean Slopes
0.48
0.04
0.06
0.20
0.28
0.36
0.70
Table 19: The adaptation speeds of GR-DUET in different types of scenes and instructions.
Type
Residential
Non-residential
Basic Inst.
Scene Inst.
User Inst.
Mean Slopes
0.34
0.18
0.21
0.55
0.19
A.8 HUMAN STUDY DETAILS
In this section, we provide more details about the human study conducted in Tab. 2 prove its reliability. Our study included 15 participants, comprising university students and staff aged between 20 and 35 years old. The participants represented diverse range of genders and backgrounds, ensuring a variety of perspectives while maintaining a degree of homogeneity necessary for unbiased evaluations. Two tasks were given to the participants and both of the tasks were straightforward and did not require specialized knowledge. In task I, the participants need to determine whether an instruction aligns with the corresponding trajectory. Task 2 requires them to assess whether the instruction exhibits a distinct speaking style. Given the simplicity of these tasks, that our participants were competent to perform the evaluation accurately and reliably. We provide the example figure of the user interface used in the human study in Fig.
confident
24

---

Published as a conference paper at ICLR 2025
due pndiegui Alpert
oigeet the
regit heg pou togrr tee
sengee
Figure 11: Examples of the problems for the participants in the human study.
A.9 INSTRUCTION DIVERSITY ANALYSIS
Although we only generate User instructions with only five characters, our approach can be extended generate instructions for thousands of characters. Specifically, by utilizing the SummScreen dataset (Chen et al., 2022a), our method can scale up to approximately 3,000 unique characters, which we believe provides sufficient diversity for research purposes. Since our work is the first to propose the problem of instruction style adaptation, the primary aim is to establish the existence of instruction adaptation as a problem and provide meaningful benchmark to test the adaptation capabilities of current methods.
To achieve these goals, we find that using five characters appears to sufficient, as evidenced by the results in Tab. 5 To further justify this, calculate the word overlap rate between the instruc- tions of the five selected characters and the remaining 173 candidate characters. This calculation based five example instructions. The results in Tab. 20 demonstrate that selected char- acters already cover a broad range of language patterns. Adding more characters would slightly increase the dataset's scope but is unnecessary for addressing the problem establishing robust benchmark. Notably, the character with the least overlap rate still shares a 65% overlap with our selected characters, highlighting the diminishing returns of including additional characters. For comparison, in the widely used R2R dataset (Anderson et al., 2018b), the evaluation split includes only 11 scans, far fewer than would exist in real-world scenarios. Yet, it is broadly accepted standard benchmark for evaluating navigation capabilities. Similarly, our choice of five characters maintains practicality while ensuring meaningful evaluation. Moreover, our dataset includes Scene instructions. which encompass styles influenced by professional language and roles, further broad- ening the diversity speaking styles. This additional dimension ensures that the five selected User instruction styles are adequate for evaluating adaptation methods.
Table 20: Word overlap rate between our characters and the remaining 173 characters.
Instruction
3
Average
Mean Overlap
0.90
0.93
0.81
0.90
0.82
0.87
25

---

Published as a conference paper at ICLR 2025
Prompt Template for G Generating Basic Instructions
System Instruction:
You are un expert in annotating instructions for navigation paths.
path is presented bwo formats:
1.A sequence of 360-degree panoramic observations from the start to the goal location, each with a red arrow indicating the direction of movement and red number on the top left comer indicating the step number, ending with penorama at the goal location.
A map image with green blocks representing discrete viewpoints red arrww indicating movement directions. Gray areas on the map indicate possible movement regions.
Glven the instruction: (Initial instruction)
Your lasks are:
Identify any emons violations the instruction based the following criteria:
The instruction must be concise, natural, and complete without being split into steps.
The instruction must specify unique, unmistakable goal location.
The instruction must avoid any ambiguity
might mislead robot.
The instruction musl match acily the map, especially the directions of movement.
2. Improve the instruction based on your analysis so that a smart robot can find the goal location after starting from the same start
location using only the imprved instruction.
You will be penalized Provide the analysis and improved instruction directly without additional commentary in the format: Analysis: xax. Improved Instruction: xXx.
new instruction: does satisfy the critetia.
User Prompt:
(panoramas along the path): (lop-dose down map image of the palb)
Figure 12: Prompt template for classifying buildings from HM3D.
As for the specific characters, we selected them based on their highly distinct speaking styles, ensur- ing diversity in age, gender, and language patterns. To measure the diversity of speaking styles, we generated rephrased instructions for each character using the same five Basic instructions and cal- culated the word overlap rate. Lower overlap rates indicate more distinct language patterns. These results, along with the characters' rankings among all characters in SummScreen, are presented in Tab. 22T, The overlap rates and rankings confirm that the chosen characters exhibit distinct and diverse language patterns, making them ideal representatives for User instructions. This diversity ensures that our dataset effectively challenges VLN models to adapt to different speaking styles.
Table 21: Word overlap rate and ranking of four selected characters in GSA-R2R among all charac- ters in SummScreen.
Character
Keith
Moira
Rachel
Sheldon
Overlap Rate
0.44
0.28
0,42
0.30
Ranking
Sth
1st
6th
2nd
A.10
PROMPT TEMPLATES
In this section, we provide the prompt templates used in our instruction orchestration pipeline:
Fig. 12
Prompt template for classifying buildings from HM3D;
Fig. 13
Prompt template for refining speaker-generated instructions into Basic instructions
(stage
of the instruction orchestration pipeline);
Fig. 14
Prompt template for rephrasing Basic instructions into Scene instructions;
Fig. 15
Prompt template for rephrasing Basic instructions into User instructions (exclud-
ing Child);
Fig. 16 Prompt template for rephrasing Basic instructions into User instructions with the Child Character;
We directly use the prompt templates for generating role files and dialogue history from RoleLLM (Wang al. 2023a).
26

---

Published as a conference paper at ICLR 2025
Prompt Template for Classifying ing Buildings
System Instruction:
You are helpful assistant in deternining the building type accorting to the provided images of the building from following candidate types: (187 building types).
All images refor to the same building.
Only give me the building type without additional information.
User Prompt:
(overview Image for building); ); (top-down views each floor); (representative panorames)
Figure 13: Prompt template for refining speaker-generated instructions into Basic instructions.
Prompt Template for G Generating Scene Instructions
System Instruction:
will provide a nav igation instruction and the comesponding building ly Your task is to transforn the instruction to match the speaking style appropriate for specified building type and typical users or visiters. Follow stops:
1. identify typical users visitors the building pe r.g. hotel: hotel owner, waiter, guest).
2. R Randomly select one of these users or visitors as the speaker:
3. Modify the instruction to match the selected speaker's conv enaaaiaational style, incorporaing minor elaborations conversational
fillers (e.g. you know', let's etc.) enbance context and make ** distinct possible from the original**
Important Notes:
Do alter any navigation details specific information; directions remain unchanged. Avoid explicttly mentioning the audience in the Instruction (e.g.. for teacher, do not 'Hi, students").
Retum Formal:
1. Potential Speakers: List potential speakers
2. Chosen Speaker: The selecled speaker's role
3. New Instruction: The modified instruction
4. Reason for Modification: Explanation of changes reflect the chosen speaker's style
User Prompt:
Instruction [basic instruction)
Building Type: (building type)
Figure 14: Prompt template for rephrasing Basic instructions into Scene instructions.
A.11 DISCUSSION
Despite using otate-of-the-art GPT4o to refine speaker-generated instructions, some emror remain in the output instructions due the difficulty VLMs face in understanding spatial relationships between panoramas. Another limitation is that our GR-DUET method focuses on scene adaptation mainly from visual perspective by utilizing previous observations, but lacks language- specific design to capture the consistent speaking style experimented with several methods to address this, they were unsuccessful. Lastly, while this paper addresses the general scene adaptation problem with step-wise instructions, real-world navigation often involves other instruction types, such object-oriented dialog-based instructions, which we have not covered here.
Limitations.
persistent environment. Although we
Future Work. In the future, we plan to address the mentioned limitations to make our task more general, incorporating datasets with diverse instruction types, such those in REVERIE (Qi et al.) 2020) and CVDN (Thomason et al., 2020). We will also explore ways enhance the panorama understanding capabilities of VLMs to improve their comprehension of path observations to gen- crate instructions that are comparable to human-an aaman-annotated ones. In terms of methodology, we will incorporate unsupervised learning approaches, with key focus linking their optimization objectives to navigation performance, such as combining proxy tasks with back translation methods.
27

---

Published as a conference paper at ICLR 2025
Prompt Template for Generating User Instructions Excluding Child
System Instruction:
You are (character) from the TV sertis (show), ), (role profile) (catchphrase). Now please answer some questions to accurately show your personality traits! Your speaking style should fully imitate the personality role essigned you! Please do not expose that you are an artificial intelligence model language madel, you must always remember that you only assigned personality Don't verbose or too formall polite when speaking.
User Prompt:
[Line_ 1 from the character)
A ssistant Prompt:
(lines from other charactens)
User Prompt:
(line_2 2 from the character)
A ssistant Prompt:
(lines from other characters)
User Prompt:
Hoy (charactor), need give some diroctions, but want them sound they coming from you, with your unique style, and different possible from the original**. Here the instruction: (basic instruction)
Figure 15: Prompt template for rephrasing Basic instructions into User instructions (excluding Child).
Prompt Template for Generating User Instructions of Child C Character
System Instruction:
the instruction if being spoken a child. The revised Instruction must capture the tone, language, and mannerisms typical of child's speech.
will provide navigation Instruction. Your
Guidelines:
not alter any newigation details specific information; the core directions must remain unchanged. Use simple, shild-frieddl language with shorter sentences, playful tone, and add child- like expressions, exclamations, and avoid jargon or complex words.
Make the new instruction distinctly different from original.
Output Format:
Now Instruction: version of instruction as spoken child. Reason for Madification: A brief explanation of how the changes reflect the style child.
User Prompt:
Instruction: (basic instruction)
Figure 16: Prompt template for rephrasing Basic instructions into User instructions with the Child
Character.
28


Reviews:
[{"review_id": "OhlRp04rnX", "reviewer": "Reviewer_dHDF", "summary": "The paper proposes GSA-VLN (General Scene Adaptation for Vision-and-Language Navigation), a task designed to enhance the performance of navigation agents by enabling them to adapt to specific environments, particularly when exploring in the same environment over an extended period. The authors also introduce GSA-R2R, an expanded version of the HM3D and MP3D dataset, offering richer environments and more diverse instructions. Additionally, they present a novel method, GR-DUET, which improves navigation performance by utilizing memory mechanisms and updating graph structures.", "questions": "1. Memory Mechanism Scalability:\nWhile the memory-based approach in GR-DUET performs well in your experiments, how does this method scale to larger or more complex environments? As the environment size or the number of instructions increases, the memory bank may become too large to manage efficiently. Could you provide further analysis or experiments that demonstrate how the method performs with continuous accumulation of data in larger datasets or more complex environments?\n\n2. the paper lacks a detailed discussion on how the memory is utilized, including how similar tasks are stored, how memory is retrieved and assessed for relevance and validity, and how prior knowledge is leveraged. Is the memory bank pre-set or updated dynamically? If it is updated dynamically, how is the correctness of the stored memories ensured, especially when handling diverse memories? How are the initial model parameters (L194, L198) initialized to ensure sufficient generalization? Please provide more details\n\n3. Furthermore, other memory-based VLN methods, such as SG-Nav [1], provide more detailed storage and query mechanisms based on topological graphs and memory updates. Could you compare your approach with SG-Nav in terms of performance or highlight any differences and advantages?\n\n4. Adaptation to Instruction Styles:\nYou mention using GPT-4 and a three-stage process to generate different instruction styles, but it remains unclear how the agent adapts to these varying styles over time. Could you provide more quantitative and qualitative results on how GR-DUET handles changes in style, particularly in OOD environments? A deeper analysis of how different speaking styles affect agent performance and adaptability would offer valuable insights into the robustness of your method in real-world scenarios, where user communication patterns may vary significantly.\n\n5. Unsupervised Learning and Adaptation Efficiency:\nThe paper suggests that agents in GSA-VLN can improve their performance over time using unsupervised learning techniques. Could you clarify how quickly the agents adapt in different environments? Are there any cases where adaptation is less effective or slower? Are there specific environments where the memory mechanism struggles to adapt? A more detailed breakdown of adaptation speed and efficiency across different environment types would help clarify the limitations of your approach and guide future improvements.\n\n6. Practical Deployment and Real-World Use Cases:\nThe GSA-VLN task is well-motivated by real-world scenarios, but the paper does not provide a detailed discussion on how the proposed method could be deployed in practical systems. Could you elaborate on the computational and memory overhead of your approach in real-time systems, such as those used in robotics or autonomous agents?\n\nReference:\n[1] Yin, Hang, et al. \"SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation.\" arXiv preprint arXiv:2410.08189 (2024).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Novelty:\nThe paper introduces a new task, GSA-VLN, which focuses on the long-term adaptation of agents within specific environments, a capability with significant potential for real-world applications.\n\nDataset Contribution:\nThe authors present the GSA-R2R dataset, which extends the existing R2R dataset by using GPT-4 and a three-stage method to generate instructions in various speaking styles. The dataset is divided into residential and non-residential environments, serving as in-distribution (ID) and out-of-distribution (OOD) data, respectively.\n\nMethod Design:\nThe GR-DUET method integrates topological graphs with memory mechanisms, effectively preserving historical information and updating it continuously during navigation. This approach demonstrates notable improvements in performance, particularly in OOD (non-residential) scenarios.\n\nExperimental Results:\nThe paper compares GR-DUET with optimization-based and memory-based methods across different environment and speaking style splits. The experiments highlight the feasibility of the GSA-VLN task and the effectiveness of the GR-DUET method in various settings.", "weaknesses": "Please see the Questions section for detailed improvement suggestions and questions.\nI look forward to the authors' responses to these questions, as addressing these points could significantly clarify some of the paper's contributions and limitations. I am open to adjusting my score if the authors provide further insights or resolve the concerns raised above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R8fHVohdRZ", "reviewer": "Reviewer_t9oJ", "summary": "This paper presents General Scene Adaptation for Vision-and-Language Navigation (GSA-VLN), a new VLN task where agents adapt to and improve in a specific environment over time, making it closer to real-world applications. To support this, the authors introduce GSA-R2R, a dataset that expands on Room-to-Room (R2R) by adding more diverse environments and instruction styles, including out-of-distribution examples. They also propose Graph-Retained DUET (GR-DUET), a method that uses memory-based navigation graphs and scene-specific training to help agents learn and retain scene-specific information, achieving strong results on the GSA-R2R benchmarks.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper introduces the novel General Scene Adaptation for Vision-and-Language Navigation (GSA-VLN) task, filling a critical gap in VLN research by focusing on adaptation in persistent environments. Rather than assuming agents will encounter only unseen environments, GSA-VLN models a more realistic scenario where agents learn and improve over time within a familiar setting. This shift in task formulation is both timely and innovative, especially as VLN moves toward practical applications.\n2. The paper demonstrates rigorous methodology in creating the GSA-R2R dataset, expanding on the Room-to-Room (R2R) dataset with a variety of environments, instruction styles, and out-of-distribution examples to thoroughly test agent adaptability. The proposed Graph-Retained DUET (GR-DUET) model is well-designed, combining memory-based navigation graphs with a scene-specific training strategy, and shows significant performance improvements across metrics. \n3. The paper is clearly organized and effectively conveys the importance of long-term scene adaptation in VLN.", "weaknesses": "1. The GR-DUET method involves a memory bank and a global graph that retains historical information across episodes. As the memory and graph size increase, the model’s computational requirements may grow significantly, particularly for long-term navigation in large environments. While the paper includes an environment-specific training strategy to limit graph expansion, providing an analysis of computational costs and potential trade-offs between memory retention and scalability would strengthen the model's practicality for deployment on resource-constrained systems.\n2. While the GSA-R2R dataset is a notable improvement over existing datasets for testing scene-specific adaptation, it may still fall short in representing the full diversity of real-world environments and interaction styles. The dataset includes a mix of residential and non-residential scenes, but further validation with a broader set of real-world environments could strengthen the model's applicability. Including additional scene types, such as commercial or outdoor spaces, or testing in dynamic environments where the layout changes over time, would push the dataset closer to real-world settings.\n3. Although the paper’s three-stage instruction generation pipeline enhances instruction diversity, more detailed analysis on how different instruction styles (e.g., Basic, User, Scene) impact agent performance would be valuable. For instance, specific ablation studies on each instruction type could clarify how robust the GR-DUET model is to variances in language, phrasing, and style. Additionally, investigating how the model generalizes across speakers with different dialects or levels of detail in instructions could provide actionable insights into improving instruction handling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UG1hasNYij", "reviewer": "Reviewer_WxGN", "summary": "This paper presents a task that requires the VLN agent to execute VLN task in only one environment while storing its historical information at the same time. To make the initial parameters of the agent more general, the authors generate more environments and more instructions by using LLM. Finally, the paper also provides some experiment results according to their proposed metrics to further highlight the efficacy of the proposed methods.", "questions": "Line 283-286, how is the navigation model used here implemented? How can we ensure that it is a good instruction discriminator? I am concerned that if the navigation model is not trained well, it will not be enough to explain the quality of the instruction.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1、This paper is written with details and clear presentations, easy to follow.\n\n2、The author solves the VLN problem from a new perspective and divides the scenarios into Residential and Non-Residential.", "weaknesses": "1、\tThe novelty of this paper is limited. The GSA-VLN TASK proposed in the paper is still a standard VLN task. The so-called “standard VLN task” mentioned by the paper also includes fine-tuning based on historical information and trained models, which are claimed as the novelty of GSA-VLN in Section 3.2. \n\n2、\tFollowing the previous comment, the GSA-R2R DATASET proposed in the paper uses more environments (HM3D), and then uses tools such as LLM to refine the dataset's quality, which has been a common practice in VLN. Also, the author should not choose to ignore the existing works (e.g., HM3D-AutoVLN[1], Scale VLN[2], YouTube-VLN[3]) have also expanded and refined the VLN dataset when comparing (Table 1). I recommend the authors compare the datasets mentioned above and include them in the main manuscript (e.g. in Table I).\n\n[1] Learning from Unlabeled 3D Environments for Vision-and-Language Navigation\n\n[2] Scaling Data Generation in Vision-and-Language Navigation\n\n[3] Learning Vision-and-Language Navigation from YouTube Videos\n\n3、The comparison metrics in the experimental part are all newly proposed by the authors, which cannot correctly reflect the effectiveness of the method proposed. I suggest that the authors conduct experimental comparisons in standard VLN using common VLN metrics, and compare them on other VLN datasets besides R2R, such as REVERIR, RxR, CVDN and SOON.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F6husP322C", "reviewer": "Reviewer_J6Xq", "summary": "This paper proposes a new task, named GSA-VLN, which require agents to execute navigation instructions within a specific scene and simultaneously adapt to it for improved performance over time. This paper also proposes a new datast, GSA-R2R, which significantly expands the diversity and quantity of environments and instructions for the Room-to-Room (R2R) dataset to evaluate agent adaptability in both ID and OOD contexts. The biggest difference between the proposed task and dataset and previous work is the diversity of instructions, i.e., different individual features and linguistic conventions are taken into account.", "questions": "In section 3.3.4, the authors invite 15 participants to evaluate the instructions. Are the backgrounds (e.g., ages, etc.) of these 15 participants sufficiently homogeneous to demonstrate the refinement of the assessment? Also, I recommend disclosing the questionnaire they used for the test.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed GSA-VLN task and  GSA-R2R dataset which considers real-world robot adaptation in persistent environments, is an interesting research direction.\n2. Overall, the writing is fluent and the figures convey the character of the task well.\n3. The proposed GR-DUET method outperforms the baselines, demonstrating its effectiveness in helping agents adapt to specific environments.", "weaknesses": "1. Some baselines are missing. I suggest to add some baseline methods based on LLM, especially the Zero-shot VLN methods. For example, InstructNav[1], NavCot[2]. The reason for this suggestion is that LLM's reasoning is now so powerful that it may be able to adapt well for different personal characteristics and language styles without the need for an additional adaption process, i.e., in zero-shot manner.  Also, these different styles are essentially generated by LLM, so I'm concerned that these understanding these different styles is a very easy and undifferentiated thing for LLM to do. \n2. In this paper, the authors generated instructions for only five different character styles. However life can be much richer in terms of characters. The paper's contribution would have been greatly enhanced if the authors could propose a method for generating instructions for different character styles in a nearly infinite number of ways.\n3. The authors propose GR-DUET, but there are few details about it. For a reader who does not know DUET well, it may cause some difficulties in reading, and I suggest the authors to add some descriptions and details in the appendix.\n\n\n\n[1] InstructNav: InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment.\n\n[2] NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GTLWmpPQjC", "reviewer": "Reviewer_J7gj", "summary": "The paper presents a novel task, GSA-VLN (General Scene Adaptation for Vision-and-Language Navigation), which trains agents to follow navigation instructions within a specific scene while adapting to it for enhanced performance over time. A new dataset, derived from the HM3D dataset, is introduced to support this task. Additionally, the authors propose a new method that serves as a baseline and achieves state-of-the-art performance in this domain.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally well motivated and the proposed tasks makes sense. For a pre-trained VLN agent, it is important to leverage the information and instructions in the new environment to further enhance it's knowledge and adapt to the new environment and uses.\n\n- A new dataset GSA-R2R based on the HM3D dataset is introduced with new instruction data collected to support the VLN task. The dataset can potentially be useful for the community.\n\n-  Extensive evaluation of current VLN methods on the new dataset and different adaption methods are benchmarked. The proposed GR-DUET method demonstrates competitive performance compared to prior work.", "weaknesses": "- The concept of adapting a model to a test scene is not entirely new, as numerous prior methods have explored unsupervised exploration or adaptation within embodied environments.\n\n- The Related Work section could be more comprehensive. For instance, some discussions are postponed to Section 4, but it's crucial to review prior work that also employs adaptation methods in VLN, particularly those utilizing memory-based approaches, and also highlight the main differences and contributions of the proposed method.\n\n- Additionally, beyond the VLN literature, how does the proposed method relate to Lifelong Learning and Test-time Adaptation?\n\n- Table 3 presents the navigation performance of various VLN models on the new dataset. Is the performance of these methods consistent with results on other benchmark datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)