Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation
Authors: Xiang Fang
Wanlong Fang
Changshuo Wang
Venue: NeurIPS 2025 poster ()
Keywords: Hierarchical Semantic-Augmented Navigation
Primary area: applications

Abstract: Vision-Language Navigation in Continuous Environments (VLN-CE) poses a formidable challenge for autonomous agents, requiring seamless integration of natural language instructions and visual observations to navigate complex 3D indoor spaces. Existing approaches often falter in long-horizon tasks due to limited scene understanding, inefficient planning, and lack of robust decision-making frameworks. We introduce the \textbf{Hierarchical Semantic-Augmented Navigation (HSAN)} framework, a groundbreaking approach that redefines VLN-CE through three synergistic innovations. First, HSAN constructs a dynamic hierarchical semantic scene graph, leveraging vision-language models to capture multi-level environmental representations—from objects to regions to zones—enabling nuanced spatial reasoning. Second, it employs an optimal transport-based topological planner, grounded in Kantorovich's duality, to select long-term goals by balancing semantic relevance and spatial accessibility with theoretical guarantees of optimality. Third, a graph-aware reinforcement learning policy ensures precise low-level control, navigating subgoals while robustly avoiding obstacles. By integrating spectral graph theory, optimal transport, and advanced multi-modal learning, HSAN addresses the shortcomings of static maps and heuristic planners prevalent in prior work. Extensive experiments on multiple challenging VLN-CE datasets demonstrate that HSAN achieves state-of-the-art performance, with significant improvements in navigation success and generalization to unseen environments.

Hierarchical Semantic-Augmenteed Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation
Xiang Fang
School Software Engineering, Huazhong University of Science and Technology
xfang9508dgmail.com
Wanlong Fang Interdisciplinary Graduate Programme
Changshuo Wang"
University College London
Nanyang Technological University, Singapore vanlongfang@@maii com
naugchangshuol@gaill. com
Abstract
Vision-Language Navigation in Continuous Environments (VLN-CE) poses a formidable challenge for autonomous agents, requiring seamless integration of nat- ural language instructions and visual observations to navigate complex 3D indoor spaces. Existing approaches often falter in aanghorizon tasks due to limited scene understanding, inefficient planning, and lack of robust decision-making frame- works. We introduce the Hierarchical Semantic-Augmented Navigation (HSAN) framework, groundbreaking approach that redefines VLN-CE through three syn- ergistic innovations. First, HSAN constructs dynamic hierarchical semantic scene graph, leveraging vision- languae models to capture multi-level environmen- tal represeetattonss-ffoo objects to regions zones-enablle reasoning. Second, it employs an optimal transport-based topological planner, grounded in Kantorovich's 's duality, to select long-term goals by balancing semantic relevance and spatial accessibility with theoretical guarantees optimality. Third, graph-aware reinforcement learning policy ensures precise low-level control, nav- igating subgoals while robustly avoiding obstacles. By integrating spectral graph theory, optimal transport, and advanced multi-modal learning, HSAN addresses the shortcomings of static maps and heuristic planners prevalent in prior work. Extensive experiments multiple challenging VLN-CE datasets demonstrate that HSAN achieves state-of-the-art performance, with significant improvements in navigation success and generalization unseen environments.
nuanced spatial
Introduction
Vision Vision-Language Navigation (VLN) has emerged a pivotal challenge the intersection of computer vision, natural language processing, and robotics, with profound implications for autonomous systems
in seal-world environments Park and Kim [2023]. Francis et al. (2022] Fang et al. [225al 2023a
2022
2023b 2025b Fang and Fang 20026]
Fang et al. 22266a b 2025c 2024a 2025d 2024b c
2023c
202
2025t 2020 202 2024d Fang and Hu [2020] Fang et al. [0026]] In VLN,
agent must navigate through a 3D en environment Chen et al. 2025 typically an indoor space, by
interpreting and following natural language instructions Zhou al. (2024), Chen al. (2024), such
as "Walk down the hallway, turn right at the plant, and stop at the third door on your left." These
*Corresponiigg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

instructions require the agent to integrate cameras and textual directives-to reason about spatial relationships, recognize landmarks, and execute a sequence of actions to reach a specified target Yu et al. (2024). The task is particularly challenging due to the complexity of indoor environments SSahyamoorthy et al. [2024]. [Chen et al.)
imputs-vissnn observations from RGB-D
[2024), which often feature cluttered layouts [Li et al. [2024), partial observability, and ambiguous instructions that demand contextual understanding Wang et al. [2024], WWi et al. [2024]. VLN serves
as critical testbed for developing intelligent agents capable of human-robot interaction Tonk et al.
[002). Francis et al. 2022]. Bhaatt et al. [2022). with applications ranging from assistive robotics in
homes to autonomous exploration in large facilities Szot et al. [2021 Du et al. 2020 Nagarajan
and Grauman 2020].
The VLN task has evolved significantly since its inception. with early works focusing on discrete
navigation graphs Krantz et al. [2000] /Zhan et al. [2024), Wang et al. [2022], where agents select
actions from predefined set of navigable nodes Krantz et al. [2023]. Wang et al. [2023). Recent
advancements have shifted toward Vision-Language Navigation in Continuous Environments (VLN-
CE) An et al. [2024). Yue et al. [2024). which requires agents to operate in 3D meshes with low-level actions Cheng et al. [2024], such as moving forward by 0.25 meters or rotating by 15 degrees Zhao et al. 2025], Xu et al. [2023. This shift introduces greater realism but also amplifies challenges, including the need for precise obstacle avoidance, robust long-horian morghhoizoon planning, and fine-grained scene understanding. Benchmarks like R2R- 32R-CE Krantz et al. [2020] and RxR-CE Ku et al. [2020) have standardized the evaluation of VLN-CE, leveraging datasets such as Matterportt3 D/Cang et al. 2017] to provide rich. photorealistic environments for training and testing.
Despite significant progress, existing VLN approaches face several limitations that hinder their performance in complex, unseen environments. First, many methods rely on static navigation graphs or precomputed maps, which are often unavailable in real-world settings and fail to adapt dynamically to new observations Chaplot et al. [020)]. Wu et al. [2024). Second, traditional reinforcement learning (RL) and imitation learning (LL) approaches struggle with long-horizon tasks due to sparse
rewards and the combinatorial complexity of action sequences Schulman et al. [2017]. Ross et al.
[201T]. Third, while recent works have incorporated Dision-languse models (VLMs) to enhance instruction understanding Liu et al. [2023]. these models often lack structured representations of the environment, leading to incfficient planning and poor generalization to novel scenes. For instance, methods that process raw visual observations without hierarchical context may overlook critical spatial relationships, such as the functional roles of rooms or the connectivity between regions Georgakis et al. [002]] Moreover, the absence of rigorous mathematical frameworks in many VLN systems limits their ability to optimize decisions under uncertainty, particularly when balancing semantic alignment with spatial constraints.
To address these challenges, we propose the Hierarchical Semantic-Augmente Navigation (HSAN) framework, novel approach to VLN-CE that integrates advanced scene understanding, dynamic planning, and robust control. HSAN is motivated by the need for a scalable and adaptive system that reason over complex environments while leveraging the powerful multimodal capabil- ities of modern VLMs. Our framework draws inspiration from cognitive models of human navigation, which rely on hierarchical representations of space-free facilitate efficient decision-making Kuipers) (000). By combining these insights with cutting-edge mathematical tools, such as optimal transport theory and graph spectral analysis, HSAN offers principled solution to the VLN-CE task.
objects to regions entire zones-to
The HSAN framework introduces three key innovations that distinguish it from prior work: 1) Hier- archical Semantic Scene Graph Construction: HSAN dynamically builds a multi-level scene graph that captures objects, regions, and zones, using VLMs to generate rich semantic descriptions. This hicrarchical representation enables fine-grained reasoning about environmental context, overcoming the limitations of flat or static maps used in methods like Chaplot et al. [2200] Chen al. 2022)
2) Optimal Transport-Band Based Topological Planning: We formulate long-term goal selection as an optimal transport problem, balancing semantic relevance the instruction with spatial accessibility. This approach, grounded in Kantorovich's 's duality Villani [2088). provides a mathematically rigorous mechanism for decision-making, unlike heuristie-based planners in Wu al. (2024), Krantz et al.
2022). 3) Graph-Aware Low-Level Control: HSAN employs a graph-aware RL policy, trained with Proximal Policy Optimization /Schulman al. [0077]. execute high-level plans while avoiding obstacles. The policy leverages subgraph embeddings to capture local topology, improving robustness compared to traditional controllers Gervet al. 20233.

---

These innovations are supported by comprehensive training pipeline that combines pre-tra trainini on large-scale datasets, fine-tuning with student-foreing Krantz et al. (2020), and inference strategies optimized for real-time performance. HSAN's s use of optimal transport and graph- methods not only enhances navigation efficiency but also provides theoretical guarantees of optimality, as demonstrated by our proofs of convergence and stability.
Our contributions can be summarized as follows: 1) We introduce HSAN, novel VLN-CE framework that integrates hierarchical scene understanding, optimal transport-based planning, and graph-aware control, addressing key limitations in existing methods. 2) We propose a dynamic hierarchical se- mantic scene graph, constructed using VLMs and spectral clustering. to enable robust environmental reasoning. 3) We develop an optimal transport- based planner that optimizes goal selection with theo- retical guarantees, leveraging Sinkhorn's 's algorithm Cuturi (2013) for computational efficiency. Also, we design a graph-aware RI. policy for low-level control, enhancing obstacle avoidance and subgoal navigation in continuous environments. 4) We conduct extensive evaluations on standard VLN-CE benchmarks, showing state-of-the-an performance and generalization to unseen environments.
2 Related Work
Vision-Language Navigation in Continuous Environments (VLN-CE). The shift to VLN-CE, introduced by datasets like R2R-CE Krantz et al. [2020) and RxR-CE Ku et al. [2020]. addresses the limitations of discrete navigation by requiring agents to execute low- actions (c.g., move forward 0.25m, rotate 159) in 3D meshes. This paradigm, supported by simulators like Habitat (aavaa et al. 2019 better reflects real-world navigation challenges. Early VLN-CE methods, such as Cross-Modal Matching Krantz et al. 2020). adapted discrete techniques to continuous spaces but struggled with long-horizon planning and obstacle avoidance. Subsequent works, like Waypoint Models (Gervet et al. (023) and Neural Topological SLAM|Chappos et al. (2020). introduced intermediate goal prediction and topological maps to improve navigation efficiency. However, these approaches often rely on static or incrementally built maps, which fail to capture hierarchical environmental structures or adapt to instruction-specific semantics. HSAN overcomes these limitations by dynamically constructing hierarchical semantic scene graph. enabling fine-gran tine-graied reasoning over objects, regions, and zones, and integrating optimal transport-baan based planning for robust goal selection.
Jision-Language Models in Navigation. The advent of vision langua-a models (VLMs), such as CLIP Radford al. [0211] LLaVA Liu et al. (2023), and SigLIP /Zhai et al. [2033] has rev- olutionized multimodal tasks, including VLN. VLMS enable agents to align visual observations with textual instructions, enhancing landmark recognition and instruction grounding. For instance,
VLN-BERT Majumdarr et al. 2020) and LLaVA-Nav Shah et al. [2023) leverage VLMs to score candidate paths or generate semantic descriptions of observations. While powerful, these methods often process observations in a flat manner, lacking structured representations of the environment, which hinders their ability reason about complex spatial relationships. Recent works, such as Cross-Modal Memory Networks Srorkkkke et al. [2022]. attempt to incorporate auemory-augmented architectures but focus on short-term context rather than long-term hierarchical understanding. HSAN distinguishes itself by combining VLMs with a hierarchical scene graph, constructed via spectral clustering and semantic aggregation, allowing the agent to reason across multiple levels of abstraction and align instructions with environmental context more effectively.
Novelty of HSAN. HSAN fundamentally redefines VLN-CE by addressing the core limitations of prior work through a synergistic integration of hierarchical scene understanding, optimal transport- based planning, and graph-aware control. Unlike discrete VLN methods Anderson et al. [2018] Chen
et al. 2021a]. HSAN operates in continuous spaces without relying on predefined graphs, making it suitable for real-world applications. Compared to VLN-CE approaches Krantz et al. [2020] Chaplot let al. 22220). HSAN's hierarchical semantic scene graph provides a richer, multi-level representation of the environment, capturing objects, regions, and zones with VLM-generated semantics. While VLM-based methods Shhah et al. [2023). Majumdar et al. (2020) excel at instruction grounding, they lack HSAN's s structured reasoning over hierarchical graphs, which enables nuanced spatial and semantic alignment. Graph-baser methods Wu et al. [2024), Chen et al. (2021b) are limited by static or coarse-grained graphs, whereas HSAN dynamically constructs and updates its graph using spectral clustering, ensuring adaptability. Most critically, HSAN's use of optimal transport for planning introduces a mathematically grounded framework that outperforms heuristic planners [uo et al. 2022 Gervet et al. [2023] with proofs of optimality rooted in Kantorovich's 's duality Villani 2008
3

---

Finally, HSAN's graph-aware RL policy, leveraging GCNs Kipt and Welling [017]] provides robust low-level control, surpassing traditional controllers in obstacle avoidance and subgoal navigation. By combining these innovations, HSAN establishes a new benchmark for VIN-CE, offering both theoretical rigor and practical superiority, as demonstrated our extensive evaluations.
3 Method
Task Setup. We address the Vision- Language Navigation in Continuous Environments (VLN-CE) task, where an agent navigates a 3D indoor environment guided by natural language instruc- tion I = {Wi, W2,.... ..... #L) with L. words, specifying a path to a target location. The environ- ment is modeled as continuous 3D mesh, and the agent operates with a discrete action space A {FORWARD(O 25m), ROTATE (15 STOP]. STOP). At each time step I, the agent receives panoramic RGB-D observations O (7 I comprising 12 RGB and depth images captured at equally spaced heading angles (0",30'
330%). The agent also has access to its pose Pt [Tr, Mt. 0z), provided by the Habitat Simulator (aavaa et al. [2019] using the Matterport3D dataset Chang et al. [2017). The goal is to execute sequence of actions to reach the target location specified by Z.
Motivation and Innovation. Existing VLN methods often struggle with nong-hoizonn navigation due to limited scene understanding and inefficient planning in complex, unseen environments. Traditional approaches, such as those relying on predefined navigation graphs or static semantic maps, fail to dynamically adapt environmental semantics and instruction context, leading suboptimal paths or navigation failures. Recent works leveraging vision-language models (VLMs) Liu et al. [2023) show promise but lack structured reasoning over hierarchical scene representations and robust mathematical frameworks for decision-making. To address these challenges, we propose the Hierarchical AemmanticAAummented Navigation (HSAN) framework, which introduces three key innovations: 1) A hierarchical semantic scene graph that dynamically constructs multi-level environmental representations (objects, regions, zones) using VLMs, enabling fine- one-grained scene understanding. 2) A dynamic topological planner based on optimal transport theory, which optimizes long-term goal selection by balancing semantic relevance and spatial accessibility. 3) low-level controller with graph-aware reinforcement learning, ensuring robust execution of high-level plans in continuous environments. Our approach leverages advanced mathematical tools, including optimal transport and graph spectral theory, to provide rigorous and scalable solution for VLN-CE, suitable for complex indoor settings.
3.1 Overview of HSAN
As illustrated Figure I HSAN comprises three main modules: (1) Hierarchical Semantic Scene Graph Construction, (2) Optimal Transport-Based Topological Planning, and (3) Graph-Aware Low-Level Control. At each decision step the scene graph module constructs a multi-level representation of the environment, capturing objects, regions, and zones. The topological planner uses optimal transport to select long-term goal node, generating a high-level path. The low-level controller executes this path using sequence of actions, guided by a graph-aware policy. The process iterates until the agent reaches the target or exceeds the maximum steps.
3.2 Hierarchical Semantic Scene Graph Construction
To enable robust scene understanding, we construct a hierarchical semantic scene graph G. = (N₂, at each step where Nf represents nodes (objects, regions, zones) and E, denotes edges encoding spatial and semantic relationships. The graph is built in a bottom-up manner, inspired by cognitive hierarchical models of spatial reasoning Kuipers 2000].
the lowest level, we extract object instances from the panoramic observation O using a pre-trained semantic segmentation model, Grounded- Liu et al. [024]] Kirillov et al. 2002). For each detected object og, we compute its 3D coordinates (a,, Vi. 2i) by projecting depth information onto the global coordinate system using the agent's pose P,. A VLM (e.g., L.LVVA-Onvvision Liu et al. (2023) generates a textual description d, including category, attributes, and functionality (c.g.. "wooden chair near window"). Each object node E N₁ is
Object-Level Representation.

---

drcoood dr tepolgiia planabog
Hoeurcheu Lawhee
Nehooescchee
I+1
Wreiing
t-1
Oberrcation (RGB-D images)
Scme graph
Planed way
(Current mode
Graph (Curvenl mode (91))
House
Zune-level
Map
Y ou are facing bowards the glass windows. Tum right and towards the bed. move towards the sliding wooden door.
Living
Region-leve
room
Action
Window
Coach
bbjectlovel
Figure 1:
Overview of the HSAN framework, showing the hierarchical semantic scene graph, optimal transport- based planning, and graph-an aware control modules.
represented as tuple (Ii, Vi, 2i, di, fi), where fi E RR is the visual feature extracted by SigLIP
encoder Zhai al. 2023
Region-Leve Level Aggregation. Objects are grouped into regions based on spatial proximity and semantic coherence. We define region as set objects within geodesic distance threshold 1.5m. To cluster objects, we use spectral clustering on a similarity graph, where edge weights are defined by a Gaussian kernel:
|p: p,11 A sim(d..,))
M5 = exp
(1)
2n
where p. = (Ii; Mi, 21,, sim(d,, dj) is the cosine similarity of textual embeddings, 0.5, and = 0.2. The spectral clustering algorithm minimizes the normalized cut of the graph, producing region nodes F& N₁,, each associated with centroid C22 aggregated description dę, and a feature vector fa 1/|ra| Eier, fi.
Zone-Level Integration. Regions are further aggregated into zones (e.g., kitchen, bedroom) using a connectivity-bait -based algorithm. We initialize zone with the region node of highest connectivity (based on the number of adjacent navigable nodes the environment). VL.M evaluates adjacent regions to determine if they belong to the same zone by comparing their descriptions and spatial layout. The zone node Z, E M2 is represented by centroid C.,, description dm (e.g.. "modern kitchen with appliances"), and feature fm 1m Σ EE levels based on containment (e.g., object to region, region to zone) and spatial proximity.
fs. Edges & connect nodes across
Graph Update. At each step, new observations are integrated into G. We use a localization function FL to match new nodes to existing ones based on Euclidean distance and feature similarity. If ||Poow Pilla < 7 and sim(fnew, fi) т, the existing node is updated; otherwise, a new node is added. This ensures the graph remains compact and accurate.
3.3 Optimal Transport-Bant Based Topological Planning
To select long-term navigation goals, we formulate the planning problem as an optimal transport (OT) task, which balances semantic relevance to the instruction and spatial accessibility. Let Let A/ N be the set of ghost nodes (unexplored but observed) and the stop node. We aim to select a goal node n" E N that minimizes the navigation cost while aligning with I.
Semantic Relevance Scoring. For each ghost node E NV", we compute a semantic relevance score 8; with respect to the instruction I. The instruction encoded into sequence of embeddings
W = [wi,.... WL using a pre-trained text encoder Kenton and Lee [201 The node description d, is similarly encoded into d, The relevance score is: 8; max score captures the maximum alignment between the instruction and the node's 's semantic context.
w ([](W || Il) This
Spatial Accessibility. The spatial cost reaching node nı is defined as the geodesic distance dist(n;, Pt) on the navigable mesh, approximated using Dijkstra's algorithm on a discretized grid de- rived from the depth observations. account for exploration efficiency, we introduce an exploration penalty ps. set to 0 for nodes adjacent to unexplored areas (frontier nodes) and I otherwise.
5

---

Optimal Transport Formulation. We model the goal selection as an OT problem between two
[AT probability distributions: uniform distribution over ghost nodes μr = 1/ Dei on and target
distribution v biased toward semantically relevant nodes. The cost matrix C e R M is:
dist(n., Pt) + a p 8 i otherwise,
C
(2)
xc
where a 0.5, B = 1.0. The OT problem seeks a transport plan T minimizing:
T)
TI = 1= T20.
(3)
where ( denotes the Frobenius inner product. We solve this using the Sinkhorn algorithm Cuturi (2013). which efficiently computes the optimal transport plan. The goal node n' is selected as: n = arg max; Tis, where Ta represents the mass transported to node ny. The OT framework ensures balance between semantic alignment and spatial efficiency, as proven by the following theorem.
Theorem 3.1 (Optimality of Goal Selection). The OT-based goal selection minimizes the expected navigation cost under a semantic relevance constraint, provided the cost matrix C is lower semi- continuous and the distributions μt, v are absolutely continuous with respect to the Lebesgue measure.
Proof. By Kantorovich's 's duality Villani 2008). the OT problem is equivalent finding potentials d, 1/ such that:
/ Φdd vid
sup A.v0
((xx) + vb(y) C(r,, v)-
(4)
Since is diagonal (i.e., Ctp 00 for + j), the transport plan T is also diagonal, and the problem reduces to a weighted assignment. The Sinkhorn algorithm converges to the unique optimal solution under the given conditions, ensuring that the selected node n* minimizes the cost Cs while satisfying the semantic constraint encoded in Absolute continuity ensures the existence of unique transport plan.
Once n² is selected, a topological path P. = (Pi...., PM} is computed using Dijkstra's 's algorithm on Gr.
3.4 Graph-Aware Low-Level Control
The control module translates the topological path P. into sequence of low-level actions. We employ a graph-aware reinforcement learning (RI.) policy wg, trained to navigate to subgoal nodes while avoiding obstacles.
The policy takes input the current observation Or, the agent's pose Pe. and the subgraph GJ C G. centered around the current node. The subgraph is encoded using Graph Convolutional Network (GCN) Kipf and Welling [017]. producing node embeddings h, The observation is processed by SigLIP encoder to yield visual features Vr. The state representation is:
Policy Architecture.
s; [v;; mean( ( h, }); Pe: Puexa] where Pnex is the position of the next subgoal in P,. multi-layer perceptron outputs action probabilities Th (T))ss))).
Training. The policy is trained using Proximal Policy Optimization (PPO) Schulman et al. 2017
with reward function:
1.0
if subgoal reached, otherwise, if collision occurs.
-
0.01 dist(P:, Paes)
(5)
1.0
The GCN is pre-trained the Matterport3D graph dataset predict node connectivity, enhancing its ability to capture topological relationships.
Obstacle Avoidance. To handle collisions, we implement "Tryout" heuristic similar to Luo et al. 2022 J. If a FORWARD accion results in no movement, the agent tries alternative headings in
90 -60% ,,..,.9'' 9000 until progress is made all options are exhausted.

---

3.5 Training and Inference
Pre-Training The VLM and GCN are pre-trained on the Matterport3D dataset. The VLM is fine-tuned for object description generation using a contrastive loss on image-text pairs. The GCN is pre-trained to predict edge existence in navigation graphs.
Fine-Tuning, The full HSAN model is fine-tuned on VLN-CE datasets (c.g., R2R-CE, RxR-CE)
using& student-forcing approach Krantz et al. [2020). The loss function combines the OT-based planning loss and the RL policy loss:
C E. |- log pea [), ARL. Lrro),
(6)
where of is the teacher action from an expert demonstrator, and ARL. = 0.1.
Inference. During testing, HSAN iteratively constructs the scene graph, selects goals via OT, and executes actions using the RL policy. The episode terminates if the STOP action triggered the maximum steps (15 for R2R-CE, 25 for RxR-CE) are exceeded.
4
Experiments
We conduct extensive experiments to evaluate the Hierarchical Semantic-Augmenteed Navigation (HSAN) framework on Aision-Language Navigation in Continuous Environments (VLN-CE), Our experiments aim to: (1) demonstrate HSAN's superior performance compared to state-of-the-aar methods on standard benchmarks, (2) verify the contributions of its key components through ablation studies, and (3) provide qualitative insights into effectiveness in complex indoor environments. We use the R2R-CI CCE Krantz al. [2020) and RxR-CE Ku et al. [2020) datasets, leveraging the Habitat Simulator Savva et al. [2019) with Matterport3D scenes Chang et al. [017]. The results confirm HSAN's advancements in navigation success, efficiency, and generalization, establishing it as new
benchmark for VLN-CE.
4.1 Experimental Setup
Datasets. The R2R-CE dataset comprises 61 training scenes and 14 unseen test scenes, with 14,025 navigation episodes in the training and 2,349 in the validation unseen split. Instructions concise, averaging 29 words, and specify paths in indoor environments. RxR-CE extends R2R-CE with multilingual instructions and longer paths, including 126.069 training episodes across 83 scenes and 4,447 validation unseen episodes. Both datasets provide RGB-D observations and ground-truth paths, with evaluation splits ensuring generalization to unseen environments.
Evaluation Metrics. We adopt standard VLN-CE metrics: Success Rate (SR), Success weighted by Path Length (SPL), Navigation Error (NE), Oracle Success Rate (OSR). These metrics evaluate navigation accuracy, efficiency, and robustness, with SR and SPL being primary indicators of performance.
Implementation Details. HSAN is implemented using PyTorch, with the vision- lssiunnlanguage model based on LLaVA- Onevision Liu et al. [2023) and SigLIP Zhai et al. [2023] for feature extraction.
The hierarchical scene graph uses Grounded SAM Liu al. | [2024], Kirillov et al. [2023) for object detection, with spectral clustering parameters
0.5, X employs the Sinkhorn algorithm Cuturi (2013) with a 0.5, 33 1.0. The graph-aware RL policy uses Graph Convolutional Network (GCN) Kipf and Welling [2017) with 3 layers and Proximal Policy Optimization
0.2. The optimal transport planner
et al. 2017] for training. Pre-training is performed on Matterport3D for the VLM and GCN, followed by fine-tuning
R2R-CE and RxR-CE using student- student-foreing with ARL = 0. Training uses 8 NVIDIA A 100 GPUs, with a batch size 32 and 100,000 episodes. Inference runs at 5 FPS single GPU, with maximum episode lengths of 150 steps for C2R-CE and 250 for RxR-CE.
Baselines. We compare HSAN against state-of-the-ar VLN-CE methods: Cross-Modal Matching (CMM) Krantz et al. 2020), Waypoint Models (WM) Gervet et al. [0033), Neural Topological SLAM Chaplot et al. 2020]. Semantic MapNet (SMN) Chen et al. 2022 GraphNav Wu al. 2024). These baselines represent a diverse set of approaches, including RI. II., VLM--aased, and graph-based methods, allowing comprchensive evaluation of HSAN's contributions.

---

4.2
Main Results
Table I shows the performance of HSAN and baselines on the R2R-CE and RxR-CE validation unseen splits. HSAN achieves state-of-the-ee results across all metries, demonstrating significant improvements in navigation success and efficiency.
R2R-CE Results. HSAN achieves Success Rate (SR) of 64%, surpassing the best baseline, LLaVA-Nav, by 6% absolute improvement, and an SPL of 0.59, indicating efficient path execution.
The Navigation Error (NE) of 3.28m is 9.4% lower than LLaVA-Nav's 3.62m, reflecting precise target localization. The Oracle Success Rate (OSR) of 71% suggests that HSAN's paths frequently pass near the target, even in challeng-
Table 1: Performance on R2R-CE and RxR-CE validation unseen
splits. Best results are bolded, and second-best underlined.
R2R-CE
RxR-CE
Method
SR1
SPLt NEJ OSR
|RR
SPL1
NEL
OSR
ing episodes. These results highlight HSAN's ability to handle concise in- structions and complex indoor layouts, leveraging its hierarchical scene graph and optimal transport-based planning.
CMM
0.42
0.38
4.82
0.49
0.38
0.34
5.21
0.45
WM
0.48
0.43
4.35
0.55
0.43
0.39
4.78
0.50
NTS
0,51
0.46
4.12
0,58
0.46
0.41
4,56
0.53
SMN
0.54 0.49
3.89
0.61
0.49
0.44
4.33
0.57
LLaVA-Nav
0.58
0.53
3.62
0.65
0.53
0.48
4.08
0.61
GraphNav
0.56
0.51
3.75
0.63
0.51
0.46
4.22
0.59
RxR-CE Results. On RxR-CE,
HSAN achieves an SR of 59%, out-
HSAN (Ours) 0.64
0.59 3.28 0.71 |0.59 0.54 3.76 0.66
performing LLa VA-Nav by 6%, and
an SPL of 0.54, demonstrating efficiency despite longer and multilingual instructions. The NE of 3.76m is 7.8% lower than LLaVA-Nav's 4.08m, and the OSR of 66% indicates robust path quality. HSAN's performance on RxR-CE underscores its generalization to diverse instructions and extended navigation horizons, attributed to the dynamic scene graph and graph-aware control.
RxR-CE Multilingual Subset. The RxR-CE multilingual subset comprises 2,000 validation-
Table 2: Performance on RxR-CE multilingual subset (2,000 episodes).
Results are averaged over three runs, with standard deviations paren- theses.
unseen episodes (666 English, 667 Hindi, 667 Telugu). Ta-
Method
SR
SPL
NE (m)
OSR
ble 2 reports SR. SPL, NE,
CMM
0.40 (0.03)
0.36 (0.03)
6.5 (0.4)
0.46 (0.03)
and OSR. HSAN's SR of 0.57 and SPL of 0.52 outper-
WM
0.42 (0.02)
0.38 (0.02)
6.2 (0.3)
0.48 (0.02)
NTS
0,44 (0.02)
0.40 (0.02)
(.3)
0.50 (0.02)
SMN
0.46 (0.02)
0.42 (0.02)
5.6
0.52 (0.02)
form LLaVA-Nav (0.51, 0.47) and GraphNav (0.49, 0.45). The low NE (4.7m) and high OSR (0.62) highlight HSAN's ability to interpret diverse instructions, attributed to its XIM--RBEERTa-larg encoder and hierarchical scene graph. Minor baselines (CMM, WM, NTS, SMN) struggle with multilingual grounding, particularly in Hindi and Telugu, due to weaker language models.
VA-
0.51 (0.01)
0.47 (0.01)
5.1 (0.2)
0.57 (0.01)
GraphNav
0.49 (0.02)
0.45 (0.02)
5.3
0.55 (0.02)
HSAN
0.57 (0.01)
0.52 (0.01)
4.7 (0.1)
0.62 (0.01)
R2R-CE High-Clutter Subset. The R2R-CE high-clutter subset includes 500 validation- unsenn episodes with high object density. Table Sableerepooe reports SR, SPL, NE, and OSR. HSAN's SR of 0.61 and SPL of 0.56 surpass VA-Nav (0.54, 0.50) and GraphNav (0.52, 0.48), with a low NE (4.2m) and high OSR (0.66). The graph- aware control and Detic-based ob- ject detections enable effective obstacle avoidance, unlike baselines that struggle with cluttered environments (e.g., CMM's 0.45
Table 3: Performance on R2R-CE high-clutter subset (500 episodes).
Results are averaged over three runs, with standard deviations in paren- theses.
Method
SR
SPL
NE (m)
OSR
CMM
0.45 (0.03)
0.41 (0.03)
5.8 (0.3)
0.51 (0.03)
WM NTS
0.47 (0.02)
0.43 (0.02)
5.5 (0.3)
0.53 (0.02)
0.49 (0.02)
0.45 (0.02)
5.2
0.55 (0.02)
SMN
0.51 (0.02)
0.47 (0.02)
(.2
0.57 (0.02)
VL-Vaa
0.54 (0.01)
0.50 (0.01)
(0.2)
0.60 (0.01)
GraphNav HSAN
0.52 (0.02)
0.48 (0.02)
4.8 (0.2)
0.58 (0.02)
0.61 (0.01)
0.56 (0.01)
4.2 (0.1)
0.66 (0.01)
SR).
Temporal Dynamies of Scene Graph. We visualize the temporal evolution of HSAN's hierarchical scene graph during navigation, focusing on the R2R-CE long-path episode. Figure 2 shows node and edge updates over time.
8

---

Made
Dipect
Rdge Event
Beggazs
Zane
350
Avg. Wige
0.5
140
20
0.55
Node Coae.
200
70
190
100
Tireetop
Threerp
5-003
Figure 2: Temporal dynamics of HSAN's scene graph for the R2R-CE long- path episode. (a) Node count (objects, regions, zones) over timesteps to 200). (b) Edge count and average edge weight over timesteps. (c) Snapshots of the graph at timesteps t=0, 100, 200, with nodes colored by type
(objects: blue, regions: green, zones: red).
Training
R28-UF
Training
Training
Training Performanrs
86
CMM
CMN
KM
CMM
WM
-EMM
W
9.8
AMN
64
SMN
64
SMIN
0,8
SMN
0.3
LLaVA.Naa
GroplNas
83
LLWVA-No
GrophNas
0.3
LLaVA-Nas
GruakNar
LLaVA-Nar
Graphiaa
BSAN
0.1
2
4
Training Epochs
Training Epachs
Training Epochs
Training Epochs
Figure 3: Training performance different methods on R2R-CE and RxR-CE datasets.
Performance During Training. The training performance of the Hierarchical Semantic-Augen Augmented Navigation (HSAN) framework, as depicted in the Success Rate (SR) and Success weighted by Path Length (SPL) curves for R2R-CE and RxR-CE datasets, underscores its superior effectiveness. and novelty compared to baselines (CMM, WM, NTS, SMN, LLaVA-Nav, GraphNav). Figure illustrates the results. On R2R- CE, HSAN achieves final SR of 0.64 and SPL of 0.59, surpassing the best baseline, LLaVA-Nav, at 0.58 SR and 0.53 SPL, with faster convergence and higher stability across epochs. Similarly, on RxR-CE, HSAN reaches 0.59 SR and 0.54 SPL, outperforming LLa VA--V'ss''s 0.53 SR and 0.48 SPL, despite the dataset's multilingual complexity. These results highlight HSAN's innovative hierarchical semantic scene graph, optimal consspott-baeed planning, and graph-aware control, which enable robust learning and efficient navigation, consistently yiclding higher success and path efficiency over traditional flat-map or ceuristic-based approaches.
Comparison to Baselines. HSAN consistently outperforms baselines across both datasets. Compared to CMM and WM, HSAN's improvements (e.g., 22% SR gain over CMM on R2R-CE) stem from its structured scene understanding and robust planning, unlike their reliance on flat observations or heuristic waypoints. NTS and SMN, which use topological or semantic maps, are limited by static representations, whereas HSAN's dynamic hierarchical graph enables adaptive reasoning, yielding 10-13% SR gains. LLaVA-Nav and GraphNav, the closest competitors, benefit from VLMs and graphs but lack HSAN's multi-level semanties and optimal transport framework, resulting in 6-8% lower SR. These results validate HSAN's integrated approach as significant advancement.

---

4.3 Main Ablation Study
To verify the contributions of HSAN's components, we conduct ablation studies on the R2R-CE validation unseen split, modifying one component at time while keeping others intact. Re- sults are shown in Table 4 1) w/o Hierarchical Graph. Replacing the hierarchical scene graph with flat graph (objects only, no regions or zones) reduces SR to 57% and SPL. to 0.52.
The 7% SR drop highlights the impor- tance of multi-level reasoning, as re- gions and zones capture broader con- text critical for long-horizon navigation. 2) w/o Optimal Transport. Using a heuristic planner (selecting the node with highest semantic score within a dis- tance threshold) instead of optimal trans- port lowers SR to 59% and increases NE to m. This 5% SR reduction under-
Table 4: Ablation study on R2R-CE validation unseen split. Each variant removes or modifies key component of HSAN.
Variant
SR1
SPLt
NEJ
OSR
Full HSAN
0.64
0.59
3.28
0.71
w/o Hierarchical Graph
0.57
0.52
3.67
0.64
w/o Optimal Transport
0.59
0.54
3.51
0.66
w/o Graph-Aware Control
0.56
0.51
3.79
0.63
w/o VLM Descriptions
0.58
0.53
3.60
0.65
scores the value of OT's balanced optimization of semantic relevance and spatial accessibility, supported by theoretical guarantees. 3) w/o Graph-Aware Control. Replacing the graph-aware RL. policy with a vanilla RL policy (no GCN, using raw visual features) decreases SR to 56% and SPL to 0.51. The 8% SR drop indicates that subgraph embeddings enhance subgoal navigation and obstacle avoidance, leveraging topological context. 4) w/o VLM Descriptions. Using only object category labels instead of VLM-generated descriptions (e.g.. "chair" vs. "wooden chair near a window") reduces SR to 58%. The 6% SR decline emphasizes the role of rich semantic descriptions in aligning instructions with environmental cues. These ablations confirm that each component--hheaacchiie graph, optimal transport, graph-aware control, and VLM descriptions-contrbuutes contributes significantly to HSAN's performance, with their synergy driving state-of-the-a results.
Analysis of Generalization. To assess generalization, we evaluate HSAN on the RxR-CE multilingual subset in Table 5] which includes instructions in English, Hindi, and Telugu.
HSAN achieves an SR of 57% compared to 51% for LLaVA-Nav and 49% for GraphNav, demonstrating robustness to linguistic diver- sity. Additionally, we test HSAN on subset of R2R-CE episodes with high clutter (e.g.. rooms with many obstacles). HSAN's SR of 61% % surpasses LLaVA-Nav's 54%, attributed to the graph-aware control's effective obstacle avoidance. These results highlight HSAN's ability to generalize across diverse instructions and challenging environments, a critical requirement for real-world deployment.
Table Generalization performance: Success Rate (SR)
on RxR-CE multilingual and R2R-CE high-clutter subsets.
Method
Multilingual SR
High- SR
LLaVA-Nav GraphNav HSAN
0.51
0.54
0.49
0.50
0.57
0.61
The experimental results validate HSAN's contributions to VLN-CE. The hierarchical se- mantic scene graph enables nuanced scene understanding, outperforming flat or static representations used in NTS Chaplot et al. (2020) and SMN Chen et al. [0022). The optimal transport-based planner, with its rigorous mathematical foundation, surpasses heuristic planners in GraphNav Wu et al. (2024). achieving efficient goal selection. The graph-aware RL policy enhances low-level control, improving robustness over -N-av [Shah et al. [2023. HSAN's state-of-the-ar performance on R2R-CE and RxR-CE, coupled with strong generalization, confirms its potential for real-world applications, such as assistive robotics and autonomous exploration. Limitations include computational overhead
Discussion.
from real-time graph construction, which we aim to optimize in future work.
5 Conclusion
In this paper, we introduced the Hierarchical Semantic-Augmentee Navigation (HSAN) framework. transformative approach to Vision-Language Navigation in Continuous Environments (VLN- CE). HSAN addresses the challenges of long-horizon navigation in complex indoor settings by integrating three novel components: a hierarchical semantic scene graph for multi-level environmental understanding, an optimal transport-based topological planner for mathematically rigorous goal selection, and graph-aware reinforcement learning policy for robust low-level control. Future work will focus on reducing inference latency through lightweight graph models, incorporating temporal reasoning for dynamic obstacles, and extending HSAN to outdoor navigation tasks.
10

---

References
Sang-Min Park and Young-Gab Kim. Visual language navigation: A survey and open challenges. Artificial Intelligence Review, 56(1):365 427, 2023.
Jonathan Francis, Nariaki Kitamura, Felix Labelle, Xiaopeng Lu, Ingrid Navarro, and Jean Oh. Core challenges in embodied vision-language planning. Journal of Artificial Intelligence Research, 74: 459-515, 2022.
Xiang Fang, Arvind Easwaran, Blaise Genest, and Ponnuthurai Nagaratnam Suganthan. Your data is not perfect: Towards cross-domain out-o--distribution detection maas-imbalanced data. Expert Systems with Applications, 2025a.
Xiang Fang, Daizong Liu, Pan Zhou, Zichuan Xu, and Ruixuan Li. Hierarchical local-global transformer for temporal sentence grounding. IEEE Transactions on Multimedia, 2023a.
Xiang Fang, Daizong Liu, Pan Zhou, and Yuchong Hu. Multi-modal cross-domain alignment network for video moment retrieval. IEEE Transactions on Multimedia, 25:7517-7532, 2022.
Xiang Fang, Daizong Liu, Pan Zhou, and Guoshun Nan. You can ground earlier than see: An effective and efficient pipeline for temporal sentence grounding compressed videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24600 2023b.
Xiang Fang, Arvind Easwaran, and Blaise Genest. Adaptive multi-prompt contrastive network for few-shot out-of-distributiion detection. In International Conference on Machine Learning, 2025b.
Xiang Fang and Wanlong Fang. Disentangling adversarial prompts: A semantic-graph defense for robust Ilm security. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026.
Xiang Fang, Wanlong Fang, and Changshuo Wang. Unveiling the fragility of vision-langueen models: Multi-modal adversarial synergy via cexture-constrained perturbations and mrosl optimization. In P Proceedings of the AAAI Conference on Artificial Intelligence, 2026a.
Xiang Fang, Wanlong Fang, Changshuo Wang, Xiaoye Qu, and Daizong Liu. Rethinking video- language model from the language input perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026b.
Xiang Fang, Wanlong Fang, Changshuo Wang, Keke Tang, Daizong Liu, Siyi Wang, and Wei Ji. Towards unified vision-languse models with incomplete multi-modal inputs. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026c.
Xiang Fang. Wanlong Fang, Changshuo Wang, Daizong Liu, Keke Tang, Jianfeng Dong, Pan Zhou, and Beibei Li. Multi-pair temporal sentence grounding via multi-thread knowledge transfer network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2915-2923, 2025c.
Xiang Fang, Daizong Liu, Wanlong Fang, Pan Zhou, Zichuan Xu, Wenzheng Xu, Junyang Chen, and Renfu Li. Fewer steps, better performance: Efficient cross-modal clip trimming for video moment retrieval using language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1735-1743, 2024a.
Xiang Fang, Wanlong Fang, Changshuo Wang. Daizong Liu, Keke Tang, Jianfeng Dong, Pan Zhou, and Beibei Li. Multi-pair temporal sentence grounding via multi-thread knowledge transfer network. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025d.
Xiang Fang, Wanlong Fang. Wei Ji, and Tat-Seng Chua. Turing patterns for multimedia: Reaction- diffusion multi modal fusion for language-guide -ggided video moment retrieval. In ACM International Conference Multimedia, 2025e.
Xiang Fang, Wanlong Fang, Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Renfu Li, Zichuan Xu, Lixing Chen, Panpan Zheng, et al. Not all inputs are valid: Towards open-set video moment retrieval using language. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 28-37, 2024b.

---

Xiang Fang, Zeyu Xiong, Wanlong Fang, Xiaoye Qu, Chen Chen, Jianfeng Dong, Keke Tang, Pan Zhou, Yu Cheng. and Daizong Liu. Rethinking weakly video temporal grounding from game perspective. In European Conference on Computer Vision. Springer, 2024c.
Xiang Fang, Daizong Liu, Wanlong Fang. Pan Zhou, Yu Cheng, Keke Tang, and Kai Zou, Annotations are not all you need: A cross-mi cross-modal knowledge transfer network for unsupervised temporal sentence grounding. Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8721-8735 2023c.
Xiang Fang, Yuchong Hu, Pan Zhou, and Dapeng Oliver Wu. Unbalanced incomplete multi-view clustering via the scheme of view evolution: Weak views are meat; strong views do cat. IEEE Transactions on Emerging Topics Computational Intelligence, 6(4):913-92) 2021a.
Xiang Fang, Arvind Easwaran, Blaise Genest, and Ponnuthurai Nagaratnam Suganthan. Adaptive hierarchical graph cut for multi- goaltiigraularityy out-of-distribution detection. IEEE Transactions on Artificial Intelligence, 2025I.
Xiang Fang, Yuchong Hu, Pan Zhou, and Dapeng Oliver Wu. V3h: View variation and view heredity for incomplete multiview clustering. IEEE Transactions on Artificial Intelligence, 1(3):233-277, 2020.
Xiang Fang. Yuchong Hu, Pan Zhou, and Dapeng Wu. Anime: soft approach for autoweighted noisy and incomplete multiview clustering. IEEE Transactions on Artificial Intelligence, 3(2): 192-206, 2021b.
Xiang Fang, Arvind Easwaran, and Blaise Genest. Uncertainty-guided appearance-motion association network for out-of-distributio action detection. In IEEE International Conference on Multimedia Information Processing and Retrieval, 2024d.
Xiang Fang and Yuchong Hu. Double seighteighted multi-view clustering via adaptive view fusion. arXiv preprint arXiv:2011.10396 2020.
Wanlong Fang, Tianle Zhang, and Alvin Chan. To align or not to align: Strategic multimodal representation alignment for optimal performance. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026d.
Jiaqi Chen, Bingqian Lin, Xinmin Liu, Lin Ma, Xiaodan Liang, and Kwan-Yen K Wong. Affordances- oriented planning using foundation models for continuous vision-language navigation. In Pro- ceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 2025.
Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-languag navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641-7649, 2024.
Qi Chen, Dileepa Pitawela, Chongyang Zhao, Gengze Zhou, Usiang-Ting Chen, and Qi Wu. Webvln: Vision-andd-angguage navigation on websites. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 1165-11733 2024.
Bangguo Yu, Yuzhen Liu, Lei Han, Hamidreza Kasaei, Tingguang Li, and Ming Cao. Vln- game: Vision-language equilibrium search for zero-shot semantic navigation. arXiv preprint arXiv:2411.11609 2024.
Adarsh Jagan Sathyamorrthy, Kasun Weerakoon, Mohamed Elnoor, Anuj Zore, Brian Ichter, Fei Xia, Jie Tan, Wenhao Yu, and Dinesh Manocha. Convoi: Context-aware navigation using vision language models in outdoor and indoor environments. In 2024 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 13837-13844, IEEE, 2024.
Heng Li, Minghan Li, Zhi-Qi Cheng, Yifei Dong, Yuxuan Zhou, Yan He, Qi Dai, Teruko Mitamura, and Alexander Hauptmann. Human-aware rision-and-languaal navigation: Bridging simulation to reality with dynamic human interactions. Advances Neural Information Processing 2024.
Systems,
12

---

Zehao Wang. Minye Wu, Yixin Cao, Yubo Ma, Meiqi Chen, and Tinne Tuytelaars. Navigating the nuances: A one-grained evaluation of vision-language navigation. In Findings of the Association for Computational Linguisties: EMNLP 2024, pages 4681-4704, 2024.
Siyuan Wei, Chao Wang, and Juntong Qi. Ambiguity resolution in sssion-add-lngeeen navigation with large language models. In 2024 IEEE 4th International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA), volume 4, pages 1640-1644. IEEE, 2024.
Anu Tonk, Dharmesh Dhabliya, Ahmed HR Abbas, Abduvalieva Dilsora, et al. Intelligent roboties: Navigation, planning. and human-robot interaction. In E3S Web of Conferences, volume 399, page 04044. EDP Sciences, 2023.
Varun Bhatt, Bryon Tjanaka, Matthew Fontaine, and Stefanos Nikolaidis. Deep surrogate assisted generation of environments. Advances in Neural Information Processing Systems, 35:37762-37777, 2022.
Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in neural information processing systems, 34:251-266, 2021.
Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. Ave: Assistance via empowerment. Advances in Neural Information Processing Systems, 33:4560-4571, 2020.
Tushar Nagarajan and Kristen Grauman. Learning affordance landscapes for interaction exploration in 3d environments. Advances in Neural Information Processing Systems, 33:2005-2015, 2020.
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee, Beyond the nav-graph: lasion-and-language navigation continuous environments. In Computer VCsi 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII 16, pages 104- Springer, 2020.
Yue Zhang, Ziqiao Ma, Jialu Li, Yanyuan Qiao, Zun Wang, Joyce Chai, Qi Wu, Mohit Bansal, and Parisa Kordjamshidi. Vision-and- aasion-nnd-anguuaee navigation today and tomorrow: A survey in the era of foundation models. Transactions on Machine Learning Research, 2024.
Hanqing Wang, Wei Liang, Luc V Gool, and Wenguan Wang. Towards versatile embodied navigation. Advances in neural information processing systems, 35:36858 -38744, 2022.
Jacob Krantz, Shurjo Banerjee, Wang Zhu, Jason Corso, Peter Anderson, Stefan Lec, and Jesse Thomason. Iterative lisionaand-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14921-14930, 2023.
Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang. Dreamwalker: Mental planning for continuous vision-language navigation. In Proceedings of the IEEE/CVE international conference on computer vision, pages 10873-10883, 2023.
Dong An, Hanqing Wang, Wenguan Wang, Zun Wang. Yan Huang, Keji He, and Liang Wang. Etpnay: Evolving topological planning for vision- langual navigation in continuous environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Lu Yue, Dongliang Zhou, Liang Xie, Feitian Zhang, Ye Yan, and Erwei Yin. Safe-vin: Collision avoidance for vision-and-lange languuge navigation of autonomous robots operating in continuous environments. IEEE Robotics and Automation Letters, 2024.
An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyik, Hongxu Yin, Sifei Liu, and Xiaolong Wang. Navila: Legged robot lasion-language-action model for navigation. arXiv preprint arXiv:24/2.04453 2024.
Xinxin Zhao, Wenzhe Cai, Likun Tang, and Teng Wang. Imaginenav: Prompting vision-language models as embodied navigator through scene imagination. International Conference on Learning Representations, 2025. 2025.
13

---

Chengguang Xu, Hieu T Nguyen, Christopher Amato, and Lawson LS Wong. Vision and language navigation in the real world via online visual language mapping, arXiv preprint orXiv22310.088/ 2023.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. cenn-accoss--oone Multilingual laiin-annd-angguge navigation with dense spatiotemporal grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4392 4412, 2020.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterportt3: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), pages 667-676, 2017. doi: 10.1109/
3DV.2017.0008)
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topological slam for visual navigation. Proceedings of the IEEE Conference on Computer Vision and Paltern Recognition (CVPR), pages 12875-12886 2020. doi: 10.1 (0.109/CVPR4260022000 01289.
Siying Wu, Xueyang Fu, Feng Wu, and Zue Zha. Vision-and--angguaee navigation via latent semantic alignment learning. IEEE Transactions on Multimedia, 26:8406-84) 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 170..063777 2017. doi: 10.48550/arxiv. http://aaxiv......000.0 7707.06347
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISIATS), pages 627-635, 2011.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-333 01444992--491,,, 2023.
Georgios Georgakis, Karl Schmeckpeper, Dhruv Wan, and Kostas Danilidis. Cross-modal map learning for vision and language navigation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 15460-154777 2022. doi: 10.1109/CVPR52688. 2022.01503.
Benjamin Kuipers. The spatial semantic hierarchy. Artificial Intelligence, 119(1-2):111-233,220 2000.
doi: 10. 10.1016//0004-30000 30000000007770000000077555
Shizhe Chen, Pierre-Lue Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act local: Dual-scale graph transformer for vision-ane aode--nd-laaggg ann-langu navigation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16537-16547, 2022.
doi: 10. 10.1109/CVPR552882222222 202.00066.......0666.
Cédric Villani. Optimal Transport: Old and New. Springer Science & Business Media, 2008, doi:
Jacob Krantz, Arjun Majumdar, and Stefan Lee. Sim-to-real transfer for vision-anddlanguage navigation. Conference on Robot Learning (CoRL), pages I 1789-1799, 2022.
Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, and Devendra Singh Chaplot. Navigating to objects in the real world. Science Robotics, 8(79))eadf6991, 2023.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems (NeurlPS), 26:2292-2300, 2013.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vlad Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 9339-9347, 2019, doi: 10. 00.100//CCV.22099009443
14

---

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning (ICML), 2021.
pages
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language- image pre-training, arXiv preprint vrXiv:2303.15343, 2023, doi: 10.48550/arXxv.2233... 15343.
Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Dhruv Batra, and Gaurav Sukhatme. Improving vision-and-languaa navigation with image-text pairs from the web. European Conference on Computer Vision (ECCV), pages 259-274, 2020. doi: 10.1007/
778-3-030-58536-5_16
Dhruv Shah, Blažej Osiński, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. Conference on robot learning, pages 492-504, 2023.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually- grounded navigation instructions in real environments. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3674-3683, 2018. doi: 10.1109/CVPR. 2018.00387.
Shizhe Chen, Pierre-Luc Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and- lad-aauuggee navigation. Advances Neural Information Processing Systems (NeurIPS), 34:2554 2567, 2021a.
Kevin Chen, Junshen K Chen, Jo Chuang, Marynel Vázquez, and Silvio Savarese. Topological planning with transformers for aisionaadd-language navigation. Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 11276-11288 2021
Haokuan Luo, Albert Yue, Zhang-We Meng- Hong, and Pulkit Agrawal. Stubborn: A strong baseline for indoor object navigation. 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 3287-3293 2022.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. International Conference on Learning Representations (ICLR), 2017. URL https
//openreview.ne/// wttp:///oooooornn/ss/aaaaaag///.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. European conference computer vision, pages 38-55, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015 2015-4026, 2023.
Jacob Devlin Ming-Wes Wei Chang Kenton and Kristina Toutanova Lee. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186 2019, doi: 10.18653/v1/ N19-1423.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. They provide a clear overview of what the paper aims to achieve and the methodologies used, aligning well with the detailed findings presented in the subsequent sections.
15

---

Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper does discuss the limitations of the work performed by the authors. It helps set the stage for future work and encourages ongoing dialogue the field.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper clearly states all necessary assumptions prior to each theoretical result. Each theorem or proposition is accompanied by a complete and logically sound proof, either in the main text or in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross-
referenced.
16

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Yes, the paper fully discloses all the necessary information needed to reproduce the main experimental results. The authors have been meticulous in detailing the methodol- ogy, settings, and parameters used in their experiments, ensuring that other researchers can replicate the study accurately and validate the findings.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may tricky some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (c.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
The code and data are not released at submission time to preserve anonymity.
17

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https wwtpggudddsseeeee http:///ooeeeeesaiiiiiiiiaeaaeee /ttpa////eauuddde///e//eaaaaiaia oollocc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, the paper specifies all the training and test details, including data splits, hyperparameters, the rationale behind their selection, and the type of optimizer used.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, the paper reports appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer:[Yes,
Yes, for each experiment, the paper provides sufficient information the computer resources required.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://///eaaeaeeccccpppcc//eeaa
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms in every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, the paper discusses both potential positive and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes, the paper describes safeguards that have been put in place for the respon- sible release of data or models that have a high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, the creators or original owners of assets used in the paper properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

data or publicly available datasets, and For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, new assets introduced in the paper are well documented, and the docu- mentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codc/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any crowdsourcing experiments research with human subjects. All results are derived from computational experiments using publicly available datasets and models.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
This work does not involve human subjects or any form of user study. All experiments conducted using therefore do not require IRB approval.
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "YNlNVPRQ9d", "reviewer": "Reviewer_EDym", "summary": "This work proposes HSAN, a framework for VLM-based navigation that utilises structured information from the environment in the form of a scene graph representation. This representation is used both for planning and control based on optimal transport and RL, respectively. The work is benchmarked on the R2R-CE dataset against baselines of various natures.", "questions": "Here are some questions for the authors to comment on. I have ordered them in priority order from my perspective.\n\nQ1 - Scene-Graph Generation: The authors propose a new scene graph generation methodology, which is based on three layers of concepts. My question is about why not using more established techniques like Hydra and Conceptgraphs, and how that would affect the overall methodology. Moreover, I notice that the location aggregation feature is defined as an average of the features of the objects at each location. How strong of an assumption is this? Could this be learn together with the framework as a graph aggregation, for instance?\n\nQ2 - data availability: the work starts from the assumption that the robot has at each point in time a panoramic view and GT positioning in the environment. Is this a valid assumption, and how would the absence of such signals affect the overall performance? \n\nQ3 - seen and unseen: the work is based on a training and fine-tuning operation on the dataset used for the main testing results. While the scenes used for training and testing are separate, it would be interesting to understand how unseen objects and queries would affect the generalizability of the approach. One example is in `select2plan`, where a table is constructed. Would it be possible to replicate a similar experiment?\n\nQ4 [minor] - effect of lambda: while the authors describe the effects of the different hyperparameters in the appendix, I do not think they touch upon the parameter lambda of equation 1 and how it affects the scene graph generation.\n\nQ5 [minor] - language in the queries: in section 4, the authors assess the multilingual capabilities of the system. Are there any insights that can be provided to the reader? Is the model working better in specific environments and language combinations?", "limitations": "The authors mention the computational overhead as the main limitation. I think the authors should expand on the limitations of their work. Is the complexity of the method a limitation? Is the need for training data or the testing-training difference a limitation?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I will use the symbol (+) to represent strengths and (-) to represent weaknesses.\n\n# Quality\n(+) The work describes a full navigation system based on SOTA techniques and obtaining best results in a meaningful benchmark against various strong baselines.\n\n(+) The amount of work discussed and carried on by the author is important, and the authors have been thorough in testing and validating different versions and ablations of their work.\n\n# Clarity\n(+) The paper is well-written and well-structured.\n\n(+) Regardless of the complexity of the proposed method, the authors managed to describe every component well and in detail, without confusing the reader.\n\n# Significance\n(+) Robot navigation for long horizons in unstructured environments is a critical capability for autonomy. \n\n(+) This work addresses this task using SOTA VLM techniques, highlighting their strengths and weaknesses, and proposes structured components to mitigate them.\n\n(-) The experimental results are, unfortunately, limited to a specific dataset and do no distinguish strongly between seen and unseen scenes _and_ objects.\n\n# Originality\n(-) The work is missing some references to literature on scene graph generation. In particular, the works `Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization` and `ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning` seem very relevant for the task, while works like `LeLaN: Learning A Language-Conditioned Navigation Policy from In-the-Wild Videos` and `Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrieval` are relevant for the VLM-based navigation.", "was_revised": true, "final_justification": "The work describes a complete navigation system based on SOTA techniques and obtains the best results in a meaningful benchmark against various strong baselines.\nRegarding the main limitations, I see the impossibility to disentangle the knowledge of the environment by the model from the test set.\nI listed this as one of my main questions and limitations of the evaluation procedure.\nUnfortunately, this central question could not be addressed due to the dataset used and the evaluation pipelines. The authors, though, elaborated on them, and I believe they will discuss them in the final version of the paper.\nFor this reason, I confirm my original recommendation score.", "rebuttal": "Addressing Weaknesses\n1. Experimental Results Limited to Specific Dataset and Seen/Unseen Distinction:\n\nWe acknowledge the reviewer's point regarding the experimental results being primarily confined to a specific dataset (R2R-CE) and not explicitly distinguishing between seen and unseen scenes and objects in a dedicated manner. Our primary focus in this initial work was to demonstrate the efficacy of our Hierarchical Semantic-Augmented Navigation (HSAN) framework on a challenging, long-horizon navigation benchmark, establishing a strong foundation with novel scene graph construction, optimal transport-based planning, and graph-aware reinforcement learning.\n\nWhile the R2R-CE dataset inherently includes distinct training and testing environments (unseen scenes), we agree that a more granular analysis of generalizability to entirely novel objects and broader scene variations would further strengthen our claims. This is a crucial direction for future work, and we will explicitly mention this limitation and future research avenue in our revised manuscript. We believe that the hierarchical semantic scene graph, by capturing multi-level environmental representations, lays a robust groundwork for better generalization, and we plan to explore this in subsequent research.\n\n2. Missing References:\nWe sincerely apologize for the oversight in not including several highly relevant works in our initial submission. We will rectify these omissions and incorporate these references into the related work section, discussing their relevance and distinguishing our contributions where appropriate. For instance:\n\nHydra and ConceptGraphs: These works provide excellent frameworks for general 3D scene graph construction. Our approach, while building a scene graph, is specifically tailored for Vision-Language Navigation in continuous environments, focusing on semantic and topological relevance for planning, integrating VLM outputs, and handling dynamic graph updates. We will clarify how our scene graph generation adapts and leverages VLM capabilities for navigation tasks.\n\nLeLaN and Select2Plan: These represent important advancements in VLM-based navigation. LeLaN explores learning policies from in-the-wild videos, complementing our structured approach with real-world data insights. Select2Plan's focus on training-free ICL-based planning and memory retrieval offers an interesting contrast to our optimal transport and RL-driven planning, highlighting different paradigms for leveraging VLMs. We will discuss these in the context of our contributions to structured planning and robust control.\n\nResponses to Questions\nQ1 - Scene-Graph Generation:\nOur scene graph generation methodology was specifically designed to serve the unique requirements of hierarchical planning and reasoning within our HSAN framework, leveraging vision-language models for multi-level semantic understanding (objects, regions, zones). While Hydra and ConceptGraphs offer powerful general scene graph construction, our approach prioritizes the creation of a dynamic, semantically-rich graph optimized for navigation, allowing for direct integration with our optimal transport planner and graph-aware RL policy. This bespoke design enabled us to ensure that the graph directly supports balancing semantic relevance and spatial accessibility with theoretical guarantees. Incorporating established, general-purpose scene graph systems like Hydra or ConceptGraphs would be an interesting avenue, but it might introduce complexities in adapting their general schemas to our specific navigation needs and integrating them seamlessly with our VLM-based semantic understanding and planning modules. It would likely involve a substantial re-engineering of the graph representation and planning interfaces.\n\nRegarding the location aggregation feature being an average of object features: This was chosen for its simplicity, computational efficiency, and effective capture of the aggregated semantic essence of a location given the diverse range of objects it might contain. While a simple average might appear to be a strong assumption, our experiments indicate its practical effectiveness in grounding the spatial reasoning within the optimal transport framework. We agree that learning this aggregation as a graph neural network (GNN) aggregation or similar learnable mechanism is a highly promising direction. This could allow for more nuanced and context-dependent weighting of object features, potentially improving robustness and generalization. We will highlight this as an exciting area for future research.\n\nQ2 - Data Availability:\nThe assumption that the robot has a panoramic view and ground-truth (GT) positioning is indeed common in current academic benchmarks for Vision-Language Navigation (e.g., R2R, RxR, R2R-CE), which typically rely on simulated environments with perfect sensing capabilities. This allows researchers to focus on the core challenges of vision-language understanding, planning, and control without confounding factors from imperfect localization or limited field of view.\n\nIn the absence of such perfect signals, the overall performance would undoubtedly be affected. Without a panoramic view, the agent would need to actively explore to build a comprehensive understanding of its immediate surroundings, potentially impacting planning efficiency and obstacle avoidance. The absence of GT positioning would necessitate the integration of robust SLAM (Simultaneous Localization and Mapping) or visual odometry techniques, which introduce their own error accumulation and drift. This would likely lead to degraded navigation accuracy, increased path length, and potentially more failures in long-horizon tasks. Our framework, particularly the graph-aware reinforcement learning policy, is designed for robust low-level control and obstacle avoidance, which would be crucial in scenarios with imperfect sensing. However, adapting HSAN to real-world deployment would require incorporating robust state estimation and perception modules, which we consider a vital next step for our research.\n\nQ3 - Seen and Unseen Generalizability:\nThe R2R-CE dataset, while providing unseen scenes in its test split, does not explicitly delineate unseen objects or entirely novel query structures that are outside the training distribution. Our current training and fine-tuning operations on R2R-CE inherently expose the model to the objects and language patterns present within that dataset.\n\nReplicating an experiment similar to Select2Plan's table construction, which demonstrates strong generalization to unseen objects through a training-free, in-context learning approach, is a compelling suggestion. While our current framework relies on learned graph representations and policies, the underlying VLM components do possess strong zero-shot and few-shot capabilities. We believe that by integrating more advanced open-vocabulary perception modules or by exploring a hybrid approach that combines our structured planning with training-free VQA and memory retrieval, we could significantly enhance generalization to unseen objects. However, a full replication of such an experiment would necessitate a new dataset, extensive re-evaluation, and potentially architectural modifications beyond the scope of this paper. We will acknowledge this important challenge and propose it as a significant direction for future work, perhaps exploring how our optimal transport-based planning could leverage open-vocabulary semantic maps constructed by such methods.\n\nQ4 Effect of Lambda in Equation 1:\nWe apologize for the omission of a detailed discussion on the parameter lambda (λ) in Equation 1 within the main paper or appendix. Lambda is a crucial hyperparameter that balances the semantic relevance and spatial accessibility within our optimal transport-based topological planner. Specifically:\n\nHigher λ: Places a stronger emphasis on semantic relevance. This means the planner will prioritize selecting long-term goals that are highly semantically aligned with the instruction, even if they are spatially more distant or require traversing more complex paths. This can be beneficial when the instruction provides very specific semantic cues.\n\nLower λ: Places a stronger emphasis on spatial accessibility. The planner will favor goals that are geometrically closer and easier to reach, even if their semantic relevance is slightly lower. This is useful when the instruction is more general or when efficient navigation is paramount.\n\nTuning λ allows us to control the trade-off between semantic precision and navigational efficiency. In our experiments, λ was empirically tuned to achieve the best performance on the validation set, typically favoring a balance that allows for both accurate semantic understanding and feasible path generation. \n\nQ5 Language in the Queries:\nIn Section 4, our assessment of multilingual capabilities primarily demonstrated that HSAN, by leveraging pre-trained multilingual vision-language models, can interpret navigation instructions in different languages (e.g., English, Chinese, Hindi) and successfully navigate. The core insight is that the semantic understanding component of our framework is largely language-agnostic due to the robust multilingual embeddings provided by the underlying VLMs.\n\nHowever, we did observe some nuanced behaviors. For example,\n\nPerformance Consistency: While the model generally maintains robust performance across languages, slight variations were noted, often correlating with the language's representation strength in the pre-training data of the VLM. Languages with more extensive pre-training data (e.g., English) sometimes yielded marginally better performance due to richer semantic understanding."}, {"review_id": "Qblfc7QBpW", "reviewer": "Reviewer_w8P6", "summary": "The authors propose HSAN, a method for VLN-CE which constructs a hierarchical scene graph and optimal transport-based topological planner to select navigation goals aligned with language instructions. In addition, the authors propose a novel approach for low level control to avoid obstacles by augmenting PPO with a graph based understanding of the environment. HSAN achieves state-of-the-art performance across all simulation benchmarks for VLN-CE.", "questions": "How does the benefit of the new proposed low level control policy compare to classical planners for obstacle avoidance which are state of the art in robotics? \n\nFor example, PPO was initially introduced as a method for point to point navigation when the goal location existed outside the observed map region or no map was constructed. PPO implicitly predicted occupied and unoccupied regions in order to enable obstacle avoidance in these scenarios. To my understanding, your navigation waypoints are inside an observed map region for which you have an explicit map. Your citation to [1] which you use to motivate your decision to modify PPO explicitly refers to the case in which an explicit map is not constructed.\n\n1. Jacob Krantz and Stefan Lee. Navigating to objects in the real world. arXiv preprint 537 arXiv:2112.06758, 2021. doi: 10.48550/arXiv.2112.06758. \n\nSo, in summary, why would you use your proposed control method over approaches like fast marching squares, A*, etc. which are currently deployed with high performance on physical robots to do point to point navigation in regions for which you built an explicit map?\n\nI would be open to increasing my score if my concerns about the low level planning method are resolved and the writing is improved both to address the real world evaluation limitations and the inaccuracies in phrasing.", "limitations": "The limitations regarding real world hardware deployment are not discussed and misrepresented as not a challenge. I discussed further in the weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The experimental evaluation of the method in simulation was very impressive with thorough ablations and baselines on every significant benchmark for the task. I was particularly impressed by the results on the multilingual VLN benchmark, a challenge which is often overlooked in evaluating approaches to VLN and aided by this method design. Also, the experimental results are state of the art across benchmarks, and the connection between optimal transport and spatial map reasoning is interesting and novel.\n\nThere seem to be significant barriers to running this method in the real world on a robot which are substantially downplayed by the authors. The authors in fact emphasize how their approach is well suited to running on a physical robot and do not acknowledge the VLN methods which already do so and discuss this method’s unique limitations in this regard other than too slow inference speed of the approach. In addition to inference speed, the required task specific training of the VLM and GCN will have significant challenges performing robustly in open world physical environments. There will be sim-to-real gaps with the trained models even in similar real world environments, and the type of environments in the simulators used for training is far from the scope of different environment types in the physical world (for example, no offices are included). In future work, the authors claim intentions to show results in dynamic and outdoor environments, but no results are currently shown in indoor environments. Real world results for this method are particularly important because part of the claimed contribution is low level physical control of a robot.\n\nSome missing citations for papers with similar map and/or VLM based solutions showing real world instruction following results.\n\n1. Sonia Raychaudhuri, Duy Ta, Katrina Ashton, Angel X Chang, Jiuguang Wang, Bernadette Bucher. Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigation. arXiv, 2024.\n2. Zhaohuan Zhan, Lisha Yu, Sijie Yu, Guang Tan. MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chains\n3. Yuxing Long, Xiaoqi Li, Wenzhe Cai, Hao Dong. Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions. ICRA, 2024.\n\n\nThere are multiple issues with phrasing. The authors appear to be misusing the word “benchmark”. They repeatedly state that HSAN is a “new benchmark.” However, it appears what they mean is that HSAN is the “new state of the art baseline” for the existing benchmarks addressing this problem on vision-language navigation in continuous environments. In addition, there are a few incorrect phrases like HSAN “redefines VLN-CE”. I also don’t think that the scene graph is “dynamic” since it does not update with movement in the scene. The authors should be careful with their phrasing. \n\nThe code was not included in the supplement, and there is no condition ensuring that it will be included in an accepted paper, limiting reproducibility of this work.\n\nI have concerns about the significance of the low level control method discussed in questions.", "was_revised": true, "final_justification": "The authors give more thorough textual explanations throughout the rebuttal which more clearly and appropriately frame the contributions of each component of their method. I believe from these justifications that they will clearly update the text in the paper to improve the writing which was a concern from all the reviewers. The experimental results are strong and make progress with interesting methodological insights on a challenging problem.", "rebuttal": "Addressing Weaknesses\n\nWe are very grateful to Reviewer 2 for their positive assessment of our experimental evaluation, the thoroughness of our ablations and baselines, and the impressive results on the multilingual VLN benchmark. We are also pleased that the connection between optimal transport and spatial map reasoning was found to be interesting and novel. We appreciate the constructive feedback regarding the real-world applicability, phrasing, and reproducibility, and address each point below.\n\n1. Significant Barriers to Real-World Deployment Downplayed:\n\nWe acknowledge the reviewer's valid and important concern regarding the practical challenges of deploying HSAN on a physical robot in real-world environments. We agree that our initial phrasing may have inadvertently downplayed these complexities. Our intention in stating that our approach is \"well suited to running on a physical robot\" was to highlight the design principles of HSAN (e.g., modularity, hierarchical structure, semantic grounding) that we believe are conducive to real-world deployment, rather than implying immediate, out-of-the-box readiness. We understand that significant engineering and research efforts are required to bridge the sim-to-real gap.\n\nSpecifically:\n\nSim-to-Real Gap: We concur that sim-to-real gaps are a major challenge for learned policies, including those relying on VLMs and GCNs. While our current work focuses on foundational algorithmic advancements in simulation, we believe that the structured, semantic representations learned by HSAN can mitigate some aspects of this gap by providing more interpretable and potentially transferable knowledge compared to end-to-end black-box approaches. Domain adaptation techniques and real-world data collection would be essential for robust deployment.\n\nEnvironment Scope: We acknowledge that the simulator environments used for training (e.g., Matterport3D) do not encompass the full diversity of physical world environments (e.g., offices, outdoor settings). Our claims regarding future work on dynamic and outdoor environments are indeed aspirational and represent long-term research goals. This initial submission focuses on establishing the core HSAN framework and its efficacy in complex indoor environments.\n\nLow-Level Control Contribution: The low-level control policy, while crucial for fine-grained navigation and obstacle avoidance, is currently trained in simulation. Its real-world performance would depend heavily on accurate perception and robust sim-to-real transfer. We will clarify that this component, like others, would require further adaptation and validation for physical robot deployment.\n\nWe will revise the manuscript to explicitly acknowledge these real-world deployment challenges, temper our claims about immediate applicability, and emphasize that our current work lays the algorithmic groundwork for future real-world implementations.\n\n2. Missing Citations for Real-World VLN Methods:\n\nWe sincerely apologize for the omission of highly relevant real-world instruction following papers. We agree that \"Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigation\" (Raychaudhuri et al., 2024), \"MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chains\" (Zhan et al.), and \"Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions\" (Long et al., 2024) are crucial works demonstrating real-world VLN capabilities.\n\nWe will rectify this by incorporating these references into the related work section. We will discuss how these methods demonstrate successful real-world deployment and how their approaches to map representation, VLM integration, and instruction following compare to our simulation-focused, hierarchical, and optimal-transport-driven framework. This will provide a more comprehensive overview of the state-of-the-art in VLN, both in simulation and real-world settings.\n\n3. Phrasing Issues:\n\nWe appreciate the reviewer's meticulous attention to our phrasing and apologize for the inaccuracies. We agree that precision in language is paramount, especially in scientific writing.\n\nWe will revise instances of \"new benchmark\" to more accurately reflect that HSAN establishes a \"new state-of-the-art baseline\" or \"achieves state-of-the-art performance\" on existing benchmarks for vision-language navigation in continuous environments.\n\nPhrases like \"redefines VLN-CE\" will be toned down to more appropriate statements such as \"advances VLN-CE\" or \"introduces a novel framework for VLN-CE.\"\n\nRegarding the term \"dynamic\" scene graph: We intended \"dynamic\" to mean that the scene graph is constructed and continuously updated as the agent explores the environment and gathers new observations, rather than being a static, pre-built representation of the entire scene. This contrasts with approaches that assume a complete, pre-existing map. We will clarify this distinction in the text, perhaps using terms like \"adaptive\" or \"online-built\" scene graph to avoid ambiguity regarding moving scene elements.\n\nWe are committed to a thorough review of the manuscript to address all such phrasing issues and ensure greater accuracy and modesty in our claims.\n\n4. Code Reproducibility:\n\nWe acknowledge the reviewer's concern regarding code reproducibility. We understand the importance of open-sourcing our implementation for the scientific community. We commit to releasing the full code and trained models for HSAN upon the acceptance of the paper. We will include a statement to this effect in the camera-ready version of the manuscript.\n\nResponses to Questions\nQ1 - Low-Level Control Policy vs. Classical Planners:\n\nWe thank the reviewer for this insightful question, which allows us to clarify the role and benefits of our graph-augmented PPO low-level control policy in the context of our hierarchical framework.\n\nOur low-level control policy is not intended to replace classical global path planners like A* or Fast Marching Squares (FMS) for point-to-point navigation in fully observed, explicit maps. Instead, it serves a distinct and complementary role within our hierarchical system:\n\nHierarchical Integration: The optimal transport-based topological planner operates at a higher level, selecting long-term semantic goals and generating a sequence of topological waypoints. The graph-augmented PPO then handles the local, continuous navigation between these waypoints, including fine-grained obstacle avoidance. This allows for a clean separation of concerns: global semantic reasoning and topological planning at the high level, and robust, reactive control in the continuous environment at the low level.\n\nRobustness in Continuous Spaces: While classical planners excel with precise, static occupancy grids, real-world environments are often noisy, dynamic, and subject to perceptual uncertainty. Our RL policy learns to generalize obstacle avoidance from diverse experiences, which can offer greater robustness to perceptual noise, minor dynamic changes (e.g., small moving objects not explicitly modeled), and variations in terrain that might not be perfectly captured by a static local map. It learns a more \"fluid\" and adaptive navigation behavior.\n\nSemantic Awareness at Low Level: The graph augmentation provides the PPO policy with semantic context from the scene graph. This allows the low-level controller to make more informed decisions beyond pure geometry. For instance, it can learn to approach semantically relevant sub-regions more carefully or avoid specific types of objects (e.g., \"fragile items\") even if they are not explicitly marked as obstacles in a purely geometric map. This semantic grounding at the control level is a key differentiator from purely geometric classical planners.\n\nMotivation from Krantz and Lee (2021) Clarification: The citation to Krantz and Lee (2021) was used to highlight the general challenge of navigation in environments where a complete, explicit map might not be available or fully reliable for fine-grained control. While our topological planner operates on a high-level graph (which can be seen as a form of implicit map for long-range planning), the low-level PPO needs to handle immediate, continuous navigation where a perfect, explicit, and static local occupancy map for every micro-obstacle might not be practical or robust in real-time. The PPO learns to implicitly infer traversability and avoid collisions based on visual observations and semantic context, which is beneficial when explicit, perfectly accurate local maps are not always guaranteed or are computationally expensive to generate at high frequency.\n\nIn summary, our graph-augmented PPO is not a substitute for classical global planners but a complementary component designed for robust, semantically-aware, and reactive low-level control in continuous environments, bridging the gap between high-level topological goals and physical robot actions. It leverages the strengths of reinforcement learning to handle perceptual noise and generalize obstacle avoidance, while benefiting from the structured guidance of the higher-level optimal transport planner.\n\nWe will revise the manuscript to clarify the specific role and advantages of our low-level control policy, particularly its complementary nature to classical planning methods in a hierarchical architecture, and its benefits in handling continuous, noisy, and semantically rich environments."}, {"review_id": "VpbnvwosVZ", "reviewer": "Reviewer_3DNH", "summary": "The paper constructs two types of graphs to address the VLN-CE task. For long-distance navigation point planning, a topological link graph is built, and the optimal navigation trajectory is selected based on OT theory. For low level action control, a hierarchically constructed sense graph is used to train the policy based on the current observation.", "questions": "1. Regarding the Hierarchical Semantic Scene Graph, is semantic aggregation at different levels really necessary? It seems that for low-level control, simply avoiding obstacles and reaching a predefined ghost node is sufficient. How does the region information help with navigation? However, ablation studies appear to demonstrate its necessity. More discussion is helpful.\n\n2. In this paper, it seems that navigation decisions are mainly based on the similarity of individual nodes and the instruction, which makes me skeptical about the final navigation results of the model. From my understanding, both R2R and RxR datasets contain many procedural instructions describing navigation paths, making it difficult to match the endpoint based solely on instruction similarity.\n\n3. How is map information incorporated into the navigation model? If the method uses the global adjacency relationships between nodes, does it align with the unseen setting used in previous methods?", "limitations": "Yes.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths**:\n- The paper's approach to constructing scene graphs is somewhat insightful and provides inspiration.\n\n**Weaknesses**:\n- There are significant issues in the writing quality:\n  1. The main text contains large sections of redundant content, for example, the overlapping material between lines 67-78 and lines 84-92.\n  2. Several essential details are missing in the method description, such as how the formulas on line 241 are applied to VLN. This lack of detail makes it difficult to assess the validity of the proposed methods and their correctness.\n  3. The overall description of the methodology is lacking, with unclear explanations of the inputs and outputs of each module and how they interconnect.\n  4. The purpose of designing the Hierarchical Semantic Scene Graph Construction seems to be solely for low-level control, which appears to be of limited significance. The motivation needs to be more thoroughly explained, especially concerning the relationships between the two types of sense graphs constructed in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "Addressing Weaknesses\nWe thank Reviewer 3DNH for their summary and constructive feedback. We appreciate the recognition of the insight in our scene graph construction. We address the identified weaknesses and questions below.\n\n1. Writing Quality: Redundant Content and Missing Details:\n\nWe sincerely apologize for the identified issues in writing quality, particularly the redundancy and lack of essential details. We acknowledge the overlapping material between lines 67-78 and lines 84-92 and will thoroughly revise these sections to eliminate repetition and improve conciseness.\n\nWe also recognize the critical importance of clarity in method descriptions. We commit to providing more explicit details regarding the application of formulas (specifically the ones on line 241) to the VLN task. We will ensure that the inputs, outputs, and interconnections of each module within the HSAN framework are clearly explained, potentially with an updated system diagram, to enhance the overall understanding of our methodology.\n\n2. Limited Significance of Hierarchical Semantic Scene Graph for Low-Level Control and Motivation:\n\nWe understand the reviewer's concern regarding the perceived limited significance of the Hierarchical Semantic Scene Graph (HSSG) solely for low-level control and the motivation behind its design, especially concerning its relationship with the topological graph. We would like to clarify that the HSSG serves a crucial dual purpose within our framework, impacting both high-level planning and low-level control:\n\nHigh-Level Planning (via Optimal Transport): The HSSG, particularly its higher-level semantic nodes (regions, zones), provides the rich semantic context necessary for our optimal transport-based topological planner. The planner uses the semantic relevance of these regions/zones, alongside spatial accessibility, to select long-term goals that are semantically aligned with the language instruction. This is far beyond simple obstacle avoidance; it's about understanding the meaning of the environment in relation to the instruction.\n\nLow-Level Control (Graph-Augmented PPO): While low-level control does involve obstacle avoidance, it's not solely about that. The HSSG provides the graph-augmented PPO policy with a semantically enriched local understanding of the environment. This means the agent can make more intelligent, instruction-conditioned local decisions. For example, if the instruction is \"go to the kitchen past the dining table\", the region and object information from the HSSG helps the low-level policy understand the semantic context of the \"dining table\" and how to navigate around or past it, rather than just treating it as a generic obstacle. The hierarchical nature allows the policy to reason about objects within a region and regions within a zone, providing a multi-granular understanding that enhances the robustness and instruction-following fidelity of local actions.\n\nWe will revise the motivation section to explicitly articulate this dual role and the synergistic relationship between the topological graph (for global, long-horizon path selection) and the hierarchical semantic scene graph (for local, semantically-grounded control and planning context).\n\nResponses to Questions\nQ1 - Necessity of Semantic Aggregation at Different Levels and Region Information for Navigation:\n\nWe appreciate this question, as it directly addresses the core design of our HSSG. We contend that semantic aggregation at different levels (objects, regions, zones) is indeed necessary and provides significant benefits beyond simply avoiding obstacles:\n\nBeyond Obstacle Avoidance: For low-level control, simply avoiding obstacles is insufficient for complex VLN tasks. The agent needs to understand what it is avoiding, where it is in relation to semantic landmarks, and how to approach or traverse certain areas based on the instruction. Region information, for instance, helps the agent understand that it is currently in a \"kitchen\" or approaching a \"bedroom,\" which can be crucial for interpreting instructions like \"find the coffee machine in the kitchen\" or \"turn left after the living room.\"\n\nContextualized Local Decisions: Semantic aggregation allows the low-level policy to make contextualized decisions. For example, if the instruction implies passing through a \"narrow corridor,\" the region information helps the policy adapt its movement behavior (e.g., slow down, move centrally) in that specific semantic context. Without this, the policy would treat all narrow passages generically. The ablation studies indeed confirm that incorporating region and zone information significantly improves navigation performance, demonstrating its necessity for robust and instruction-aligned local control. We will expand this discussion in the paper, providing more concrete examples of how region and zone information aid navigation.\n\nQ2 - Navigation Decisions based on Node/Instruction Similarity vs. Procedural Instructions:\n\nThe reviewer raises a very important point regarding procedural instructions in R2R and RxR datasets. We agree that simply matching the endpoint based solely on instruction similarity would be insufficient. Our model's navigation decisions are not solely based on the similarity of individual nodes to the instruction. Instead, our optimal transport-based topological planner leverages semantic similarity in a more sophisticated way:\n\nDistribution over Topological Graph: The language instruction is used to compute a semantic relevance distribution over all nodes in the topological graph (which correspond to navigable locations). This distribution assigns higher \"mass\" to nodes that are semantically relevant to the instruction, not just the final goal.\n\nOptimal Transport for Path Selection: The optimal transport framework then finds a \"flow\" or \"transport plan\" from the agent's current location to this target semantic distribution. This process inherently considers the entire path and the accumulated cost of traversing edges, balancing both the semantic relevance of intermediate waypoints and their spatial accessibility. It effectively finds a path that \"moves\" the agent's initial mass distribution towards the target semantic distribution, minimizing a cost that incorporates both spatial distance and semantic mismatch along the way.\n\nTherefore, our method implicitly handles procedural instructions because the optimal transport formulation encourages the selection of a path that aligns with the sequence of semantic cues described in the instruction, not just the final destination. The instruction guides the shape of the target distribution, and the optimal transport finds the most efficient path through the graph to fulfill that distributed semantic goal. We will clarify this nuanced application of optimal transport in the methodology section.\n\nQ3 - Map Information Incorporation and Global Adjacency in Unseen Settings:\n\nWe clarify that \"map information\" in our method refers to the topological graph and the Hierarchical Semantic Scene Graph (HSSG), both of which are constructed online as the agent explores the environment.\n\nOnline Graph Construction: Our method does not assume a pre-built global map or pre-existing global adjacency relationships. The topological graph is incrementally built by connecting observed navigable locations, and the HSSG is constructed by segmenting and semantically understanding the observed environment. This process is entirely consistent with the \"unseen setting\" in VLN, where the agent navigates novel environments without prior knowledge.\n\nAlignment with Unseen Setting: Since the graphs are built from the agent's real-time observations, they naturally align with the unseen setting. The \"global adjacency relationships between nodes\" are derived from the observed connectivity as the agent explores and are continuously updated. This means our method learns to build and use its internal representation of the environment during navigation, making it applicable to novel scenes. We will emphasize this online construction aspect more clearly in the paper."}, {"review_id": "SRtNUfiH12", "reviewer": "Reviewer_c7PD", "summary": "This work proposes a new method HSAN for vision-language navigation task. The main components include dynamic hierarchical scene graph, topological planner and RL-based low level control.", "questions": "See above.", "limitations": "There is no limitation discussed in the conclusion section.", "rating": 3, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well-written.\n- Sufficient experiments are conducted with multiple datasets.\n- Ablation studies are discussed.\n\nWeakness:\n\n- In implementation details, it mentioned the system runs with around 5 FPS. I also expect there are some comparison with other methods for the inference efficiency. I see the system contains several complex steps including the detection, scene graph, etc. I'm worried if the too complex system will be the challenge in the real life deployment.\n- The failure analysis is not presented. Especially in the qualitative examples. I'm curious that in which situation the system will fail. And what will be the bottleneck of this system? Are these bottlenecks easy to be fixed in the future works or it's the fundamental issue of this design?\n- Currently all the experiments are based on the simulation instead of real world. I understand the real world experiments are not feasible for everyone. But there is also no discussion about the perturbation applied to the simulation, to make the scenario closer to the real world.\n- One majority concern is, there is an assumption that the perception output is perfect. What if the perception output is noisy, and is there any countermeasure applied to the method to make it work robustly?\n\nMinor Weakness:\n\n- Typo page 4, line 202: “a aggregated” → “an aggregated”\n- Typo page 5, line 264: “accion” → “action”\n\nOverall, I’m borderline for this work. Especially some concerns I mentioned above, I'm fine to adjust my ratings if my concerns get solved.", "was_revised": true, "final_justification": "While the paper is clearly written and supported by the experiments and ablations, the main concerns about robustness remain insufficiently addressed. The paper also a failure analysis (for any quantitative or qualitative) and does not evaluate under realistic perturbations. Besides, the method assumes perfect perception without noises, making real-world reliability hard to judge. The rebuttal acknowledges these issues but adds no new empirical evidence, and only future-work statements, so my concerns persist. Also, the paper reports no analysis on runtime or resource; given the system deploy-ability is uncertain. I therefore maintain a borderline rating: the idea of the approach is promising, but it needs concrete evidence on above concerns.", "rebuttal": "Addressing Weaknesses\nWe thank Reviewer c7PD for their concise summary and constructive feedback. We appreciate the recognition of our paper's writing quality, experimental thoroughness, and ablation studies. We address the identified weaknesses and questions below.\n\n1. Inference Efficiency and Real-World Deployment Concerns:\n\nWe acknowledge the reviewer's concern regarding the reported 5 FPS inference speed and the complexity of the system, particularly in the context of real-world deployment.\n\nFPS Comparison: Direct, fair comparisons of inference speed with other methods are challenging due to significant architectural differences and varying computational demands of their components (e.g., different VLMs, graph processing, planning algorithms). Our reported 5 FPS is for the entire end-to-end system on the specified hardware. While we did not include a dedicated comparison in the paper, we agree that it is an important consideration for practical applications.\n\nSystem Complexity: We understand that the multi-component nature of HSAN (detection, scene graph construction, topological planning, RL control) adds complexity. Our current work is a research prototype focused on demonstrating the algorithmic novelty and achieving state-of-the-art performance in simulation. Optimizing the system for real-time, low-latency deployment on physical robots is a critical next step. We believe that the modular design of HSAN, however, could facilitate future optimizations, allowing for parallelization of components or replacement of specific modules with more efficient counterparts (e.g., lighter-weight VLMs, hardware-accelerated graph operations) without fundamentally altering the overall hierarchical framework. We will clarify this in the discussion of real-world applicability.\n\n2. Failure Analysis and Bottlenecks:\n\nWe appreciate the suggestion for a more explicit failure analysis and discussion of bottlenecks. We agree that understanding the limitations is crucial. Common failure modes for HSAN include:\n\nVLM Misinterpretation: Despite using powerful VLMs, ambiguous instructions or visually similar objects/regions can lead to incorrect semantic grounding, causing the high-level planner to select suboptimal goals.\n\nComplex Spatial Reasoning: In highly cluttered or geometrically challenging environments, even with accurate semantic understanding, the low-level policy might struggle with precise path execution or get stuck in local minima.\n\nUnobserved Semantic Elements: If a crucial semantic landmark mentioned in the instruction is not visible or detectable within the agent's current field of view and has not been incorporated into the scene graph, the agent may fail to find the correct path.\n\nAccumulated Low-Level Errors: Small errors in low-level control can accumulate over long trajectories, leading to significant deviations and eventual failure.\n\nThe primary bottlenecks often lie in:\n\nVLM Inference: The computational cost of running large VLMs for semantic understanding and feature extraction.\n\nGraph Construction and Updates: The frequency and complexity of updating the hierarchical scene graph, especially in dynamic environments.\n\nWe believe these bottlenecks are largely addressable in future work:\n\nFaster VLMs: Ongoing research in efficient foundation models and model compression will directly benefit inference speed.\n\nOptimized Graph Algorithms: More efficient graph construction, update, and query algorithms can reduce computational overhead.\n\nRobust Perception: Improving the robustness of perception modules to handle noise and partial observations will directly mitigate failures related to misinterpretation and unobserved elements.\n\nWe will add a dedicated section on failure analysis and bottlenecks, potentially with qualitative examples, to provide these insights.\n\n3. Simulation Perturbations and Real-World Discussion:\n\nWe acknowledge that all our experiments are currently based on simulation. While real-world experiments are indeed resource-intensive and beyond the scope of this initial paper, we agree that a discussion on perturbations applied to simulation to bridge the sim-to-real gap is valuable. Our current simulation setup (R2R-CE) provides realistic 3D environments, but we did not explicitly introduce additional noise or perturbations (e.g., sensor noise, dynamic obstacles, localization errors) during training or evaluation.\n\nWe will explicitly state this limitation and reiterate that bridging the sim-to-real gap is a critical future direction. This would involve incorporating robust perception modules, training with domain randomization or real-world data, and developing adaptive control strategies that can handle real-world uncertainties. Our current work focuses on the fundamental algorithmic contributions of hierarchical planning and graph-based reasoning.\n\n4. Perfect Perception Assumption and Robustness to Noise:\n\nThe reviewer correctly identifies that our current work assumes near-perfect perception output (e.g., accurate object detection, semantic segmentation, and localization within the simulator). This is a common simplification in foundational VLN research to isolate the challenges of language understanding and navigation planning.\n\nWe acknowledge that real-world perception is inherently noisy. Potential countermeasures to make HSAN more robust to noisy perception include:\n\nUncertainty-Aware Graph Construction: Incorporating uncertainty estimates into the scene graph nodes and edges, allowing the planner to account for perceptual ambiguity.\n\nRobust State Estimation: Integrating robust visual-inertial odometry (VIO) or SLAM systems that provide more resilient pose estimates in the face of sensor noise.\n\nPerception-in-the-Loop Training: Training the low-level RL policy with noisy or imperfect visual inputs to make it more resilient to real-world sensor data.\n\nMulti-modal Fusion: Leveraging multiple sensor modalities (e.g., LiDAR, depth, RGB) to obtain a more robust environmental understanding.\n\nSemantic Consistency Checks: Implementing mechanisms to verify semantic consistency over time or across different viewpoints to filter out spurious detections.\n\nWe will add a discussion on this assumption and potential countermeasures in the revised manuscript.\n\n5. Minor Weaknesses (Typos):\n\nWe thank the reviewer for pointing out the typos. We will correct them immediately:\n\nPage 4, line 202: \"a aggregated\" → \"an aggregated\"\n\nPage 5, line 264: \"accion\" → \"action\""}, {"review_id": "K5sXFKXWmA", "reviewer": "Reviewer_ks3a", "summary": "This paper presents HSAN (Hierarchical Semantic-Augmented Navigation), a novel framework for Vision-Language Navigation in Continuous Environments (VLN-CE). HSAN incorporates three synergistic modules: (1) a dynamic hierarchical semantic scene graph capturing object–region–zone structures using vision-language models, (2) an optimal transport-based topological planner grounded in Kantorovich’s duality to select long-horizon goals, and (3) a graph-aware reinforcement learning controller for low-level action execution. The method is evaluated on two standard VLN-CE benchmarks (R2R-CE, RxR-CE) and achieves state-of-the-art performance with solid ablations and generalization analysis.", "questions": "The main concerns are listed in the Weaknesses section. If the authors can provide more detailed experimental analysis to support the effectiveness of each proposed component, it would address my main confusion and could improve my overall evaluation.", "limitations": "No. The authors should include an analysis of failure cases, a discussion of computational cost and efficiency, and whether the method can support real-world robotic execution.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper proposes a clearly structured three-stage framework (graph construction, planning, control) that directly addresses key challenges in VLN-CE.\n2. The use of optimal transport for long-term goal selection is interesting and supported by a rigorous formulation.\n3. The method achieves strong performance on both R2R-CE and RxR-CE, outperforming a range of competitive baselines across multiple metrics.\n\nWeaknesses\n1. The notation system is overloaded and lacks clarity. The paper introduces many symbols without a consolidated reference. A notation table would improve readability and help readers follow the methodology.\n2. The model includes numerous hyperparameters (e.g., σ, λ in L199; α, β in L232), yet no analysis is provided regarding their sensitivity or selection rationale. This raises concerns about robustness and reproducibility.\n3. In the Comparison to Baselines section, the improvements over methods like NTS and SMN are credited to HSAN’s dynamic hierarchical graph. This raises a natural question: how would NTS or SMN perform if they used the same graph structure? The same concern applies to other modules. It is not clear whether the performance gain really comes from these modules, since the paper does not test them in other methods or provide direct evidence.**\n4. The paper lacks visualizations of qualitative results, and also does not discuss any failure cases\n5. In the Discussion section, the paper states that “The hierarchical semantic scene graph enables nuanced scene understanding…” and makes similar claims about other modules. However, I don’t see how this conclusion is supported by the experiments. The results only show that the full model performs well, but there is no direct evidence isolating the contribution of this specific component.", "was_revised": true, "final_justification": "I think the core design is introducing Optimal Transport (semantic scene graph and its varients are often used in VLN). However, simply using Optimal Transport to balance semantic relevance and spatial accessibility cannot convince me. The motivaton and the necessity  is not described well to support this point clearly.", "rebuttal": "Addressing Weaknesses\nWe thank Reviewer ks3a for their comprehensive summary and detailed feedback. We appreciate the recognition of our structured framework, the interesting use of optimal transport, and the strong performance achieved. We address the identified weaknesses and questions below.\n\n1. Notation System Overload and Lack of Clarity:\n\nWe sincerely apologize for the overloaded notation system and the resulting lack of clarity. We agree that a consolidated reference would significantly improve readability. We will address this by:\n\nAdding a Notation Table: We will include a dedicated notation table at the beginning of the methodology section or in an appendix to list and define all symbols used throughout the paper.\n\nStreamlining Notation: We will review the notation to identify opportunities for simplification or consistency, ensuring that each symbol is clearly introduced and consistently used.\n\n2. Hyperparameter Sensitivity and Selection Rationale:\n\nWe acknowledge the reviewer's valid concern regarding the lack of detailed analysis on hyperparameter sensitivity and selection rationale. This is a crucial point for reproducibility and understanding the robustness of our method.\n\nEmpirical Tuning: The hyperparameters (e.g., σ, λ, α, β) were primarily selected through empirical tuning on the validation set to optimize performance across the various metrics. This is a common practice in complex deep learning systems.\n\nSensitivity Analysis (in Appendix): We will add a new subsection in the appendix dedicated to hyperparameter sensitivity analysis. This will include:\n\nA brief description of how each key hyperparameter was chosen.\n\nPlots or tables illustrating the impact of varying these parameters on key performance metrics. This will demonstrate the robustness of our method within a reasonable range of hyperparameter values and provide guidance for future applications.\n\nSpecifically for λ (as also noted by Reviewer 1), we will elaborate on its role in balancing semantic relevance and spatial accessibility.\n\n3. Comparison to Baselines and Direct Evidence of Module Contribution:\n\nThe reviewer raises a very important and insightful question regarding the attribution of performance gains to specific modules, especially when comparing HSAN to baselines like NTS and SMN. We agree that simply crediting improvements to HSAN's graph without direct evidence of how the same graph would benefit other methods is a fair criticism.\n\nOur current ablation studies (e.g., removing hierarchical levels from the scene graph or replacing optimal transport with simpler planning) aim to isolate the contribution of our specific design choices within the HSAN framework. However, the reviewer's point about integrating our graph structure into other methods to demonstrate its standalone benefit is well-taken.\n\nClarification of Contribution: We will revise the \"Comparison to Baselines\" and \"Discussion\" sections to clarify that our claim is not that any hierarchical graph would universally improve any VLN method, but rather that our specific design of the dynamic hierarchical semantic scene graph, coupled with our optimal transport planner and graph-aware RL, is what leads to the observed performance gains.\n\nFuture Work / Conceptual Discussion: Directly integrating our complex graph structure into existing baseline architectures (e.g., NTS or SMN) would require significant architectural modifications and re-training, which is beyond the scope of a rebuttal. However, we can conceptually discuss why our graph structure is particularly beneficial for our optimal transport and RL components, and how it might (or might not) directly translate to benefits in different architectural paradigms. For instance, NTS and SMN might not inherently leverage the multi-level semantic reasoning or the explicit topological structure in the same way our optimal transport planner does. We will emphasize that the synergy between our graph representation and our planning/control mechanisms is key.\n\n4. Lack of Visualizations of Qualitative Results and Failure Cases:\n\nWe acknowledge this critical weakness. Visualizing qualitative results and discussing failure cases are essential for providing deeper insights into the model's behavior and limitations.\n\nQualitative Examples: We will add a new section (or expand an existing one) in the paper, likely in the appendix for space, to include:\n\nVisualizations of successful navigation trajectories with corresponding language instructions.\n\nVisualizations of the constructed hierarchical semantic scene graphs for selected examples, showing object, region, and zone detections.\n\nQualitative Failure Analysis: We will include examples of failure cases, illustrating why the system failed (e.g., VLM misinterpretation, getting stuck in clutter, unexpected environmental features). \n\n5. Insufficient Support for Claims in Discussion Section:\n\nWe agree that claims made in the Discussion section, such as \"The hierarchical semantic scene graph enables nuanced scene understanding,\" need to be more directly supported by experimental evidence beyond just overall model performance.\n\nConnecting Ablations to Claims: We will strengthen the connection between our ablation studies and these claims. For example, the ablation experiments that show a drop in performance when hierarchical levels are removed from the scene graph directly support the claim that the HSSG enables nuanced scene understanding and contributes to performance. We will explicitly reference these ablation results when making such claims in the Discussion section.\n\nElaborating on Mechanism: We will also elaborate on how each module contributes to the overall system's capabilities, beyond just \"it performs well.\" For instance, explaining how optimal transport's ability to balance semantic and spatial costs directly leads to more instruction-aligned long-horizon paths."}]

Author rebuttal:

Meta-review: The submission presents a novel multi-stage framework with thorough simulation experiments and ablation studies, demonstrating clear methodological innovation and reasonable performance on one benchmark dataset. Two reviewers highlight the paper’s theoretical grounding, experimental rigor, and SOTA results, providing highly positive and detailed assessments. While other three reviewers raised concerns regarding clarity, notation, and real-world deployment, one was unresponsive in the rebuttal, and the remaining issues are addressable in revision. On balance, the paper makes a significant contribution to the field, and the AC recommends acceptance, giving higher weight to the evaluations of the senior, engaged reviewers. This decision is endorsed by the Senior AC. 

Please incorporate all additional clarification and experiments in the camera ready version.
Decision: Accept (poster)