Title: Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Hierarchical Semantic-Augmented Navigation
Abstract: Vision-Language Navigation in Continuous Environments (VLN-CE) poses a formidable challenge for autonomous agents, requiring seamless integration of natural language instructions and visual observations to navigate complex 3D indoor spaces. Existing approaches often falter in long-horizon tasks due to limited scene understanding, inefficient planning, and lack of robust decision-making frameworks. We introduce the \textbf{Hierarchical Semantic-Augmented Navigation (HSAN)} framework, a groundbreaking approach that redefines VLN-CE through three synergistic innovations. First, HSAN constructs a dynamic hierarchical semantic scene graph, leveraging vision-language models to capture multi-level environmental representations—from objects to regions to zones—enabling nuanced spatial reasoning. Second, it employs an optimal transport-based topological planner, grounded in Kantorovich's duality, to select long-term goals by balancing semantic relevance and spatial accessibility with theoretical guarantees of optimality. Third, a graph-aware reinforcement learning policy ensures precise low-level control, navigating subgoals while robustly avoiding obstacles. By integrating spectral graph theory, optimal transport, and advanced multi-modal learning, HSAN addresses the shortcomings of static maps and heuristic planners prevalent in prior work. Extensive experiments on multiple challenging VLN-CE datasets demonstrate that HSAN achieves state-of-the-art performance, with significant improvements in navigation success and generalization to unseen environments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents HSAN (Hierarchical Semantic-Augmented Navigation), a novel framework for Vision-Language Navigation in Continuous Environments (VLN-CE). HSAN incorporates three synergistic modules: (1) a dynamic hierarchical semantic scene graph capturing object–region–zone structures using vision-language models, (2) an optimal transport-based topological planner grounded in Kantorovich’s duality to select long-horizon goals, and (3) a graph-aware reinforcement learning controller for low-level action execution. The method is evaluated on two standard VLN-CE benchmarks (R2R-CE, RxR-CE) and achieves state-of-the-art performance with solid ablations and generalization analysis.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper proposes a clearly structured three-stage framework (graph construction, planning, control) that directly addresses key challenges in VLN-CE.\n2. The use of optimal transport for long-term goal selection is interesting and supported by a rigorous formulation.\n3. The method achieves strong performance on both R2R-CE and RxR-CE, outperforming a range of competitive baselines across multiple metrics.\n\nWeaknesses\n1. The notation system is overloaded and lacks clarity. The paper introduces many symbols without a consolidated reference. A notation table would improve readability and help readers follow the methodology.\n2. The model includes numerous hyperparameters (e.g., σ, λ in L199; α, β in L232), yet no analysis is provided regarding their sensitivity or selection rationale. This raises concerns about robustness and reproducibility.\n3. In the Comparison to Baselines section, the improvements over methods like NTS and SMN are credited to HSAN’s dynamic hierarchical graph. This raises a natural question: how would NTS or SMN perform if they used the same graph structure? The same concern applies to other modules. It is not clear whether the performance gain really comes from these modules, since the paper does not test them in other methods or provide direct evidence.**\n4. The paper lacks visualizations of qualitative results, and also does not discuss any failure cases\n5. In the Discussion section, the paper states that “The hierarchical semantic scene graph enables nuanced scene understanding…” and makes similar claims about other modules. However, I don’t see how this conclusion is supported by the experiments. The results only show that the full model performs well, but there is no direct evidence isolating the contribution of this specific component.\n\nQUESTIONS:\nThe main concerns are listed in the Weaknesses section. If the authors can provide more detailed experimental analysis to support the effectiveness of each proposed component, it would address my main confusion and could improve my overall evaluation.\n\nLIMITATIONS:\nNo. The authors should include an analysis of failure cases, a discussion of computational cost and efficiency, and whether the method can support real-world robotic execution.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new method HSAN for vision-language navigation task. The main components include dynamic hierarchical scene graph, topological planner and RL-based low level control.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is well-written.\n- Sufficient experiments are conducted with multiple datasets.\n- Ablation studies are discussed.\n\nWeakness:\n\n- In implementation details, it mentioned the system runs with around 5 FPS. I also expect there are some comparison with other methods for the inference efficiency. I see the system contains several complex steps including the detection, scene graph, etc. I'm worried if the too complex system will be the challenge in the real life deployment.\n- The failure analysis is not presented. Especially in the qualitative examples. I'm curious that in which situation the system will fail. And what will be the bottleneck of this system? Are these bottlenecks easy to be fixed in the future works or it's the fundamental issue of this design?\n- Currently all the experiments are based on the simulation instead of real world. I understand the real world experiments are not feasible for everyone. But there is also no discussion about the perturbation applied to the simulation, to make the scenario closer to the real world.\n- One majority concern is, there is an assumption that the perception output is perfect. What if the perception output is noisy, and is there any countermeasure applied to the method to make it work robustly?\n\nMinor Weakness:\n\n- Typo page 4, line 202: “a aggregated” → “an aggregated”\n- Typo page 5, line 264: “accion” → “action”\n\nOverall, I’m borderline for this work. Especially some concerns I mentioned above, I'm fine to adjust my ratings if my concerns get solved.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThere is no limitation discussed in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper constructs two types of graphs to address the VLN-CE task. For long-distance navigation point planning, a topological link graph is built, and the optimal navigation trajectory is selected based on OT theory. For low level action control, a hierarchically constructed sense graph is used to train the policy based on the current observation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- The paper's approach to constructing scene graphs is somewhat insightful and provides inspiration.\n\n**Weaknesses**:\n- There are significant issues in the writing quality:\n  1. The main text contains large sections of redundant content, for example, the overlapping material between lines 67-78 and lines 84-92.\n  2. Several essential details are missing in the method description, such as how the formulas on line 241 are applied to VLN. This lack of detail makes it difficult to assess the validity of the proposed methods and their correctness.\n  3. The overall description of the methodology is lacking, with unclear explanations of the inputs and outputs of each module and how they interconnect.\n  4. The purpose of designing the Hierarchical Semantic Scene Graph Construction seems to be solely for low-level control, which appears to be of limited significance. The motivation needs to be more thoroughly explained, especially concerning the relationships between the two types of sense graphs constructed in the paper.\n\nQUESTIONS:\n1. Regarding the Hierarchical Semantic Scene Graph, is semantic aggregation at different levels really necessary? It seems that for low-level control, simply avoiding obstacles and reaching a predefined ghost node is sufficient. How does the region information help with navigation? However, ablation studies appear to demonstrate its necessity. More discussion is helpful.\n\n2. In this paper, it seems that navigation decisions are mainly based on the similarity of individual nodes and the instruction, which makes me skeptical about the final navigation results of the model. From my understanding, both R2R and RxR datasets contain many procedural instructions describing navigation paths, making it difficult to match the endpoint based solely on instruction similarity.\n\n3. How is map information incorporated into the navigation model? If the method uses the global adjacency relationships between nodes, does it align with the unseen setting used in previous methods?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose HSAN, a method for VLN-CE which constructs a hierarchical scene graph and optimal transport-based topological planner to select navigation goals aligned with language instructions. In addition, the authors propose a novel approach for low level control to avoid obstacles by augmenting PPO with a graph based understanding of the environment. HSAN achieves state-of-the-art performance across all simulation benchmarks for VLN-CE.\n\nSTRENGTHS AND WEAKNESSES:\nThe experimental evaluation of the method in simulation was very impressive with thorough ablations and baselines on every significant benchmark for the task. I was particularly impressed by the results on the multilingual VLN benchmark, a challenge which is often overlooked in evaluating approaches to VLN and aided by this method design. Also, the experimental results are state of the art across benchmarks, and the connection between optimal transport and spatial map reasoning is interesting and novel.\n\nThere seem to be significant barriers to running this method in the real world on a robot which are substantially downplayed by the authors. The authors in fact emphasize how their approach is well suited to running on a physical robot and do not acknowledge the VLN methods which already do so and discuss this method’s unique limitations in this regard other than too slow inference speed of the approach. In addition to inference speed, the required task specific training of the VLM and GCN will have significant challenges performing robustly in open world physical environments. There will be sim-to-real gaps with the trained models even in similar real world environments, and the type of environments in the simulators used for training is far from the scope of different environment types in the physical world (for example, no offices are included). In future work, the authors claim intentions to show results in dynamic and outdoor environments, but no results are currently shown in indoor environments. Real world results for this method are particularly important because part of the claimed contribution is low level physical control of a robot.\n\nSome missing citations for papers with similar map and/or VLM based solutions showing real world instruction following results.\n\n1. Sonia Raychaudhuri, Duy Ta, Katrina Ashton, Angel X Chang, Jiuguang Wang, Bernadette Bucher. Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigation. arXiv, 2024.\n2. Zhaohuan Zhan, Lisha Yu, Sijie Yu, Guang Tan. MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chains\n3. Yuxing Long, Xiaoqi Li, Wenzhe Cai, Hao Dong. Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions. ICRA, 2024.\n\n\nThere are multiple issues with phrasing. The authors appear to be misusing the word “benchmark”. They repeatedly state that HSAN is a “new benchmark.” However, it appears what they mean is that HSAN is the “new state of the art baseline” for the existing benchmarks addressing this problem on vision-language navigation in continuous environments. In addition, there are a few incorrect phrases like HSAN “redefines VLN-CE”. I also don’t think that the scene graph is “dynamic” since it does not update with movement in the scene. The authors should be careful with their phrasing. \n\nThe code was not included in the supplement, and there is no condition ensuring that it will be included in an accepted paper, limiting reproducibility of this work.\n\nI have concerns about the significance of the low level control method discussed in questions.\n\nQUESTIONS:\nHow does the benefit of the new proposed low level control policy compare to classical planners for obstacle avoidance which are state of the art in robotics? \n\nFor example, PPO was initially introduced as a method for point to point navigation when the goal location existed outside the observed map region or no map was constructed. PPO implicitly predicted occupied and unoccupied regions in order to enable obstacle avoidance in these scenarios. To my understanding, your navigation waypoints are inside an observed map region for which you have an explicit map. Your citation to [1] which you use to motivate your decision to modify PPO explicitly refers to the case in which an explicit map is not constructed.\n\n1. Jacob Krantz and Stefan Lee. Navigating to objects in the real world. arXiv preprint 537 arXiv:2112.06758, 2021. doi: 10.48550/arXiv.2112.06758. \n\nSo, in summary, why would you use your proposed control method over approaches like fast marching squares, A*, etc. which are currently deployed with high performance on physical robots to do point to point navigation in regions for which you built an explicit map?\n\nI would be open to increasing my score if my concerns about the low level planning method are resolved and the writing is improved both to address the real world evaluation limitations and the inaccuracies in phrasing.\n\nLIMITATIONS:\nThe limitations regarding real world hardware deployment are not discussed and misrepresented as not a challenge. I discussed further in the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes HSAN, a framework for VLM-based navigation that utilises structured information from the environment in the form of a scene graph representation. This representation is used both for planning and control based on optimal transport and RL, respectively. The work is benchmarked on the R2R-CE dataset against baselines of various natures.\n\nSTRENGTHS AND WEAKNESSES:\nI will use the symbol (+) to represent strengths and (-) to represent weaknesses.\n\n# Quality\n(+) The work describes a full navigation system based on SOTA techniques and obtaining best results in a meaningful benchmark against various strong baselines.\n\n(+) The amount of work discussed and carried on by the author is important, and the authors have been thorough in testing and validating different versions and ablations of their work.\n\n# Clarity\n(+) The paper is well-written and well-structured.\n\n(+) Regardless of the complexity of the proposed method, the authors managed to describe every component well and in detail, without confusing the reader.\n\n# Significance\n(+) Robot navigation for long horizons in unstructured environments is a critical capability for autonomy. \n\n(+) This work addresses this task using SOTA VLM techniques, highlighting their strengths and weaknesses, and proposes structured components to mitigate them.\n\n(-) The experimental results are, unfortunately, limited to a specific dataset and do no distinguish strongly between seen and unseen scenes _and_ objects.\n\n# Originality\n(-) The work is missing some references to literature on scene graph generation. In particular, the works `Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization` and `ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning` seem very relevant for the task, while works like `LeLaN: Learning A Language-Conditioned Navigation Policy from In-the-Wild Videos` and `Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrieval` are relevant for the VLM-based navigation.\n\nQUESTIONS:\nHere are some questions for the authors to comment on. I have ordered them in priority order from my perspective.\n\nQ1 - Scene-Graph Generation: The authors propose a new scene graph generation methodology, which is based on three layers of concepts. My question is about why not using more established techniques like Hydra and Conceptgraphs, and how that would affect the overall methodology. Moreover, I notice that the location aggregation feature is defined as an average of the features of the objects at each location. How strong of an assumption is this? Could this be learn together with the framework as a graph aggregation, for instance?\n\nQ2 - data availability: the work starts from the assumption that the robot has at each point in time a panoramic view and GT positioning in the environment. Is this a valid assumption, and how would the absence of such signals affect the overall performance? \n\nQ3 - seen and unseen: the work is based on a training and fine-tuning operation on the dataset used for the main testing results. While the scenes used for training and testing are separate, it would be interesting to understand how unseen objects and queries would affect the generalizability of the approach. One example is in `select2plan`, where a table is constructed. Would it be possible to replicate a similar experiment?\n\nQ4 [minor] - effect of lambda: while the authors describe the effects of the different hyperparameters in the appendix, I do not think they touch upon the parameter lambda of equation 1 and how it affects the scene graph generation.\n\nQ5 [minor] - language in the queries: in section 4, the authors assess the multilingual capabilities of the system. Are there any insights that can be provided to the reader? Is the model working better in specific environments and language combinations?\n\nLIMITATIONS:\nThe authors mention the computational overhead as the main limitation. I think the authors should expand on the limitations of their work. Is the complexity of the method a limitation? Is the need for training data or the testing-training difference a limitation?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}