Title: NavBench: Probing Multimodal Large Language Models for Embodied Navigation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Embodied Navigation

Multimodal Large Language Models
Abstract: Multimodal Large Language Models (MLLMs) have demonstrated strong generalization in vision-language tasks, yet their ability to understand and act within embodied environments remains underexplored. We present NavBench, a benchmark to evaluate the embodied navigation capabilities of MLLMs under zero-shot settings. NavBench consists of two components: (1) navigation comprehension, assessed through three cognitively grounded tasks including global instruction alignment, temporal progress estimation, and local observation-action reasoning, covering 3,200 question-answer pairs; and (2) step-by-step execution in 432 episodes across 72 indoor scenes, stratified by spatial, cognitive, and execution complexity. To support real-world deployment, we introduce a pipeline that converts MLLMs' outputs into robotic actions. We evaluate both proprietary and open-source models, finding that GPT-4o performs well across tasks, while lighter open-source models succeed in simpler cases. Results also show that models with higher comprehension scores tend to achieve better execution performance. Providing map-based context improves decision accuracy, especially in medium-difficulty scenarios. However, most models struggle with temporal understanding, particularly in estimating progress during navigation, which may pose a key challenge.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a benchmark with Multimodal Large Language Models for Embodied Navigation. Two components are included in this work, navigation comprehension and execution. This work also supports that converts MLLMs' outputs into robotic actions.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n    - This paper is will written, and easy to follow.\n    - This paper proposes a navigation benchmark specifically for MLLMs, focusing on exploring their long-term processing capabilities in 3D scene.\n    - This paper implements a systematic comparison and analysis of the performance trends of closed-source and open-source MLLMs in embodied reasoning and decision-making.\n \n- Weakness\n    - This paper proposes a new benchmark, which is mainly constructed based on existing datasets such as R2R and RxR, and the evaluation metrics lack novelty. I recommend that the authors further consider designing new input-output formats and evaluation criteria to comprehensively assess the spatial reasoning abilities of multimodal large models.\n    - This paper proposes a pipeline for real-world deployment, but the current description is rather general. I suggest the authors provide additional details regarding the generation process of the waypoint heatmap, the algorithm used to extract predicted waypoints, and the specific methods for calculating the angles and distances between different waypoints. Including algorithmic descriptions and visualizations of these steps would significantly improve the clarity and reproducibility of the pipeline.\n    - This paper evaluates proprietary MLLMs (e.g., GPT-4o) and open-source MLLMs (e.g., InternVL2.5-2B). I have the following questions and suggestions: (1) Have the authors considered including more open-source models (e.g., Gemini series) and proprietary models (e.g., LLaVA-Next) to enrich the comparison? (2) In Table 1, the open-source models cover a wide range of parameter sizes (e.g., 2B, 3B, 7B, 11B). For fair comparison, it is recommended to match model sizes as much as possible; (3) It would be beneficial to see more ablation studies to provide a finer-grained analysis of the current MLLMs’ performance and limitations in zero-shot navigation tasks within 3D environments.\n\nQUESTIONS:\n- The main concerns are listed in the Weaknesses section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents NavBench, a benchmark designed to evaluate the embodied navigation capabilities of Multimodal Large Language Models (MLLMs) under zero-shot settings. \n\nThe benchmark consists of two components: (1) navigation comprehension tasks evaluating global instruction alignment, temporal progress estimation, and local observation-action reasoning; (2) step-by-step execution tasks across different difficulty levels. \n\nThe authors evaluate both proprietary and open-source MLLMs on NavBench, highlighting persistent challenges in temporal reasoning.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The paper addresses a clear gap in evaluating MLLMs for embodied navigation, moving beyond static vision-language tasks to dynamic, sequential decision-making scenarios.\n\n2. The three-level comprehension assessment (global, progress, local) is cognitively grounded and provides diagnostic insights.\n\n3. The systematic categorization of tasks by spatial, cognitive, and execution complexity with both automatic scoring and human validation is methodologically sound.\n\n4. The paper evaluates multiple models across different scales and provides a detailed analysis of failure modes, revealing important insights about temporal reasoning limitations.\n\n## Weaknesses\n1. For Temporal Progress QA pairs construction, how the authors do the filtering (\"using a curated list of valid instruction-path pairs\") is not clear.\n\n2. The weights α, β, γ for difficulty scoring are \"empirically set\" but the justification is unclear.\n\nQUESTIONS:\n1. Could the authors provide more details about the quality control methods of constructed Temporal Progress Estimation data?\n\n2. Regarding the complexity score weights (α,β,γ), the paper states they are \"empirically set.\"  Was there a specific calibration step against the human ratings, or were they chosen based on prior work and intuition?\n\nLIMITATIONS:\nThe paper would benefit from a more detailed discussion of potential biases in the constructed QA pairs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an evaluation benchmark for multimodal language models for vision-and-language embodied navigation tasks, specifically in zero-shot settings. Their methodology proposes evaluating both the comprehension as well as execution in the navigation task. They also develop a pipeline for real-world deployment of these agents in environments. They show how this analysis can reveal limitations in current models in temporal understanding and other factors, that previous metrics obfuscate; and also show how current zero-shot models can be thoroughly tested for such environments.\n\nSTRENGTHS AND WEAKNESSES:\n1. The evaluation proposed by this paper is important to the field of vision/language navigation and mulltimodal models.\n2. It draws from relevant existing work to use metrics that are well-tested and thought out (e.g., not just goal accuracy and trajectory metrics, but weighted averages of them and so forth) and combines them into one benchmark,\n3. They show how this benchmark reveals weaknesses of models that previous benchmarks and metrics could not easily delineate.\n4. It is well written and explained and all metrics are easily reproducible and will be useful for future work.\n5. They illustrate how models they evaluate can be deployed in the real world, specifically on a dual-arm robot.\n6. They evaluate a sufficient range of models (including the state-of-the-art LLMS (although they might not be the best VLN models).\n\nQUESTIONS:\n1. There are now a few vision and language datasets that come in multiple languages (e.g, english -> trajectories, hindi -> trajectories). It would be worth discussing whether this evaluation, especially for the comprehension component, needs to factor a language id in, and if there would be any differences expected. If not (which, OTTH, there should not be currently) it would be worth pointing out that a strength of this methodology is that it is language agnostic, and also, future work on the multilingual datasets might be interesting insights from such metrics.\n2. Can we get an upperbound on model performance for VLN models? The numbers in the table are fairly low, which is expected for zero-shot LLMs, but the best-performing VLN models (that are trained specifically for this task) would be good to see alongside, to calibrate the difference.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces NavBench to evaluate the multimodal large language models' capabilities, including navigation comprehension and navigation execution capabilities in VLN. Experiments and analyses with 2 closed models and 4 open-sourced models on the NavBench benchmark show some interesting findings.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths:\n\t- The paper is well-written and easy to follow.\n\t- The paper introduces three navigation comprehension tasks and one navigation execution task to evaluate the commonly used VLMs' capabilities. The introduced tasks are carefully designed and expected to be quite useful to evaluate thee VLMs' capabilities in the context of navigation.\n\t- The tested models are quite diverse, including both proprietary and open-source models.\n\t- The proposed difficulty level for navigation execution tasks is novel and useful.\n- Weaknesses:\n\t- The paper is a benchmark paper but not submitted to the benchmark track. So we cannot really check the content and quality of the benchmark. Moreover, the paper does not discuss the license regarding the benchmark release. Considering that Matterport3D is not fully open-sourced and has its own license, so it is important to discuss the license if the authors plan to release the benchmark.\n\t- The paper uses panoramic images as observation format for all the tasks in the benchmark. This is no longer the common setting for the VLN-CE task, which is a more popular task nowadays. The VLN-CE task usually adopts egocentric videos as observation input, so the findings of this paper may have limit to extend to it.\n\t- The navigation execution task in the benchmark, where a model is required to pick up the next-step viewpoint, is almost the same as the common setting of VLN task. So there is a concern why not directly use the standard VLN task?\n\nQUESTIONS:\n- The length of trajectories matters for temporal progress estimation. So is it possible to do an analysis about the relationship between errors and trajectory length?\n- The paper does not mention whether chain-of-thought thinking is used for models' predictions. Considering there are some discussions about whether thinking is useful for navigation tasks in recent literature, I suggest to add some comparisons between using thinking and without thinking. \n- Considering the reasoning capability of thinking models, is it possible to test performance of some reasoning models such as o4-mini in this benchmark?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-11"}