NavBench: Probing Multimodal Large Language Models for Embodied Navigation
Authors: Yanyuan Qiao
Haodong Hong
Wenqi Lyu
Dong An
Siqi Zhang
Yutong Xie
Xinyu Wang
Qi Wu
Venue: NeurIPS 2025 poster ()
Keywords: Embodied Navigation
Multimodal Large Language Models
Primary area: applications

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated strong generalization in vision-language tasks, yet their ability to understand and act within embodied environments remains underexplored. We present NavBench, a benchmark to evaluate the embodied navigation capabilities of MLLMs under zero-shot settings. NavBench consists of two components: (1) navigation comprehension, assessed through three cognitively grounded tasks including global instruction alignment, temporal progress estimation, and local observation-action reasoning, covering 3,200 question-answer pairs; and (2) step-by-step execution in 432 episodes across 72 indoor scenes, stratified by spatial, cognitive, and execution complexity. To support real-world deployment, we introduce a pipeline that converts MLLMs' outputs into robotic actions. We evaluate both proprietary and open-source models, finding that GPT-4o performs well across tasks, while lighter open-source models succeed in simpler cases. Results also show that models with higher comprehension scores tend to achieve better execution performance. Providing map-based context improves decision accuracy, especially in medium-difficulty scenarios. However, most models struggle with temporal understanding, particularly in estimating progress during navigation, which may pose a key challenge.

NavBench: Probing Multimodal Large Language Models for Embodied Navigation
Yanyuan Qiao
Haodong Hong2 23
Wenqi Lyu
Dong An Qi Wu
Siqi Zhang
Yutong Xie Xinyu Wang'
Swiss Federal Institute of Technology Lausanne (EPFL)
The University of Queensland $CIRO Data61 "The University of Adelaide Mohamed bin Zayed University of Artificial Intelligence Tongji University
Project Website
Abstract
Multimodal Large Language Models (MLLMs) have demonstrated strong gener- alization in vision-language tasks, yet their ability understand and act within embodied environments remains underexplored. We present NavBench, a bench- mark to evaluate the embodied navigation capabilities of MLLMs under zero-shot settings. NayBench consists of two components: (1) navigation comprehension, assessed through three cognitively grounded tasks including global instruction alignment, temporal progress estimation, and local observation-action reason- ing, covering 3,200 question-answer pairs; and (2) step-by-step execution in 432 episodes across 72 indoor scenes, stratified by spatial, cognitive, and execution complexity. To support real-world deployment, verts MILMs' outputs into robotic actions. We evaluate both proprietary and open-source models, finding that GPT-4o performs well across tasks, while lighter open-source models succeed in simpler cases. Results also show that models with higher comprehension scores tend to achieve better execution performance. Provid- ing map-based context improves decision accuracy, especially in medium-difficut scenarios. However, most models struggle with temporal understanding, particu- larly in estimating progress during navigation, which may pose key challenge.
introduce a pipeline that con-
Introduction
Multimodal Large Language Models (MLLMs) 1 have achieved impressive performance across wide range of vision-language tasks, demonstrating strong cross- moosl reasoning and zero-shot generalization. These models excel at answering visual questions 4), interpreting videos [5], and performing complex multimodal reasoning [6] As their capabilities expand, central question emerges: do these models truly understand how to act in the physical world, or are they simply adept processing static inputs?
Recent work has begun to explore MLLMs' potential in embodied tasks by evaluating their spatial reasoning in 3D environments 87 However, these tasks primarily focus perception and passive scene understanding, without assessing the model's ability to make decisions or take actions. In comparison, navigation is core embodied task that involves interpreting natural language instructions, analyzing visual observations, and making a sequence of decisions to reach a goal. Although navigation plays crucial role in real-world applications, it remains relatively underexplored in the context of MLLMs. Traditional embodied navigation benchmarks, such as Room-to Ro-Room (R2R) 9
*Correppondnng author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Comprehension Progress Level Temporal Progress Estimation
Comprchension Local Level
Local Diservatiin-Actinn Reasoning
Given the steps taken so far, how far along in the instruction?
move in direction (c), what next?
Execution
Comprehension Global Level
Step-by-step Navigation
Global Instruction Alignment
on instruction, steps
should I take to reach the goat?
Having completed the navigation, which instraction best describes the path followed!?
Difficulty Level
Nidax
Bied
Oogaltive Conplesity
Spatial Complesity
Exeration Complecity
Figure 1: NavBench evaluates MLLMs across three comprehension tasks and a step-by-step execution task, assessing their ability to understand navigation behavior, track progress, reason about observation and action, and act accordingly. The step-by-step navigation is assessed from different difficulty levels, which is defined by cognitive, spatial, and execution complexity.
and ObjectNav 100,, were developed prior to the emergence of foundation models. These benchmarks rely on sask-specific supervision and often reduce evaluation to final success rates, providing limited insight into whether model genuinely understands the navigation behavior. In many cases, agent may reach the goal by exploiting dataset biases or learning shortcuts, without correctly grounding the instruction following the intended path.
Similar how humans acquire embodied skills by first understanding a task and then learning to execute it, evaluating the embodied capabilities of generalist MLLMs also requires examining two fundamental aspects. First, can the model comprehend what navigation behavior represents, such as identifying the intent behind completed trajectory? Second, can it act autonomously to complete a navigation task, making step-by-step decisions in unfamiliar environments? Furthermore, navigation tasks in real-world environments can vary significantly in difficulty due to differences in spatial layout, instruction complexity, and required decision maaisinn-mmking steps. For example, navigating across multiple rooms with ambiguous instructions poses greater challenges than following simple step-by-step commands in a single hallway. However, most existing benchmarks treat all navigation episodes equally difficult, failing to capture this essential variation.
To fill these gaps, we introduce NayBench, a benchmark designed to systematically evaluate MLLMs in embodied navigation under zero-shot settings. NavBench decomposes the evaluation into two complementary components: Navigation Comprehension, which assesses whether a model under- stands and aligns with intended navigation behavior, and Navigation Execution, which evaluates the model's s ability to make accurate step-by-step decisions. To reflect real-world variability, NavBench incorporates fine- fine-grained difficulty classification based on spatial, cognitive, and execution com- plexity. In addition, it provides a deployable real-world navigation pipeline to bridge the gap between simulation and practical embodiment.
First, for navigation behavior comprehension, inspired by cognitive studies of human spatial rea- soning (1I, NavBench introduces three fine-grained evaluation tasks designed to assess distinct reasoning capabilities at three levels: global, progress, and local. It includes 3,200 question-answer pairs. Specifically, Global Instruction Alignmen! evaluates the model's ability to match given trajectory with the most appropriate instruction. The candidate instructions are designed with subtle semantic differences, such variations in directional cues and landmark descriptions, to encourage genuine spatial reasoning. Temporal Progress Estimation measures requiring the model to infer progress within multi-step instructions based partial trajectory. Local Observation-Acee Actionn Inference evaluates the model's ability to reason about the spatial consequences of individual actions by either predicting the future observation given an action or identifying the action that caused a visual transition. Together, these tasks provide a comprehensive framework for assessing global semantic reasoning, temporal understanding, and local spatial inference in navigation.
awareness by
Second, NayBench introduces fine-grained difficulty classification with three levels: easy, medium, and hard, based on cognitive, spatial, and execution complexity. This allows detailed analysis of

---

models' generalization and cociisonnmaking performance across varying levels of difficulty, The benchmark includes 432 navigation cases across 72 scenes.
Finally, to bridge the gap between simulator-based evaluation and world deployment, we design practical navigation pipeline that connects MI.LM outputs to executable actions real robots. This pipeline includes a waypoint selection module, an MLLM-based navigator, and a low-level controller, demonstrating the deployability of our framework in physical environments.
We evaluate both closed-source and open-source MLLMs on NavBench While GPT-4o currently achieves the best overall performance, we observe that lightweight models such (wen2.5-VL-77B are capable of reliably completing easy navigation tasks. Notably, this trend is also reflected in our real-world deployment experiments, suggesting that NavBench may serve practical tool for analyzing the embodied capabilities of both general and resource-efficien MLLMs. Furthermore, our results suggest several notable trends: (1) comprehension and execution abilities appear to be closely related, (2) temporal reasoning may pose persistent challenge for current models, and (3) compact open-source models can, under certain conditions, approach the performance of proprietary ones, indicating their potential utility in practical settings.
In summary, our main contributions are as follows: (1) We introduce NavBench, a benchmark for evaluating MLLMs in embodied navigation under zero-shot settings. (2) We decompose the evaluation into two components: Navigation Comprehension, with tasks targeting spatial, temporal, and local reasoning, and Navigation Execution, which assesses decision decision-making making across difficulty levels. (3) We develop a deployment pipeline that maps MILLM outputs to real-world robot actions. (4) We perform a detailed evaluation and analysis of both closed-source and open-source MILMs, uncovering trends in their reasoning and execution performance across embodied tasks.
2 Related Work
Benchmarks for MLLMs Recent progress in Multimodal Large Language Models (MLLMs) 12
14 has driven the development of benchmarks assessing visual understanding and cross-modal reasoning. Early efforts such as VQA GQA [16,, OK-VQA [17], and TextVQA [8]] focus on specific tasks like factual or commonsense question answering. More recent benchmarks including MME [9], MMBencc [20]. MM- Vet (2I). and MathVista [6] aim for broader coverage, evaluating perception and reasoning across diverse domains. However, these mainly target static tasks and do not reflect MLLMs' ability to act in dynamic environments. To bridge this gap. some recent work has begun evaluating spatial reasoning in embodied settings. SpatialBench [7]. ScanReason [22]. and VSI-Bench [8 assess 3D spatial understanding using panoramas, semantic layouts, or textual scene descriptions. While insightful for embodied perception, they remain limited passive tasks and do not assess decision-making or sequential interaction. In parallel, traditional embodied navigation benchmarks such as R2R(9) REVERIE 23), and ObjectNav [10] have long been used to test Instruction-followii and mainly evaluate success rates without probing intermediate reasoning. Although REVERIE increases instruction abstraction, it retains similar path lengths and decision complexity, limiting its capacity to reveal behavioral differences. More recendl, Wang et al. (24) proposed a fine- grained evaluation framework for instruction understanding in VLN via multiple-choice questions, offering interpretability beyond end-to-end metrics. Still, their setup is restricted to small supervised models and lacks real-world deployment and zero-shot inference. To the best of our knowledge, no existing benchmark offers a comprehensive evaluation of MLLMs in embodied navigation that jointly considers instruction understanding, sequential decision-making, difficulty stratification, and real-world transferability.
agents. However, they were designed for fully supervised settings
Embodied Navigation Embodied navigation tasks require an agent to reach a goal location within
an environment, guided by a description such as an image [25] 26), object 110 27). or natural language instruction [9] [28 (29] Among these, language-guid
navigation has attracted significant attention for its potential facilitate intuitive human-robot interaction. Researchers have explored diverse instruction formats, including step-by-step 9 30). dialog-based, [31], and goal- or intention- oriented instructions
23 32.. Traditional approaches train navigation policies using annotated
datasets, incorporating modules to improve object relation understanding [33 34, 33). vision-langure
alignment [36]] 37 memory (39 40
and spatial reasoning [4 43 44 While effective on
benchmarks, these methods often suffer from limited generalization due to dataset biases (5 46 47
To mitigate this, recent work turns to MILMSs for zero-shot embodied navigation, leveraging their generalization abilities. Some use MLLMs to localize goal-relevant regions [48 49 50), while others
3

---

Global Level
Progress Level
Local Level
Global Instruction Alignment
Temporal Progress Estimation
Local ObservationAAtiion Reasoning
You coerled thth secsuence of paroo vieno repensent a invigation path from the starting point the goall ocaé on ty cerrect instrutioon
Question:
You are givan nesigation Intruction disieed Into muriiple ontertttucioons along wan cepectory
Quention:
Question: Given tre siez and tar viess the diediistionn thatls mistlbbdyt to load the larget
Your
vast all way forward towards the door
is wick the stee's a and into the first doo way on you!
right ins the doomwwy
Lobric che.
the betinours turn right,
ooug?
Sue tracruction on 2: walt down the stop and into the yard
thi doonway the right and stop front of ofene bed
selectite ocation matching thenspacted ofter
fult the
dereight unitl
the
Subsinsrruttion Ri anthan the supvinstraction on stop, treeeeooeeeeeeecon
Turn left eci the room. left and go into the betroom wai srete entronce
Figure 2: Illustration of the Navigation Comprehension task.
These approaches reduce reliance on task-specific training but still lack fine-grained evaluation: most benchmarks focus solely on final success rates, offering limited insight into the model's reasoning process. To address this, we introduce NavBench, a benchmark that systematically evaluates both the reasoning and execution capabilities of MLLMs in embodied navigation.
employ prompt-based guidance for instruction following (1
52 53 54
3
Benchmark Design
3.1 Task Formulation
We evaluate the navigation capabilities of MILMss by decomposing the task into two components: Navigation Comprehension, which assesses the understanding of navigation behavior, and Navigation Execution, which focuses step-by-step decision making.
Navigation Comprehension It investigates whether the model can understand and reason about implicit navigation behaviors, including aligning instructions with trajectories, estimating progress along a plan, and predicting the spatial consequences of actions. These tasks span different reasoning levels (global, progress, and local) and serve as diagnostic probes for navigation understanding. Illustrations of the three comprehension tasks are shown in Figure
Global Level - Global Instruction Alignment: Given a navigation trajectory and several candidate instructions, the model is required to determine which instruction aligns with the executed path. This task tests the model's understanding of the overall intent and structural coherence of the navigation behavior.
Progress Level - Temporal Progress Estimation: Provided with partial trajectory and a list of segmented sub-instructions, the model must identify the sub-instruction that was most recently completed. This evaluates the model's capacity to monitor task progress and comprehend the temporal structure of instructions.
Local Level Local Observation -cction Reasoning: To evaluate the model's 's ability reason about the spatial consequences of individual actions. We design two variants: (1) Future-Observation Prediction - the model observes the current view and an action, and selects the correct resulting view. (2) Future-Action Prediction the model observes two consecutive views and must identify the action that caused the transition.
Navigation Execution It examines whether an MLLM can make accurate, step-by-step movement decisions in an embodied environment based on the current observation and instruction. We conduct this evaluation in a zero-shot setting (54) within the Matterport3D simulator (55). categorizing tasks into three difficulty levels (easy, medium, and hard) to assess performance. To ensure fair and standardized evaluation protocol, we evaluate MLLMs via viewpoint selection rather than low-level action prediction (e.g., turning moving forward). This abstraction. consistent with prior embodied navigation benchmarks g 23 allows us to focus on high-level semantic reasoning grounded in language and vision, while avoiding the confounding variability introduced by continuous control. It also facilitates zero-shot evaluation and comparability across different models. Notably, while our
Thh questions in the figure are slightly simplified for clarity and brevity.

---

Navigation Comprehension
Navigation Execution
Benchmark Statisties
Step A1
Source Raw Datosets
Step
Sample Navigation Epicodes MatterPort3D Simulater
Creeo SoNS PIESTNES DTESE N
RER
e
(Meilkdd bpot
Comp.
Propoondd opc)
1230
Step 52
Compute Difficulty Sores Spatial/Cogeiiive/ee Execution
Extract Multimodal Navigation Data Instruction/ /Tajjctory/ Single Viewa/ Panoramic
DCL
1X SE
Nerggaee Epoudde Sounee
Dificaty Lovcs
Human Difficulty Rating
Aeg
Step Al M
Construct Q&A Samples
Ptobal/Proggrss///eo Local Action Level
Assign Difficulty Level Essy/ Essy/Medium/Herd
Medion
Exec, Tepe Statictics by Dufficutty
(b)
Figure 3: NavBench construction pipeline and statistics. (a) QA generation for comprehension tasks at global, progress, and local levels. (b) Execution pipeline combining automatic difficulty scor- ing and human ratings. (c) Benchmark statistics, including comprehension (comp.) task distribution, QA counts, and execution statistics (e.g., instruction length, steps, distance).
simulator setup centers on abstracted decision-making, Section iillustrates how this framework can be extended to real-world navigation by converting viewpoint selection into low-level control.
Specifically, at each step, the model receives the current panoramic observation, the natural language instruction, and a list of candidate navigable viewpoints. The model must select the next location to move to, thereby executing the instruction step-by-step until the goal is reached. Formally, each step t of navigation episode, the MLLM receives an instruction x = {wi, W2, .... wL) of length L, of candidate navigable observation viewpoints O, {or of .... of J, and optional context C, (such as navigation history or previous actions). The agent must select an action corresponding to one of the navigable directions:
a MLLM(x, O,, C.).
This decision process may involve reasoning about the instruction, interpreting the current view, leveraging prior context, and anticipating the result of each candidate action.
3.2 Dataset Construction
Data Sources NayBenche is constructed by reorganizing and enriching grn-ggraied navigation data with multimodal observations to enable zero-shot evaluation of MLLMs. We start by collecting instruction-trajectooyy pairs from multiple embodied navigation benchmarks, including R2R (9]. RxR (30, GEL-R2R [56), and FGR2R [57]. These datasets serve as annotation sources, but do not include the visual inputs needed for multimodal reasoning. To address this gap, we use the Matterport3D simulator extract both panoramic and single-viewpoint RGB images aligned with navigation trajectories. The image extraction process involves traversing agent paths, sampling intermediate viewpoints, and rendering corresponding visual observations. All visual and textual data are then organized into a unified structure that supports multiple reasoning tasks and enables consistent QA generation across comprehension and execution settings. Figure 3 shows the overall benchmark construction pipeline.
Statisties We report statistics in Figure C)., including distribution of comprehension subtasks and coverage of scenes and episodes in execution. These statistics reflect the scale and diversity of the benchmark across reasoning levels and scenes.
3.2.1 Question-and-Answer Pairs Collection
We design three diagnostic tasks targeting global alignment, temporal progress estimation, and local spatial and action reasoning, In total, collect 3,200 wwwpstiion-nn--nsseeee pairs evaluate comprehension capacity in embodied navigation.
Global Instruction Alignment To evaluate MILMss ability to align spatial trajectories with semanti- cally consistent instructions, we construct a multiple-choice dataset comprising 1,200 examples. Each example consists of panoramic trajectory and five candidate instructions, including ground-truth
5

---

and four distractors. The distractors are generated using four perturbation strategies: (1) Basic: random instructions sampled from unrelated trajectories, testing global relevance: (2) Directional replacements, where spatial terms (e.g., "left", "north") are substituted using POS tagging via NLTK, probing directional grounding: (3) Object replacements, where noun phrases are replaced with unre- lated landmarks drawn from an external
dataset [56], evaluating object-trajectory sub-instructions [57] are permuted to disrupt temporal structure while preserving grammaticality. Each instruction set is randomly ordered and paired with a panoramic trajectory composed of viewpoint sequences and movement annotations. The design promotes multimodal spatial reasoning and reduces reliance on superficial cues.
grounding: (4) Shuffled segments, where
Progress Estimation This task is designed to evaluate model's ability to perform temporal reasoning and monitor execution progress during navigation. Each full navigation instruction is segmented into a sequence of sub-instructions, and each sub-instruction is aligned with a corresponding portion of the agent's is trajectory. We leverage fine-grained annotations [57]. which provide this alignment between individual sub-instructions and the associated panoramic viewpoints traversed during execution. To construct evaluation examples, we truncate the trajectory at intermediate points that mark the end of specific sub-instructions. The model is presented with the truncated panoramic trajectory along with the full list of sub-instructions, and is required to predict the index of the last completed one. To ensure data quality and minimize ambiguity, we applied a combination of automatic filtering and manual validation to retain instruction -ath pairs with well-defined temporal boundaries (details in Appendix). In total, we collect 1,000 such examples for evaluation.
Local Observation-Action Reasoning We design two multiple-choice reasoning tasks to evaluate a model's capacity for local spatial and action reasoning inference. Both tasks present ambiguous scenarios that require fine-grained visual discrimination and understanding of plausible transitions. In Prediction, the model receives a current view and an action, and must choose the correct resulting view from set of candidates. In Future-Action Prediction, the model observes two consecutive views and selects the action that best explains the transition. For both tasks, distractors are carefully sampled from nearby observations or visually similar actions to ensure ambiguity and challenge. We collect 500 examples for each format, yielding a total of 1,000 samples. All questions formatted as multiple-choice queries to ensure consistency across evaluation tasks.
3.2.2 Navigation Episodes Collection
We sample 432 navigation cases from 72 unique scenes in the Matterport3D simulator [55]. To systematically assess the difficulty of each case, we define composite complexity score across three orthogonal dimensions: spatial, cognitive, and execution complexity. Each dimension is derived from structural properties of the environment linguistic cues in the instruction, following the methodology inspired by (58 [5]. In addition, human evaluation is conducted to further support and validate the difficulty classification process.
Spatial Complexity quantifies the geometric and topological challenges of navigation trajectory. We consider four features: (1) total path length d, (2) standard deviation of turn angles 0, (3) vertical range z as a proxy for elevation change, and (4) 2D spatial area A covered by the path. A binary indicator I(2 1.5) is included to capture significant clevation changes such as floor transitions. These features are computed from agent poses and scene connectivity data. The spatial complexity score is defined as:
i spatial a1 log(1 + d) + a2 log((1 + 0) + as I[2 > 1.5) + as log(1 + A).
(2)
Cognitive Complexity reflects the linguistic difficulty of navigation instructions. We extract five features using dependency parsing: (1) instruction length L, (2) number of verbs V. (3) number of spatial terms (e.g., left, upstairs), (4) number of landmark mentions M (e.g., kitchen), and (5) number of subordinate clauses C' (e.g., relcl, advel). The cognitive complexity score is defined as:
cognitive = 31 log(1 + L.) + B2 log(1 + V) + 33 log(1 (1 + 5) + B4 log(1 + M) + B5 C. (3)
Execution Complexity It measures the behavioral effort required to complete the navigation. We consider: (1) number of steps N, (2) number of turns T, (3) floor change indicator F, and (4) number of decision points D. The score is computed as:
L exccution 71 logg1 + N) + 72 log(1 7) + 73 F Y4 D.
(4)

---

Normalization Each raw complexity score d is normalized to the range [1, 9] using non-linear mapping:
log(1 b) - log(1 + $ mmin)
i round [1+ 8-
2
(5)
log(1 + 1max) log(1
4 min)
The weights a, 3, and y are empirically to balance the contribution each factor.
Human Evaluation To complement the automatic scoring, we conducted a human evaluation to validate difficulty annotations. A group of annotators independently rated each along the three defined dimensions, using a 1-9 scale with detailed guidelines aligned to our scoring criteria. Further details are provided in the Appendix.
Difficulty Categorization Based on the final scores, each case is categorized into one of three levels, illustrated in Figure
Spatial
Hexd
Easy (score 1-3): Short paths with simple instruc- tions, few steps, minimal spatial reasoning, and clear landmarks.
Medium (score 4-6): Instructions with moderate length, multiple landmarks or spatial phrases, and medium-length paths.
Execution
Cognitive
Hard (score 7-9): Long trajectories guided by com-- plex step instructions, often involving floor transitions and multiple spatial references.
Figure 4: Radar chart of average complexity scores across cognitive, spatial, and execution dimensions for different difficulty levels.
Real- World Deployment Pipeline
Waypoint Predictor
MILM Decision
Low-Level Controller
Narigstion
Reletive Rehation & Dirtonce
Robet Movement
MLLM
APE
Figure 5: Overview of the real-world embodied navigation pipeline.
To demonstrate the real-world feasibility of MLLM-guided embodied navigation, we implement modular pipeline that complements our benchmark evaluation, as illustrated Figure consists of three modules: (1) a Waypoint Predictor that extracts RGB and depth inputs to generate candidate waypoints, (2) an MLLM Decision Module that selects the most goal-aligned waypoint, and (3) a Low-Level Controller that translates the selected waypoint into motion commands for execution physical robot. The system deployed on dual-arm mobile robot equipped with an RGB-D camera and evaluated in real indoor environments. More details are provided in the Appendix.
Evaluation on NayBench
5.1 Settings
Models We evaluate both proprietary and open-source MILMs widely adopted in recent research. Proprietary models include GPT-4o, GPT-4o-mini, Gemini-2.0-flash Open-source models include
InterMVL2.5-2B88B) [60], Qwen2.5-VLL3BB7B [61], LLaVA-One VisioonnVVisi (62]. LLaVA-Next-7B.,
and Llama3.22 2 1IB 633
Implementation Details Proprietary models are accessed via APIs, while open-source models deployed using vLLM (64 and Imdeploy 655 on a single NVIDIA A6000 GPU (48GB). Simulator- based evaluations are conducted in the Matterport3D Simulator (55] built on high-resolution RGB-D scans of real indoor environments such as homes and offices. It provides realistic visual inputs and discrete agent movement within a 3D mesh making it standard testbed embodied navigation. For real-world deployment, we integrate our pipeline with dual-arm composite mobile robot equipped with an Intel RealSense D435 camera and Water Drop 2 wheeled base. All physical experiments conducted in controlled indoor lab to assess robustness and feasibility.
7

---

Table I: Performance comparison Navigation Comprehension and Execution.
Navigation Comprehension
Navigation Execution
Model
Global
Progress
Local
Comp. Avg
Easy
Medium
Hard
Exec. Avg
Accuracy
SR
SPL
SPL
Chance Lovel (Random)
19.33
25.4
29.34
24.65
16.41
9.57
7.17
3.72
7.33
4.99
8.19
VIN--Bench (tiny) Performance
Human Level
88.33
79.00
91.67
81.53
75.00
65.17
81.59
GGT-44
51.67
45,00
63.00
53.89
66.08
49.01
43.79
36.44
25.00
20.11
40.07
1Qwen2.5-VL-7B
36.67
32.00
47.00
38.56
46.25
35.59
25.27
18.93
12.50
5.93
24.41
GPT-4a
51.33
42,90
65.80
53.34
67.36
54.31
6.
35.71
27.78
21.15
41.33
GPT-4o-mini
50.33
29.90
59.03
46.42
46.53
40.44
28.47
24.90
15.28
12.29
27.99
Gemini-2.0-Bane
79.68
40,30
32.00
50,66
61.81
45.05
46.53
39.08
25.69
16.64
39.13
o4-mini
76.67
43.60
58.70
59.66
47.92
44.77
26.39
22.70
15.97
10.13
28.98
Open-Source Models
Intem\L2..-2B
67.25
23.40
11.25
33.97
25. 69
25.29
6.94
6.68
7.64
5.86
13.02
Qwen2.5-VL
21.30
5.24
IntemVLL.5588
62.75
28.50
28.12
39.79
28.47
28.19
7.66
7.42
7.64
6.18
14.26
5-V
57.58
31.20
47.00
45.26
41.67
32.55
22.92
17.43
10.42
5.67
21.77
LLaVa-OneVision-T2
31.17
26.60
39.00
32.26
31.25
17.64
15.58
7,80
15.02
7.84
13.86
LLaVA-Next-7B
38.33
27.40
28.50
27.08
25.95
7.69
7.64
6.07
14.54
1Jama3.2 V-iiion- IR
36.00
23.40
29.10
14.75
27.08
25.90
10.42
9.19
10.02
7.60
15.04
Note: Dark teal
and light teal indicate the top-performing closed and open-source models per column.
indicates results evaluated on the NavBench (tiny) subset.
30
Scores
Scores
Basic
Direction
Object
Shuffle
Action
Observation Ower2
Intent2 5-28
Dowr2 541-30
randem
Intervi1 2.5-2
GPE-Ac
Down22 545--10
LLeth-Creihsen-2
GPT-40
1v5.2.55
LLak Oww.iiioennnn
Iniemi12 5-08
Liame3 am-2222 2211
GPI--bomin
InterVi2.5-8
Liamed 2-taaon--//i---11e
Figure 6: Model Performance under Different Instruction Perturbations.
Figure 7:
Model performance on Local
Observation-Action Reasoning.
Evaluation Metrics Our benchmark includes both multiple-choice reasoning and embodied naviga- tion execution tasks. For multiple-choice questions, follow standard practice (5) and Accuracy as the primary metric, which measures whether the model selects the correct answer from a set of candidates based on the provided information. For execution tasks, we adopt standard metrics in embodied navigation [0]. Success Rate (SR) measures the percentage of episodes where the target object is visible from the agent's final viewpoint, defined as being within a 3-meter radius. Success weighted by Path Length (SPL) adjusts SR by path efficiency and is computed as:
N Σ
E,
SPL
(6)
max(es, pt)
where N is the number of episodes, S, E {0, 1} indicates success, & is the shortest path, and is the path length.
VLN--enchh (tiny) Human Performance To provide an upper-bound reference, we additionally report human performance compact subset of VL.N-Bench, denoted VI.N-Bench (tiny), which was manually annotated and evaluated following the same protocol.
5.2 Performance
We begin by examining the relationship between comprehension and execution. As shown Table model performance on comprehension and execution tasks remains closely aligned. Among closed models, o4-mini achieves the highest comprehension average (59.66%) and maintains competitive execution performance (28.98%). GPT-4o follows with 53.34% and 41.33%, respectively, suggesting

---

Table 2: Impact of map information on GPT-4o.
Typet
Diff.
Map
SR
SPL
Avg
Gain
Irarret flan (s
Madlgral lccion (DN0)
X
67.36
54.31
60.84
bike flas purf))
Easy
Intluchcche Heveotrt (SS))
v
70.14
54.11
62.13
+1.29
x
41.67
35.71
38.69
Med.
46.53
39.86
43.20
+4.51
Figure 8: Distribution of navigation types identified from manual analysis.
x
27.78
21.15
24.47
Hard
29.17
22.32
25.75
+1.28
that 04-mini excels in understanding navigation instructions, while GPT-4o is stronger in executing them. Among open-source models, Swen2.55VL-77. achieves the best overall performance (45.26%, 21.77%), approaching GPT-4o-mini (46.4 (46,42%, 27. 77.99% and demonstrating potential for practical deployment in real-world robotics. Turning to comprehension subtasks, VL2.r52L2.5-22 performs strongly on Global Instruction Aligmment (67.25%), even surpassing GPT-4o (51.33%). However, its accuracy drops sharply more challenging reasoning tasks. In particular, Progress Estimation remains a consistent weakness across models; aside from GPT-4o (42.90%), all others perform poorly, highlighting current MILMs' limitations in temporal reasoning. We next analyze how models perform across navigation difficulty levels. Most open-source models can only reliably complete Easy episodes, while GPT-4o maintains relatively strong results across all levels, suggesting better generalization. These findings suggest several overarching insights. First, comprehension and execution abilities are strongly linked. Second, temporal reasoning, particularly progress tracking, remains a major bottleneck. Third, compact open-source models like Qwen2.5-VL-7 can offer competitive performance with significantly lower resource requirements, making them promising for embodied applications.
5.3 Discussion
Breakdown of Distractor Types in Instruction Alignment We further analyze performance on the Global Instruction Alignment task by breaking down results across four distractor types: basic, direction, object, and shuffle As shown in Figure 6 most models handle the basic condition well, indicating their ability reject unrelated instructions. However, performance under direction and object perturbations varies significantly across models, suggesting inconsistent grounding of spatial terms and landmarks. Notably, all models perform poorly under the shuffle condition, where sub- instructions are reordered but their content remains unchanged. This result is particularly revealing: despite the presence of the same entities and actions, altering the temporal structure makes the instruction much harder for models to interpret. The models' failure in this setting highlights their limited ability to reason about temporal order within complex instructions. This finding aligns with the low scores observed in the Progress Estimation task, reinforcing that current MLLMs struggle with temporal understanding across both instruction-level and trajectory-level reasoning.
Future-Action and Future-Observation Reasoning We analyze performance on the Local Inservation-Action Reasoning task, which includes two subtasks: Future-Action and Future- Observation Prediction. As shown in Figure 7 models show consistent performance across both, with GPT-4o clearly outperforming all others, consistent with its strong results in Navigation Execution. These subtasks reflect complementary reasoning skills. Future-Action Prediction tests whether model can infer the spatial transition between two views, while cature-Observation Prediction re- quires anticipating how the environment changes after given action. Both capabilities are critical for navigation, where agents should reason about cause and effect in spatial transitions.
Effect of Map Information on Action Decisions Although our benchmark evaluations assume no access to map information, reflecting real-world constraints, we investigate whether providing map connectivity can enhance action selection. Specifically, we follow the approach introduced in MapGPT, where topological relationships between explored nodes are encoded text prompts. Using GPT-4o, we compare performance with and without map input across different difficulty levels. As shown in Table 2 the presence of map information consistently improves success rates, with the largest gain observed under medium difficulty, yielding increase of 4.86 percentage points. This suggests that access to structured spatial context can facilitate better high-level reasoning and planning, especially in medium complexity settings where spatial ambiguity is more common.

---

Table 3: Performance comparison with and without CoT prompting.
Navigation Comprehension
Navigation Execution
Model
Global
Progress
Local
Comp. Avg
Fasy
Medium
Hard
Exec. Avg
Accuracy
SR
SPL
SPL
SR
SPL
GPT-4o
51.33
42.90
65,80
53.34
67.36
54.31
41.67
35.71
27.78
21.15
41.33
GPT-4o CoT
60.42
40.20
60.75
53.79
61.11
49.04
44.44
36.88
30.56
23.20
40.87
Qwen2.5-VL-7B
$7.58
31.20
47.00
45.26
22,92
17.43
10.42
5.67
21.77
Qwen2.5-VL VL-7B Col
60.50
31.40
48.00
46.63
43.75
33.06
22.92
15.62
11.11
7.19
22.28
Effect of Chain-of-Thoughh We incorporated Chain-of-Thhughh (CoT) prompting following 666 by prepending "Let's think step by step" to the instruction input. As shown in Table experiments were conducted using two of the strongest models in our benchmark: GPT-4o and Qwen2.5-VL-7B. The results show that CoT prompting brings noticeable improvement in the Global Instruction Alignment task. GPT-4o improved by 9.09% and Qwen2.5- -LL-BB improved by 2.92%. However, the gains in other comprehension tasks were marginal or slightly negative. We hypothesize that simple CoT prompting does not sufficiently enhance performance in spatial or temporal reasoning tasks, which often require more structured, multi-step planning rather than generic step-by-step thinking. For the navigation execution task, we observed little benefit from CoT prompting. This likely because the task itself already follows step-by-ster process: each time step, the model receives the full instruction history and must decide the next action. Therefore, additional CoT prompting provides limited benefit in this context.
Error Analysis (1) We manually analyze 100 failed cases to understand model failures. Based on thought traces and action sequences, we identify four common error types: (a) Incorrect Plan: the plan misaligns with the instruction; (b) Misaligned Action: the plan valid, but the chosen movement does not follow it; (c) Failure to Stop: the agent overshoots the goal or stops early; and (d) Hallucinated Movement: the model selects a nonexistent location. The error distribution is shown in Figure 8 These patterns align with weaknesses in comprehension tasks. For example, type (c) reflects poor Progress Extimation. This suggests execution failures often stem from temporal and spatial reasoning limitations, reinforcing the diagnostic value of NavBench.
(2) We further examine the impact of trajectory length on temporal reasoning. Test samples are grouped by length into short (1-2 steps), medium (3-4), and long (5+). For GPT-4o, the error rate increases from 35.3% (short) 42.9% (medium) and 76. 1 (long), showing that longer trajectories amplify temporal reasoning challenges. In contrast, weaker models such as LLa Vision-7B and Intern VL2.5-2B maintain high error rates across all lengths, indicating persistent difficulty in progress estimation regardless of path complexity.
WerldWorld Validation To assess the feasibility of our real-world deployment pipeline, conduct pilot study in an indoor environment using GPT-4o and Swen2.5-VL-77 the top proprietary and open-source models from our benchmark. Each model is tested on 10 cases, achieving success rates of 60% and 40%, respectively. These results show that both can handle simple navigation tasks in real-world settings. Their success trends mirror execution performance in Table where both models outperform others in their categories. This suggests that NavBench's 's simulation- based evaluation reliably reflects real-workd embodied performance.
Conclusion
This paper presents NayBench, diagnostic benchmark designed to evaluate MLLMs in embodied navigation under zero-shot settings. It decomposes the evaluation into two components: Navigation Comprehension, which evaluates global instruction alignment, temporal progress estimation, and local observation-acte reasoning through three cognitively grounded tasks, and Navigation Execution, which examines step-by-step secision-maaing across varying levels of difficulty. Additionally, we develop pipeline for real-world deployment of MLLM-driven agents. Through evaluation and targeted analysis, NavBench reveals limitations in temporal understanding and action grounding that are not captured by standard success metrics. It also shows that lightweight open-source models can be effective simpler navigation scenarios. We hope NavBench can serve as useful resource for analyzing the embodied capabilities of MLLMs and supporting future work in this direction.
10

---

References
[1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances Neural Information Processing Systems, volume 36, 2023.
[2] Peng Wang. Shuai Bai, Sinan Tan, Shijic Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint MxXii:2409.12199 2024.
[3] Gemini Team, Petko Georgiev Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05536 2024.
[4] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 1904-6913,200
[5] Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang. Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang. Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. ArXiv, abs/2405.21075. 2024.
[6] Pan Lu, Hritik Bansal, Tony Xia. Jiacheng Liu, Chunyuan Li. Hannanch Hajishirzi, Hao Cheng, Kai-Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
[7] Wenxiao Cai et al. Spatialbot: Precise spatial understanding with vision language models. In IEEE international conference roboticy and automation, 2025.
[8] Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[9] Peter Anderson, Qi Wa, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-len aan-uang navigation: Interpreting navigation instructions in real environments. In CVPR, pages 3674-3683, 2018.
[10] Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33:4247 4258, 2020.
[11] Benjamin Kuipers. The spatial semantic hierarchy. Artificial intelligence, 119(1-2):111-33,, 2000.
[12] Zhe Chen, Jiannan Wa, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Intern vl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks, IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages 24185-24198. 2023.
[13] lean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooci, Marianne Monteiro, Jacob Menick, Sebastian Borgcaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. ArXiv, 1is/2204.14198, 2022.
[14] Jinze Bai, Shuai Bai. Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilitics. ArXiv, abs/2308. 000008..1966. 12966, 2023.
[15] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26286-26296. 2023.
[16] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-670000 6709, 2019.

---

[17] Kenneth Marino, Mohammad Rastegari. Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/evf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[18] Amanpreet Singh. Vivek Natarajan, Meet Shah, Yu Jiang. Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[19] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
[20] Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216-233. Springer, 2025.
[21] Weihao Yu, Zhengyuan Yang. Linjie Li. Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308, 02490, 2023.
[22] Chenming Zhu, Tai Wang, Wenwei Zhang, Kai Chen, and Xihui Liu, Scanreason: Empowering 3d visual grounding with reasoning capabilities. In Eumpean Conference on Computer Vision, 2024.
[23] Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang. Chunhua Shen, and Anton van den Hengel. REVERIE: remote embodied visual referring expression in real indoor environments. In CVPR, pages 9979-9988, 2020.
[24] Zehao Wang. Minye Wu, Yixin Cao, Yubo Ma, Meiqi Chen, and Tinne Tuytelaars. Navigating the nuances: A fine-grained evaluation of vision-language navigation. ArXiv, abs/2409.17315 2024.
[25] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In JEEE international conference robotics and automation, pages 3357-3364. IEFF, 2017.
[26] Jacob Krantz, Stefan Lee, Jitendra Malik, Dhruv Batra, and Devendra Singh Chaplot. Instance-specific image goal navigation: Training embodied agents to find object instances. arXiv preprint arXiv:2211. 2022.
[27] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9339-9347, 2019.
[28] Shizhe Chen, Pierre-Louis Gubur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision- and--andd-dae language navigation. Advances Neural Information Processing Systems, 2021.
[29] Yue Zhang al. Vision-ame and language navigation today and tomorrow: A survey in the era of foundation models. Transactions on Machine Learning Research, 2024.
[30] Alexander Ku, Peter Anderson, Roma Patel, Eugene Ic, and Jason Baldridge. Room-across-room: Mul- tilingual sasion-and-language navigation with dense spatiotemporal grounding. In Proceedings of the Conference on Empirical Methods Natural Language Processing, pages 4392-4412, 2020.
[31] Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and- dialog navigation. In CoRL, pages 406,406,2220 2019.
[32] Hongcheng Wang, Andy Guan Hong Chen, Xiaoqi Li, Mingdong Wu, and Hao Dong. Find what you want: Learning demand conditioned object attribute space for demand-driven navigation. In Advances in Neural Information Processing Systems, 2023.
[33] Yuankai Qi, Zizheng Pan, Shengping Zhang, Anton van den Hengel, and Qi Wu. Object-and-actionn aware model for visual language navigation. In European Conference on Computer Vision, pages 303-317. Springer, 2020.
[34] Yicong Hong, Cristian Rodriguez, Yuankai Qi, Qi Wu, and Stephen Gould. Language and visual entity relationship graph for agent navigation. Advances in Neural Information Processing Syxtems, 33:7685- 7696, 2020.
[35] Dong An, Yuankai Qi. Yan Huang. Qi Wu, Liang Wang, and Tieniu Tan. Neighbor-vier view enhanced model for vision and language navigation. In Proceedings of the 29th ACM International Conference on 2021.
Multimedia, pages
12

---

[36] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning generic agent for vision-and-lant language navigation via pre-training In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 13137-13146, 2020.
[37] Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid. Airbert: In- domain pretraining for vision- ande--nn---a and-language navigation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1634-1643 2021.
[38] Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Improving vision-and-language navigation with image-text pairs from the web. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pages 259-274. Springer, 2020.
[39] Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev, History aware multimodal transformer for iision-and-languaa navigation. Advances in nearal information processing systems, 34:5834-5847, 2021.
[40] Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop: History-and-order aware pre-training for vision-and- language navigation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[41] Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang. and Qi Wu. Hop+: and order-aware pre-training for vision-and-land vision-and-language navigation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7)::524-8533 2023.
[42] Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. Bevbert: Mul- timodal map pre-training for language- guided navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2737-2748, 2023.
[43] Rui Liu, Xiaohan Wang. Wenguan Wang, and Yi Yang. Bird's view scene graph for vision-language language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10968- 10980, 2023.
[44] Zihan Wang, Xiangyang Li, Jiahao Yang. Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and and and-language navigation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 15625- 15636, 2023.
[45] Zun Wang. Jialu Li, Yicong Hong, Yi Wang. Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scaling data generation in vision-and-lanee languug navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12009-120200 2023.
[46] Aishwarya Kamath, Peter Anderson, Su Wang, Jing Yu Koh, Alexander Ku, Austin Waters, Yinfei Yang. Jason Baldridge. and Zarana Parekh. A new path: Scaling vision-and-languas navigation with synthetic instructions and imitation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10813-10823, 2023.
[47] Jiazhao Zhang, Kunyu Wang. Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang. Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-based based vlm plans the next step for vision-and-languuage navigation. arXiv preprint arXiv:2402. 15852, 2024.
[48] Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, and Xin Eric Wang. Esc: Exploration with soft commonsense constraints for zero-shot object navigation. In International Conference on Machine Learning, pages 42829-42842. PMLR, 2023.
[49] Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ihharco, Ludwig Schmidt, and Shuran Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 23171-23181, 2023.
[50] Naoki Yokoyama, Schoon Ha, Dhruv Batra, Jiuguang Wang, and Bernadette Bucher. Vlfm: Vision- language frontier maps for zero-shot semantic navigation. In IEEE international conference on rabotics and automation, pages 42-48. IEEF, 2024.
[51] Yuxing Long, Wenzhe Cai, Hongeheng Wang, Guanqi Zhan, and Hao Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. arXiv preprint arXiv:2406. 2024.
[52] Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and languag navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641-7649, 2024.
13

---

[53] Yanyuan Qiao, Wenqi Lyu, Hui Wang, Zixu Wang. Zerui Li, Yuan Zhang, Mingkui Tan, and Qi Wu. Open- nav: Exploring zero-shot Usion-andd-anguage navigation in continuous environment with open-source Ilms. arXiv preprint arXiv:2409.1877 18794, 2024.
[54] Jiaqi Chen, Binggian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang. and Kwan- Wong. Mapgpt: Map- guided prompting with adaptive path planning for Eision-and-languee navigation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9796-9810, 2024.
[55] Angel X. Chang, Angela Dai, Thomas A. Funkhouser, Maciej Halber, Matthias NieBner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from RGB-D data in indoor environments. In 3DV. pages 667 0667676,220 2017.
[56] Yibo Cui, Liang Xic, Yakun Zhang, Mcishan Zhang, Ye Yan, and Erwei Yin. Grounded entity- landmark adaptive pre-training for vision-and and- language navigation. IEEE/CVF International Conference on Com- puter Vision, pages 12009- 12019, 2023.
[57] Yicong Hong, Cristian Rodriguez, Qi Wu, and Stephen Gould. Sub-instruction aware rision-and-languaae navigation. In Proceedings of the Conference on Empirical Methods Natural Language Processing, pages 3360-3376, 2020.
[58] Xinyu Wang, Bohan Zhuang, and Qi Wu. Are large vision language models good game players? In Proceedings of the International Conference on Learning Representations, 2025.
[59] Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang. Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang. and Minlic Huang. Benchmarking complex instruction-followii with multiple constraints composition. ArXiv, hwtp/447700997.. 2024.
[60] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuchui Wang, Qingyun Li, Yiming Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang. Tan Jiang, Bo Wang. Conghui Hc, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang. Wenqi Shao, Pei Chu, Zhongying Tu, Tong He. Zhiyong Wu, Hui Deng. Jiaye Ge. Kaiming Chen, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahu Lin, Yunfeng Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. ArXiv, abs/2412.052711, 2024.
[61] Shuai Bai. Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. ArXiv, abs/2502. 133923, 2025.
[62] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint wxXv:2408.003326 2024.
[63] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[64] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattentoo.. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[65] LMDepply Contrihutors. Lmdeploy: A toolkit for compressing, deploying, and serving 1lm. https //ithub. wtrthuu.ccomiiee www.ccommmntti homm//nnterriiM/mdpeloo.. 2023.
[66] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Lc, and Denny Zhou. ofain-of-thoughh prompting clicits reasoning in large language models. In Annual Conference on Neural Information Processing Systems, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the contributions of the paper, including the design of NavBench, task decomposition, difficulty stratification, and real- world deployment, all of which are substantiated by experiments. (See Section and Abstract)
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper discusses limitations of the proposed benchmark in the supplemen- tary material.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include any theoretical results or formal proofs; it purely empirical and benchmarking- -ased.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper includes sufficient implementation details and describes benchmark construction, task setup, and evaluation procedures. (See Sections 3-5)
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
16

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We will release the dataset and code. Anonymized supplementary material includes reproduction instructions. (Details in Supplementary)
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwtpcc/pccppcpcccceooooooop gwwcc//guuee///eee / semeiissinnpp ssinnolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper details all relevant settings, including model types, deployment methods, simulator configuration, robot setup, and evaluation metrics. (See Section 5.1)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not report error bars or significance tests. The evaluation focuses on average performance across tasks and difficulty levels.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the hardware setup (e.g.. A6000 GPU, robot platform), and simulator environments used for training and testing. (See Section 5.1)
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conforms with the NeurlPS Code of Ethics. All data used are from publicly available benchmarks.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work presents benchmark for evaluating MILMS embodied naviga- tion using publicly available data in simulated and controlled lab environments. It does not involve human subjects, private data, or direct deployment scenarios, and we do foresee any immediate societal impact.
18

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.oymmly of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper does not release any models or datasets that pose high risk for misuse. All evaluated models are publicly available, and the benchmark is constructed from existing datasets with no sensitive content.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All used datasets (e.g., R2R, RxR, GEL-R2R) and models (e.g.. GPT-4o, Qwen2.5- VL) are publicly available and properly cited in the paper. We respect their terms of use and follow the licenses specified by their original creators.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thwode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The paper introduces new benchmark, NavBench, constructed from existing datasets but reorganized into new evaluation tasks with structured question-answer pairs and difficulty annotations. We provide documentation and plan to release anonymized assets upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any new crowdsourcing or research with human participants. All data used were sourced from publicly available datasets that already contain necessary annotations.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
20

---

Answer: [NA]
Justification: The paper does not involve research with human subjects as defined by IRB standards. All human judgments were conducted internally by the authors without collecting personal or sensitive data.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
This work systematically evaluates the embodied reasoning capabilities of existing LLMs (e.g.. GPT-4o, Qwen2.5-VL) in navigation tasks. These models are not developed by the authors, but they are central to the paper's 's evaluation design and analysis. Their usage is described in detail throughout the paper.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non- standard components.
Please refer to our LLM policy (https://neurd)) hatpr//ppp..ccoo.ffeeeeeeee///2 for what should or should be described.
21


Reviews:
[{"review_id": "FC2y7A4TjH", "reviewer": "Reviewer_vvYt", "summary": "This work introduces NavBench to evaluate the multimodal large language models' capabilities, including navigation comprehension and navigation execution capabilities in VLN. Experiments and analyses with 2 closed models and 4 open-sourced models on the NavBench benchmark show some interesting findings.", "questions": "- The length of trajectories matters for temporal progress estimation. So is it possible to do an analysis about the relationship between errors and trajectory length?\n- The paper does not mention whether chain-of-thought thinking is used for models' predictions. Considering there are some discussions about whether thinking is useful for navigation tasks in recent literature, I suggest to add some comparisons between using thinking and without thinking. \n- Considering the reasoning capability of thinking models, is it possible to test performance of some reasoning models such as o4-mini in this benchmark?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- Strengths:\n\t- The paper is well-written and easy to follow.\n\t- The paper introduces three navigation comprehension tasks and one navigation execution task to evaluate the commonly used VLMs' capabilities. The introduced tasks are carefully designed and expected to be quite useful to evaluate thee VLMs' capabilities in the context of navigation.\n\t- The tested models are quite diverse, including both proprietary and open-source models.\n\t- The proposed difficulty level for navigation execution tasks is novel and useful.\n- Weaknesses:\n\t- The paper is a benchmark paper but not submitted to the benchmark track. So we cannot really check the content and quality of the benchmark. Moreover, the paper does not discuss the license regarding the benchmark release. Considering that Matterport3D is not fully open-sourced and has its own license, so it is important to discuss the license if the authors plan to release the benchmark.\n\t- The paper uses panoramic images as observation format for all the tasks in the benchmark. This is no longer the common setting for the VLN-CE task, which is a more popular task nowadays. The VLN-CE task usually adopts egocentric videos as observation input, so the findings of this paper may have limit to extend to it.\n\t- The navigation execution task in the benchmark, where a model is required to pick up the next-step viewpoint, is almost the same as the common setting of VLN task. So there is a concern why not directly use the standard VLN task?", "was_revised": true, "final_justification": "All my concerns proposed in my weakness and questions, except the use of panoramic images instead of egocentric images, are addressed by the authors' responses. So I would like to raise my score from 4 to 5.", "rebuttal": "We appreciate your constructive feedback and suggestions. Please find our responses to each comment below.\n\n---\n> **Q1:** Benchmark transparency and licensing concerns.\n\n**A1:** Thank you for raising these concerns. We agree that benchmark transparency and reproducibility are essential. To support this, we provide detailed descriptions in Section 3 of the main paper and include additional details, sample annotations, and evaluation scripts in the supplementary material. We hope this allows reviewers to assess the quality and reproducibility of the benchmark.\n\nRegarding licensing, we strictly adhere to Matterport3D’s academic usage terms. We do not release any rendered images or meshes. Instead, we provide structured annotations (e.g., instructions, trajectories, and QA pairs) and scripts that allow users to reproduce the visual input via the official simulator. This practice follows the precedent set by benchmarks like R2R. A licensing statement will be included in the final release to ensure compliance.\n\n---\n> **Q2:** Use of panoramic images instead of egocentric video in VLN-CE tasks.\n\n**A2:** Although egocentric videos have become a common input format in recent VLN-CE tasks, panoramic observations remain a viable and widely adopted setting in prior and recent works. For example, both ETPNav [1] and Open-Nav [2] conduct experiments on VLN-CE with panoramic observations, highlighting their utility in evaluating high-level navigation reasoning.\n\nWe choose panoramic inputs for two key reasons. First, panoramic observations offer a complete 360-degree view of the environment, which provides better spatial grounding for language-conditioned navigation and reduces the ambiguity that may arise from a narrow field-of-view. This is particularly important when evaluating multimodal large language models (MLLMs), which benefit from a richer global context to perform reasoning and instruction alignment.\n\nSecond, panoramic views can be practically implemented in real-world navigation systems. In our real-robot deployment, we rotate the robot in place to capture 12 directional images, forming a panoramic observation. This setup shows that panoramic input is not limited to simulation and can be readily applied in real environments.\n\nTherefore, although our benchmark currently adopts panoramic views, we consider it compatible with the goals of VLN-CE evaluation and can provide complementary insights into model capabilities. We also consider extending our benchmark to support egocentric video inputs in future iterations.\n\n---\n> **Q3:** Why not directly use the standard VLN task for execution?\n\n**A3:** While our execution task shares the same basic formulation as the standard VLN setting, where the agent selects the next-step viewpoint, our design emphasizes a different evaluation focus. We aim to assess the capabilities of MLLMs under zero-shot conditions, without any task-specific training or fine-tuning.\n\nA key distinction lies in our explicit difficulty-aware design. Instead of treating all trajectories uniformly, we define a composite complexity score across three orthogonal dimensions: spatial, cognitive, and execution complexity. This allows us to categorize the tasks into three difficulty levels (easy, medium, and hard) and conduct fine-grained evaluation. For example, GPT-4o achieves a success rate of 67.36% on the easy subset, suggesting that it may already be suitable for deployment in relatively simple real-world scenarios.\n\n---\n> **Q4:** Error analysis on trajectory length in progress estimation.\n\n**A4:** We appreciate this insightful suggestion. We agree that trajectory length may significantly affect a model’s ability to estimate temporal progress. To investigate this, we grouped the test samples by trajectory length into three buckets: short (1–2 steps), medium (3–4), and long (5 or more), and computed the error rates for different models. For GPT 4o, the strongest overall performer, we observed a clear increase in error rate from 35.3% (short) to 42.9% (medium) and 76.1% (long). This indicates that longer trajectories pose greater challenges for temporal reasoning, possibly due to accumulated ambiguity or compounding errors. In contrast, lower-performing models such as LLaVA-OneVision-7B and InternVL2.5-2B show consistently high error rates across all lengths (often above 80–90%), suggesting they struggle regardless of trajectory complexity and may operate close to random guessing. We will include this analysis in the revised version.\n\n---\n> **Q5:** Chain-of-thought prompting for navigation tasks.\n\n**A5:** Thank you for the helpful suggestion. We incorporated Chain of Thought (CoT) prompting following [3] by prepending “Let’s think step by step” to the instruction input. We conducted experiments using two of the strongest models in our benchmark: GPT-4o and Qwen2.5-VL-7B.\n\nOur results show that CoT brings noticeable improvement in the Global Instruction Alignment task. GPT-4o improved by 9.09 percent, and Qwen2.5-VL-7B improved by 2.92 percent. However, the gains in other comprehension tasks were marginal or slightly negative. We hypothesize that simple CoT prompting may not sufficiently enhance the model’s performance in spatial and temporal reasoning tasks, which often require more structured or multi-step planning rather than generic step-by-step thinking.\n\nFor the navigation execution task, we observed little benefit from CoT prompting. This is likely because the task is already structured in a step-by-step fashion. At each step, the model receives the full instruction history and must decide on the next action. Therefore, additional prompting provides limited benefit.\n\nWe will include this analysis and the following performance comparison table in the revised version:\n\n**Table 1: Performance Comparison with and without CoT Prompting**\n\n| Model               | Global | Progress | Local  | Comp. Avg | Easy SR | Easy SPL | Med. SR | Med. SPL | Hard SR | Hard SPL | Exec. Avg |\n|---------------------|--------|----------|--------|-----------|---------|----------|--------|----------|---------|----------|-----------|\n| **GPT-4o**           | 51.33  | 42.90    | 65.80  | 53.34     | 67.36   | 54.31    | 41.67  | 35.71    | 27.78   | 21.15    | 41.33     |\n| **GPT-4o + CoT**     | 60.42  | 40.20    | 60.75  | 53.79     | 61.11   | 49.04    | 44.44  | 36.88    | 30.56   | 23.20    | 40.87     |\n| **Qwen2.5-VL-7B**    | 57.58  | 31.20    | 47.00  | 45.26     | 41.67   | 32.55    | 22.92  | 17.43    | 10.42   | 5.67     | 21.77     |\n| **Qwen2.5-VL-7B + CoT** | 60.50  | 31.40    | 48.00  | 46.63     | 43.75   | 33.06    | 22.92  | 15.62    | 11.11   | 7.19     | 22.28     |\n\n*Note: Comp. = Comprehension; Exec. = Execution; SR = Success Rate; SPL = Success weighted by Path Length; Med. = Medium.*\n\n---\n> **Q6:** Performance of reasoning-focused models such as o4-mini.\n\n**A6:** Based on your valuable suggestion, we included the reasoning-focused model o4-mini in our evaluation. As shown in the table below, o4-mini demonstrates strong performance on the comprehension tasks, particularly on the Global Instruction Alignment task. It achieves the highest average comprehension score among all models, outperforming GPT-4o’s 53.34.\n\nHowever, its performance on the navigation execution task is less impressive and is generally on par with GPT-4o-mini. This aligns with our observation from the CoT experiments: o4-mini appears to be better suited for reasoning-based tasks that involve question answering and language understanding, but shows limitations when applied to decision-making tasks that require grounding and multi-step action planning.\n\nThese findings and discussions will be incorporated into the revised version.\n\n**Table 2: Performance of o4-mini across comprehension and execution tasks.**\n\n| Model     | Global | Progress | Local  | Comp. Avg | Easy SR | Easy SPL | Med SR | Med SPL | Hard SR | Hard SPL | Exec. Avg |\n|-----------|--------|----------|--------|-----------|---------|----------|--------|----------|---------|----------|-----------|\n| o4-mini   | 76.67  | 43.60    | 58.70  | 59.66     | 47.92   | 44.77    | 26.39  | 22.70    | 15.97   | 10.13     | 28.98     |\n\n*Note: Comp. = Comprehension; Exec. = Execution; SR = Success Rate; SPL = Success weighted by Path Length; Med. = Medium.*\n\n---\nWe hope these clarifications and additional experiments address your concerns. Thank you again for your encouraging feedback and constructive suggestions.\n\n---\n### References\n[1] *ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments*. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024.  \n[2] *Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMs*. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2025.  \n[3] *Chain-of-thought prompting elicits reasoning in large language models*. Advances in Neural Information Processing Systems (NeurIPS), 2022."}, {"review_id": "n0bIxFlbmY", "reviewer": "Reviewer_uQaG", "summary": "This paper introduces an evaluation benchmark for multimodal language models for vision-and-language embodied navigation tasks, specifically in zero-shot settings. Their methodology proposes evaluating both the comprehension as well as execution in the navigation task. They also develop a pipeline for real-world deployment of these agents in environments. They show how this analysis can reveal limitations in current models in temporal understanding and other factors, that previous metrics obfuscate; and also show how current zero-shot models can be thoroughly tested for such environments.", "questions": "1. There are now a few vision and language datasets that come in multiple languages (e.g, english -> trajectories, hindi -> trajectories). It would be worth discussing whether this evaluation, especially for the comprehension component, needs to factor a language id in, and if there would be any differences expected. If not (which, OTTH, there should not be currently) it would be worth pointing out that a strength of this methodology is that it is language agnostic, and also, future work on the multilingual datasets might be interesting insights from such metrics.\n2. Can we get an upperbound on model performance for VLN models? The numbers in the table are fairly low, which is expected for zero-shot LLMs, but the best-performing VLN models (that are trained specifically for this task) would be good to see alongside, to calibrate the difference.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. The evaluation proposed by this paper is important to the field of vision/language navigation and mulltimodal models.\n2. It draws from relevant existing work to use metrics that are well-tested and thought out (e.g., not just goal accuracy and trajectory metrics, but weighted averages of them and so forth) and combines them into one benchmark,\n3. They show how this benchmark reveals weaknesses of models that previous benchmarks and metrics could not easily delineate.\n4. It is well written and explained and all metrics are easily reproducible and will be useful for future work.\n5. They illustrate how models they evaluate can be deployed in the real world, specifically on a dual-arm robot.\n6. They evaluate a sufficient range of models (including the state-of-the-art LLMS (although they might not be the best VLN models).", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your positive and constructive feedback. Thank you for recognizing the value of our work. We provide clarifications on your insightful suggestions below.\n\n---\n> **Q1:** Multilingual support in the benchmark.\n\n**A1:**  Thank you for the constructive suggestion regarding multilingual evaluation. While our current benchmark is conducted in English, we agree that assessing model generalization across languages is a valuable direction.\n\nTo explore this, we conducted a preliminary study on the Global Instruction Alignment task, which is part of our navigation comprehension evaluation. This task involves aligning a language instruction with a visual trajectory and is particularly suitable for analyzing the effect of instruction language, as it directly tests the model’s ability to semantically match textual descriptions with visual paths.\n\nWe selected two representative models, GPT-4o (the strongest closed-source model) and Qwen2.5-VL-7B (the best-performing open-source model), and translated 30 randomly sampled instructions into four languages. Two of them, Italian and German, are typologically close to English, while the other two, Hindi and Telugu, are linguistically distant and included in the multilingual RxR dataset. To isolate the effect of instruction language, the rest of the prompt was kept in English. The translations were performed automatically without human post-editing to simulate a zero-shot cross-lingual setting.\n\nThe accuracy results  are shown below:\n\n**Table 1: Multilingual Global Instruction Alignment Accuracy**\n\n| Model              | English | Hindi | Telugu | Italian | German |\n|--------------------|---------|-------|--------|---------|--------|\n| GPT-4o             | 55.00   | 37.50 | 49.17  | 45.00   | 51.67  |\n| Qwen2.5-VL-7B      | 62.50   | 48.33 | 32.50  | 60.83   | 54.17  |\n\nWe observe that performance degradation is relatively minor for Italian and German, while both models exhibit noticeable drops on Hindi and Telugu. This supports the reviewer’s intuition that language differences can impact model performance, particularly when the language diverges significantly from English.\n\nSince this is a small-scale pilot study based on automatic translation without human post-editing, we treat the results as exploratory. In future work, we plan to expand multilingual evaluation with larger-scale experiments and explore the inclusion of language ID tokens to improve cross-lingual robustness. We will summarize this discussion in the revised version.\n\n---\n> **Q2:** Upperbound performance.\n\n**A2:** Thank you for this suggestion. Including an upper bound is indeed important for interpreting zero-shot MLLM performance. Due to time constraints, we are currently unable to provide results from task-specific trained VLN models, but we plan to include them in future versions for a more complete comparison.\n\nIn the meantime, we provide a human performance reference derived from a compact subset of the benchmark. Given the scale of the dataset, full human annotation is infeasible, so we randomly sampled 392 representative instances across all tasks. The resulting accuracy (84.11% for comprehension and 81.59% for execution) serves as a practical empirical upper bound for interpreting current MLLM performance. \n\n---\nWe thank the reviewer again for the thoughtful suggestions. We believe these additions will further improve the clarity and scope of our work, and we will incorporate the relevant discussions into the revised version."}, {"review_id": "xki33R1fY0", "reviewer": "Reviewer_5z5E", "summary": "This paper presents NavBench, a benchmark designed to evaluate the embodied navigation capabilities of Multimodal Large Language Models (MLLMs) under zero-shot settings. \n\nThe benchmark consists of two components: (1) navigation comprehension tasks evaluating global instruction alignment, temporal progress estimation, and local observation-action reasoning; (2) step-by-step execution tasks across different difficulty levels. \n\nThe authors evaluate both proprietary and open-source MLLMs on NavBench, highlighting persistent challenges in temporal reasoning.", "questions": "1. Could the authors provide more details about the quality control methods of constructed Temporal Progress Estimation data?\n\n2. Regarding the complexity score weights (α,β,γ), the paper states they are \"empirically set.\"  Was there a specific calibration step against the human ratings, or were they chosen based on prior work and intuition?", "limitations": "The paper would benefit from a more detailed discussion of potential biases in the constructed QA pairs.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. The paper addresses a clear gap in evaluating MLLMs for embodied navigation, moving beyond static vision-language tasks to dynamic, sequential decision-making scenarios.\n\n2. The three-level comprehension assessment (global, progress, local) is cognitively grounded and provides diagnostic insights.\n\n3. The systematic categorization of tasks by spatial, cognitive, and execution complexity with both automatic scoring and human validation is methodologically sound.\n\n4. The paper evaluates multiple models across different scales and provides a detailed analysis of failure modes, revealing important insights about temporal reasoning limitations.\n\n## Weaknesses\n1. For Temporal Progress QA pairs construction, how the authors do the filtering (\"using a curated list of valid instruction-path pairs\") is not clear.\n\n2. The weights α, β, γ for difficulty scoring are \"empirically set\" but the justification is unclear.", "was_revised": true, "final_justification": "The paper quality is good, and the authors have addressed my concerns. I will keep my positive rating.", "rebuttal": "We sincerely thank the reviewer for the constructive and encouraging feedback. We provide clarifications below for the points raised.\n\n---\n> **Q1:** More details about the quality control methods of constructed Temporal Progress Estimation data.\n\n**A1:** Thank you for raising this point. The QA pairs for the Temporal Progress Estimation task were selected from a larger candidate pool through a combination of automatic filtering and manual review.\n\nWe first applied an automatic rule to exclude QA pairs in which two adjacent sub-instructions ended at the same viewpoint, as such cases typically indicate unclear temporal boundaries. After removing these structurally ambiguous pairs, we manually reviewed the remaining candidates and excluded samples that lacked a clear or meaningful sense of temporal progress, such as those where the chunk images showed minimal spatial change or the sub-instruction-to-trajectory alignment was unclear.\n\nThis two-stage filtering process ensured that the final QA pairs exhibit visually and semantically distinct progression steps. We will include these filtering details in the revised version.\n\n---\n> **Q2:** Clarification on complexity score weights.\n\n**A2:** Thanks for your question. To determine the weights in our complexity scoring framework, we conducted a detailed statistical analysis of the numerical features associated with each complexity dimension: spatial, cognitive, and execution.\n\nFirst, we extracted structured features from each trajectory. For spatial complexity, we included path length, standard deviation of turning angles, elevation range, and two-dimensional area coverage. Cognitive complexity was based on instruction length, number of verbs, number of spatial terms, number of landmark mentions, and number of subordinate clauses. Execution complexity was measured by the number of steps, number of turns, presence of floor changes, and number of decision points.\n\nSecond, we sampled data from 50 scenes and computed summary statistics for each feature, including mean, standard deviation, and min–max range. This analysis helped us understand the scale and variation of each feature. For example, instruction length exhibited a broader numerical spread than the number of landmark mentions, which would otherwise lead to a disproportionate impact on the cognitive complexity score.\n\nThird, we initialized all feature weights equally within each complexity category and then adjusted them based on the observed distributions to ensure balanced contributions. Features with inherently larger value ranges were assigned smaller weights to avoid dominating the overall score. We also tuned the aggregation weights across the three complexity dimensions to ensure a balanced contribution to the final difficulty score.\n\nDuring the adjustment process, we also considered the distribution of human difficulty ratings to improve the alignment between automatic scores and perceived complexity. To evaluate the robustness of the weighting scheme, we performed a sensitivity analysis by perturbing each weight by 20 percent while keeping the others fixed. As shown in Figure 1 (supplementary), over 90 percent of samples retained their original difficulty category, indicating that the scoring method is relatively stable.\n\nThese details will be incorporated into the revised manuscript to improve clarity."}, {"review_id": "tnItBmR8KQ", "reviewer": "Reviewer_eAcf", "summary": "This paper proposes a benchmark with Multimodal Large Language Models for Embodied Navigation. Two components are included in this work, navigation comprehension and execution. This work also supports that converts MLLMs' outputs into robotic actions.", "questions": "- The main concerns are listed in the Weaknesses section.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- Strengths\n    - This paper is will written, and easy to follow.\n    - This paper proposes a navigation benchmark specifically for MLLMs, focusing on exploring their long-term processing capabilities in 3D scene.\n    - This paper implements a systematic comparison and analysis of the performance trends of closed-source and open-source MLLMs in embodied reasoning and decision-making.\n \n- Weakness\n    - This paper proposes a new benchmark, which is mainly constructed based on existing datasets such as R2R and RxR, and the evaluation metrics lack novelty. I recommend that the authors further consider designing new input-output formats and evaluation criteria to comprehensively assess the spatial reasoning abilities of multimodal large models.\n    - This paper proposes a pipeline for real-world deployment, but the current description is rather general. I suggest the authors provide additional details regarding the generation process of the waypoint heatmap, the algorithm used to extract predicted waypoints, and the specific methods for calculating the angles and distances between different waypoints. Including algorithmic descriptions and visualizations of these steps would significantly improve the clarity and reproducibility of the pipeline.\n    - This paper evaluates proprietary MLLMs (e.g., GPT-4o) and open-source MLLMs (e.g., InternVL2.5-2B). I have the following questions and suggestions: (1) Have the authors considered including more open-source models (e.g., Gemini series) and proprietary models (e.g., LLaVA-Next) to enrich the comparison? (2) In Table 1, the open-source models cover a wide range of parameter sizes (e.g., 2B, 3B, 7B, 11B). For fair comparison, it is recommended to match model sizes as much as possible; (3) It would be beneficial to see more ablation studies to provide a finer-grained analysis of the current MLLMs’ performance and limitations in zero-shot navigation tasks within 3D environments.", "was_revised": true, "final_justification": "The author provides more details about experiments and motivation. I recognize their efforts about exploring MLLMs in embodied navigation, and give a positive final score.", "rebuttal": "We thank the reviewer for taking the time to provide detailed feedback. We have carefully reviewed all the comments and provide point-by-point responses below. We hope these clarifications help resolve any potential misunderstandings and more clearly convey the contributions and intentions of our work.\n\n---\n\n> **Q1:** Novelty concern regarding reliance on existing datasets and suggestion to design new input/output formats and evaluation metrics.\n\n**A1:** We understand the reviewer’s concerns, which may reflect a possible misunderstanding about the novelty of our task design and evaluation setup. We clarify each point below:\n\n**(1) Use of existing datasets.**\nOur benchmark utilizes navigation paths and instructions from R2R and RxR due to their wide adoption and strong coverage of realistic indoor scenes. However, we did not simply concatenate or reuse these datasets. All visual inputs were re-collected using the Matterport3D simulator to align with our specific task requirements, and the underlying data were reorganized to support our newly designed tasks.\n\nRepurposing established datasets is a widely accepted practice in benchmark design, as demonstrated by recent works such as VSI-Bench [1] and EmbodiedScan [2]. What distinguishes our work is the construction of entirely new evaluation tasks, including three comprehension tasks (global alignment, temporal progress estimation, and local action reasoning) and an execution task. These are based on restructured inputs and newly defined outputs, supporting a zero-shot evaluation protocol for MLLMs that is fundamentally different from prior trajectory-following settings.\n\n**(2) Evaluation metrics.**\nWe adopt QA-style accuracy for comprehension and use step-wise success rate along with standard navigation metrics such as SR and SPL for execution. Although these metrics are commonly used in prior work, we found them to be practical and interpretable in the context of our newly designed tasks. They allow us to capture model performance at different stages of navigation, from high-level understanding to low-level execution. \n\n**(3) Designing new input/output formats and assessing spatial reasoning.**\nThank you for the suggestion. While our benchmark does not isolate spatial reasoning as a standalone capability, it is explicitly embedded in tasks such as Local Observation-Action Reasoning. In this task, the model is required to predict how the visual scene would change after taking a specific action, which involves understanding the spatial layout and action affordances within the 3D environment. In future extensions of the benchmark, we will consider designing additional spatial reasoning tasks tailored to embodied navigation scenarios.\n\n---\n> **Q2:** More details of the real-world deployment pipeline.\n\n**A2:** Thank you for your suggestion. As we were constrained by the main paper’s space limit, we included the detailed description of the deployment pipeline in the supplementary material (Section C). This includes panoramic image construction, depth estimation, feature fusion with dual ResNet-50 backbones, transformer-based heatmap generation, waypoint selection using NMS, and translation into robot actions (Equations 1–2).\n\nWhile additional visualizations cannot be included here due to formatting limits, we will incorporate them in the revised version to improve clarity. These materials will also be released alongside our code to support reproducibility.\n\n---\n> **Q3:** Suggestions to (1) include more models (e.g., Gemini, LLaVA-Next) for broader comparison, (2) control for model size across comparisons, and (3) conduct more ablation studies.\n\n**A3:** Thank you for the valuable suggestions. We address each point below:\n\n**(1) Inclusion of more models.**\nWe evaluated additional models to further enrich the comparison as suggested. This includes both proprietary models (e.g., Gemini-2.0-flash, OpenAI o4-mini) and open-source models (e.g., LLaVA-Next-7B). Results are summarized below:\n\n**Table 1: Performance comparison on Navigation Comprehension and Execution**\n| Model               | Global | Progress | Local  | Comp. Avg | Easy SR | Easy SPL | Med. SR | Med. SPL | Hard SR | Hard SPL | Exec. Avg |\n|---------------------|--------|----------|--------|-----------|---------|----------|--------|----------|---------|----------|-----------|\n| **Gemini-2.0-flash**           | 79.68  | 40.30    | 32.00  | 50.66     | 61.81   | 45.05    | 46.53  | 39.08    | 25.69   | 16.64    | 39.13     |\n| **o4-mini**   | 76.67  | 43.60    | 58.70  | 59.66     | 47.92   | 44.77    | 26.39  | 22.70    | 15.97   | 10.13     | 28.98     |\n| **LLaVA-Next-7B**    | 38.33  | 27.40    | 28.50  | 31.41     | 27.08   | 25.95    | 11.81  | 7.69    | 7.64   | 6.07     | 14.54     |\n\n*Note: Comp. = Comprehension; Exec. = Execution; SR = Success Rate; SPL = Success weighted by Path Length; Med. = Medium.*\n\n**(2) Controlling for model size.**\nSince Open-source models vary considerably in size and architecture, it is a bit challenging to make them exact size matching. Thus, we grouped models into approximate size tiers (e.g., 2–3B, 7–8B) to ensure general comparability while covering a diverse set of representative models. We will continue to expand the benchmark with additional models organized by size to facilitate more fine-grained comparisons.\n\n**(3) Additional ablation studies.**\nWe conducted an ablation study to assess the effect of Chain-of-Thought (CoT) prompting by prepending “Let’s think step by step” to the text input, following [3]. For a representative comparison, we selected one proprietary model (GPT-4o) and one open-source model (Qwen2.5-VL-7B) based on their strong overall performance.\n\n**Table 2: Performance Comparison with and without CoT Prompting**\n| Model               | Global | Progress | Local  | Comp. Avg | Easy SR | Easy SPL | Med. SR | Med. SPL | Hard SR | Hard SPL | Exec. Avg |\n|---------------------|--------|----------|--------|-----------|---------|----------|--------|----------|---------|----------|-----------|\n| **GPT-4o**           | 51.33  | 42.90    | 65.80  | 53.34     | 67.36   | 54.31    | 41.67  | 35.71    | 27.78   | 21.15    | 41.33     |\n| **GPT-4o + CoT**     | 60.42  | 40.20    | 60.75  | 53.79     | 61.11   | 49.04    | 44.44  | 36.88    | 30.56   | 23.20    | 40.87     |\n| **Qwen2.5-VL-7B**    | 57.58  | 31.20    | 47.00  | 45.26     | 41.67   | 32.55    | 22.92  | 17.43    | 10.42   | 5.67     | 21.77     |\n| **Qwen2.5-VL-7B + CoT** | 60.50  | 31.40    | 48.00  | 46.63     | 43.75   | 33.06    | 22.92  | 15.62    | 11.11   | 7.19     | 22.28     |\n\n*Note: Comp. = Comprehension; Exec. = Execution; SR = Success Rate; SPL = Success weighted by Path Length; Med. = Medium.*\n\nWe observed that CoT prompting yielded moderate gains in global instruction alignment but had limited or even negative effects in spatial and temporal reasoning tasks. For the execution task, CoT brought little benefit, likely because our step-wise formulation already structures the decision-making process.\n\nWe will include these results and tables in the revised version.\n\n---\n> **Q4:** The formatting of this paper requires further refinement. \n\n**A4:** Thank you for the suggestions. We will carefully revise the manuscript to address the listed formatting issues and these improvements will be reflected in the revised version.\n\n---\nWe sincerely hope our responses help clear up any confusion and alleviate your concerns.\n\n---\n### References\n[1] *Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces*. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.  \n[2] *EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI*. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.  \n[3] *Chain-of-thought prompting elicits reasoning in large language models*. Advances in Neural Information Processing Systems (NeurIPS), 2022."}]

Author rebuttal:

Meta-review: This paper introduces NavBench, a benchmark for evaluating multimodal LLMs in embodied navigation, with tasks covering global instruction alignment, temporal progress estimation, local reasoning, and step-wise execution under a novel difficulty-aware design. The work is well-written, and evaluates a diverse set of open-source and proprietary models, revealing important insights into temporal reasoning challenges, trade-offs between comprehension and execution, and the limited benefits of chain-of-thought prompting. While some novelty concerns exist due to reliance on existing datasets and panoramic input formats rather than the increasingly standard egocentric videos, the benchmark nonetheless fills a clear gap in evaluating navigation reasoning and supports real-world deployment scenarios. During rebuttal, the authors convincingly addressed licensing, data filtering, complexity weighting, error analysis, multilingual evaluation, and extended the model comparisons, which resolved most reviewer concerns.
Decision: Accept (poster)