CitySeeker: How Do VLMs Explore Embodied Urban Navigation with Implicit Human Needs?
Authors: Siqi Wang
Chao Liang
Yunfan Gao
Erxin Yu
Sen Li
Jing Li
Haofen Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Embodied Urban Navigation
Vision-Language Models
Urban Intelligence
Spatial Cognition
Urban Planning
Primary area: datasets and benchmarks
TL;DR: We introduce CitySeeker, a novel benchmark for evaluating Vision-Language Models on embodied urban navigation driven by implicit human needs.

Abstract: Vision-Language Models (VLMs) have made significant progress in explicit instruction-based navigation; however, their ability to interpret implicit human needs (e.g., ''I am thirsty'') in dynamic urban environments remains underexplored. This paper introduces CitySeeker, a novel benchmark designed to assess VLMs’ spatial reasoning and decision-making capabilities for exploring embodied urban navigation to address implicit needs. CitySeeker includes 6,440 trajectories across 8 cities, capturing diverse visual characteristics and implicit needs in 7 goal-driven scenarios. Extensive experiments reveal that even top-performing models (e.g., Qwen2.5-VL-32B-Instruct) achieve only 21.1% task completion. We find key bottlenecks in error accumulation in long-horizon reasoning, inadequate spatial cognition, and deficient experiential recall. To further analyze them, we investigate a series of exploratory strategies—Backtracking Mechanisms, Enriching Spatial Cognition, and Memory-Based Retrieval (BCR), inspired by human cognitive mapping's emphasis on iterative observation-reasoning cycles and adaptive path optimization. Our analysis provides actionable insights for developing VLMs with robust spatial intelligence required for tackling ''last-mile'' navigation challenges.

Published as a conference paper at ICLR 2026
CITYSEEKER: How Do VLMS EXPLORE EMBODIED URBAN NAVIGATION WITH IMPLICIT HUMAN NEEDS?
Siqi Wang 1,233 12. Chao Liang", Yunfan Gao Erxin Yu², Sen Li Jing Haofen Wang"*
Polyy Shenzhen Research Institute, Shenzhen, China
Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China Tongji University, Shanghai, China
*Nanjing Institute of Geography and Limnology, Chinese Academy of Sciences, Nanjing, China
(siqi23, waagi, erxin. @ccnnrectt poreeettppoll....... pollu.hk
Liangchao@niglaan acc ac. en, 2311821@ta 1221@tnggi..eee edg..innddcccn jing-amelios li wiipolyu.edu..le... .k, carter whfcarter@gmai com
uut.
ABSTRACT
Vision-Language Models (VLMs) have made significant progress in explicit instruction-based navigation; however, their ability to interpret implicit human needs (e.g.," am thirsty") dynamic urban environments remains underexplored. This paper introduces CitySeeker, novel benchmark designed to assess VLMs' spatial reasoning and decision decision-making capabilities for exploring embodied urban navigation to address implicit needs. CitySeeker includes 6,440 trajectories across 8 cities, capturing diverse visual characteristics and implicit needs in 7 goal-driven scenarios. Extensive experiments reveal that even top-performing models (e.g., 1-1.225-V--22--nstttte
achieve only 21.1% task completion. We find key bottlenecks in error accumulation long-horizon reasoning, inadequate spatial cognition, and deficient experiential recall. To further analyze them, we investigate series of exploratory strategies Backtracking Mechanisms, Enriching Spatial Cognition, and Memory-BB Memory-Based Retrieval (BCR), inspired by human cognitive mapping's s emphasis on iterative Abservation-reasoniig cycles and adaptive path optimization. Our analysis provides actionable insights for developing VL.Ms with robust spatial intelligence required for tackling "last-mile" navigation challenges.
INTRODUCTION
Explicit Step-by-See Step Instruction
"Go straight and turn right at the intersection. straight until
outdoor your right.
(23
Chengdu
The restaurant front of you."
(23.99)
(19.2%)
Implicit Need Instruction
Transit (7.7%)
Eary
(12.7
Cali-Oreetted
Demand-Drin -Drivee
"Could you please direct me to the restaurant?'
"I'm starting to feel hungry."
(14.2%)
Medium
7.7%
Shanzhen (3 (.3%%
Semantic Preterence
Infrastructure 0%
Shanghai
Latent (23.0
New York (19.2%)
Abstract Demand
(20.4% 4%)
Difficulty
City
Category
Figure I: Navigation Instructions indicating ex- plicit needs (top) and implicit needs (bottom). "Corresoonding authors
Figure 2: The statistics of CitySeeker with 6,440 trajectories in diverse scenarios.

---

Published as a conference paper at ICLR 2026
Vision-Language Models (VLMs), with their advanced vision- grounded reasoning and language generation capabilities, are increasingly being applied to complex tasks like Embodied Urban Navigation (Zhang et al. 2024a). Autonomous embodied navigation in open urban environments is a comerstone for realizing a new generation of intelligent services, leading to a rising demand for VLMS that can guide service robots, drones, or an AR assistant through urban settings. However, the capabilities of VLMs in this domain remain underexplored, as recent advances have focused on agents that follow explicit step-by-step instructions (e.g.. "Proceed straight until the sculpture fountain, turn right, then continue until reaching McDonald's ""). Such approaches, which we describe as explicit needs, rely on pre-constructed navigation directives rather than natural human commands, and face critical limitations in dynamic or novel urban scenarios (Wang et al. 2021 2024b).
Table 1: Overview of Vision-Language Navigation datasets.
Dataset
Instruction Type Instruction
Environment
Source
City Nodes
Avg.Lnngh Avg. Token
Talk Walk Vries
2015
Explicit
786
GridWolld
Rendering
31,5
Room- Ro-Romm Aiinne ot
Explicit
21,567
Indoor
Panoramas
10,800
6.1
29.0
Truchdomn n nen of ai 201o
Explieir
9,326
Outdoor
Sweet View
20/641
35.2
89.6
TalkeNNov (Vienaeg2 et mi 2021
Explicir
10,714
Oundoor
Panoramas and Map
21,233
40.0
68.8
StreetNay Duin
Explicit
644,415
Outdoor
Street View
1,194m
7.13
map2sog (Scurtinnn
Explicit
7.672
Outdoor
OpenStrcetMap
29.641
55.1
Seeker
Implicit
6,110
Outdoor-Dynamie Street View and Map
41,128
18.
11.11
In contrast, real-world human instructions often involve abstract goals that pertain to implicit
needs (Zhou et al. 2024b). These needs are often unannotated on traditional maps or lack the granularity needed for last-mile navigation, and are implicit on multiple aspects: functional (e.g.. when finding a restroom, recognizing the affordance that one is also available in McDonald's s, or inferring that "I'm thirsty" can be resolved by a convenience store or even a public water fountain), spatial (e.g., upon seeing more visually prominent landmark like a shopping complex, inferring that a Starbucks is likely nearby, or realizing that a cinema is often hidden inside a mall), and semantic (e.g.. subjective qualities like "romantic" "upscale" restaurant). Addressing these common needs is a fundamental challenge for goal-oriented navigation, requiring an agent to ground environmental intent through active exploration and spatial reasoning to solve the crucial "last-mile" problem, especially in contexts like city walks (Gao et al. 2024). We illustrate this concept in Figure D
The ability to understand human needs in urban space is fundamental to Embodied Urban Naviga- tion (Liu et al. 2024; Hirose et al. (023). While prior work has explored VLM's ability to interpret human needs, these efforts have been largely confined constrained settings like indoor environments and 3D games ( (Wu et al. [0224a] Hu et al. (225)] Zhang et al.] 2024b). This raises a critical scientific question: Can VLMs develop intrinsic spatial awareness (Yang et al. (024) for Embodied Navigation in open-world urban settings to address implicit human needs? This task introduces unique chal-
lenges: (1) Dynamic Visual Complexity, with diverse and changing road networks and storefronts; (2) Free-Form Instruction Parsing for goals in flexible language; and (3) Long-Horizon Reasoning across extensive distances. The latter is not merely about path length but requires robust, multi-hop reasoning that couples semantic inference with visual grounding. For instance, to address "I need temporary place with Wi-Fi to work," an agent must ground the abstract function of "working." While it may infer POI categories like Cafe or Library, the final decision depends visually grounding location's 's suitability by dynamically searching for real-world cues-such as a storefront's 's ambiance, patrons with laptops, or even a "Free Wi-Fi" sign-that confirm it meets the user's need in real time.
Humans address these challenges using cognitive maps maps-mental mental representations of spatial relation- ships and environmental attributes (Epstein et all. 2017 Tolman 1948). By combining observation with prior knowledge, humans can dynamically update their spatial understanding, infer latent prop- erties, and formulate grounded plans from abstract coals-capabilttiss yet be replicated in VLMs
for outdoor navigation (Clemenson et al. 2021 Farzanfar et al. 2023 Momennejad et al. 2024 Wu et al., 2024b). Inspired by this, we propose CitySeeker, a novel benchmark for autonomous embodied urban navigation. It assesses Implicit-Need-Drinin translating an implicit need into a concrete visual search by using semantic inference to infer possible targets and grounding this understanding in a continuous stream of observations. To systematically probe this ability, its task categories represent a spectrum of varying cognitive difficulty, from direct recognition ("Basic POI") to highly abstract reasoning ("Absttract Demand," "Semantic Preference"). The benchmark is implemented through 6,440 trajectories across 8 globally distributed urban regions with diverse layouts and visual characteristics (T.
Visual Grounding: the process of
Our extensive experiments reveal that existing VLMs generally underperform, exhibiting significant trajectory deviation and deficient spatial cognition. Building on these benchmark findings, we
2

---

Published as a conference paper at ICLR 2026
further investigate advanced strategies that endow the agent with human-like cognitive capabilities. We propose and analyze a triad of exploratory approaches: Backtracking mechanisms mimic self- correction, Spatial Cognition Enrichment mimics mental map building, and Memory-Basee Retrieval mirrors recalling past experiences. These BCR strategies offer a concrete roadmap to elevate VLM spatial intelligence. From a cognitive science perspective, our research is aligned with exciting developments in Al concerning Spatial Mental Models within LLMs (Momennejad et al. 2024 Yang et al. 2024 Wu et al., 2024b), probing the intrinsic spatial intelligence of these models. In summary, our contributions are threefold:
CitySeeker the first turge-scale benchmark for embodied urban navigation that addresses implicit needs across diverse multi-city settings, incorporating real-world visual diversity, long-horizon planning, and unstructured instructions.
We design VLM-based framework and suite of human-inspired cognitive strategies (BCR) that translate implicit needs into multi-step plans through iterative posservation-reasonne cycles.
Through extensive experiments, we surface key bottlenecks in VLM spatial reasoning and crys- tallize them-aan their fixes-into the BCR triad Backtracking. Cognitive-map enrichment, and Retreeval-augmented memory, offering insightful guidance for advancing spatial intelligence.
2 RELATED WORK
Spatial Cognition and Mental Models. The concept of spatial oognition-intermal mental represen- tations of space-originan from cogmitive science and has long been considered fundamental to how humans navigate [Tolman 1948 Epstein et al. 2017). A recent paradigm shift has been spurred by findings that large models can develop emergent "world models" and representations of space and time without supervised training (Gurnee & Tegmark] (023). This has inspired new line of inquiry dedicated to probing the intrinsic spatial intelligence of these models (Yang et al.) 2025| Yin et al.) 2025; Feng et al.) 2025 Manh et al. 2025). Our work provides the first attempt to systematically evaluate the emergent spatial cognition of VLMs through complex urban navigation tasks.
Vision-Language Navigation (VLN). Research in VLN has explored diverse instruction formats, including step-by-step, dialog-basee (Goetting et al. 2024 Zhou et al. (0024)). and goal- or intention- oriented instructions (Wang al.) [2023] Qi al.] 2020al Chiang et al. 2024 Liu al. 2025
However, the majority of outdoor VLN research, such as Touchdown (Chen et al. 2019 have
focused agents following explicit step-by-step directives. This has led to the development of various methods (Xu et al., 2024. [Schumann et al. [024]] Dorbala et al. [2022] Pan et al. 2024 Zhao et al. 2025) -excel at direct text-visual matching but are less equipped for the abstract reasoning
required by implicit goals. Moreover, many solutions are limited by a reliance on static representations
of predefined environment and poor generalizability (Cao et al.) 2024 Elnoor et al. [024 Zhan et al.) 2024), revealing a gap in handling dynamic, real-world urban navigation. Goal-oriented navigation requires agents to use perception and commonsense reasoning to ground abstract concepts. Its form focuses on zero-shot open-world navigation (Mirowski et al.) 2018)maaauda et al.) [2022] /hou et al.,
2024b), which has largely been limited to indoor environments. CitySeeker is the first benchmark to evaluate this goal-oriented reasoning for implicit needs in open- world cities.
3 CITYSEEKER BENCHMARK
The CitySeeker dataset comprises 6,440 route instances and corresponding natural language instruc- tions covering 7 carefully curated categories of everyday human needs. These categories span typical requests from target recognition (e.g., finding nearby facilities or brands), contextual inference (e.g.. inferring that a restroom is likely available McDonald's, (')s) and attribute analysis (e.g., guiding to a bank with accessible facilities), to abstract and subjective reasoning (c.g., finding restaurant suitable for a team gathering). The design these categories is guided by three core criteria: (I) semantic complexity of goals, (2) spatial reasoning requirements, and (3) real-world applicability.
CitySeeker is collected from eight distinct metropolitan areas maans-B Shanghai, Shenzhen, Chengdu, Hong Kong, London, and New York- catttrinn diverse architectural layouts and dy- namic street-level visuals. Overall, the benchmark includes over 41,128 street-view panoramas (from Google or Baidu Maps since 2024) to ensure realistic appearance variations. As shown in Figure 2
3

---

Published as a conference paper at ICLR 2026
Collection
Human-in-the-loop Validation
Street Views
Road Network with POI
WB
Question Template Design
Target Visiallity Verification
Question Generation
Route Dataset
Question: I'n feeling thirsty and would lo Like sorething bo drank.
Tartude" Beaclige"
Tatt porpeerriie Searpiaar
MATCH jlase Poeet (OONTAINS sheviscksPPor
1200999
Supeemoker ker. Food and prek Cate
DetivsPPacis
200 Tork
Topological Graph Construction
Route Generation
Visblla POI Mapping
wtt..
Figure 3: Construction Process of the CitySecker Dataset.
our dataset balances instruction types, urban regions, and navigation difficulty (based on trajectory length). These categories collectively capture broad range and diversity of real-world navigation challenges, with 19.2% involving POI navigation, 20.4% requiring the interpretation of abstract demands, and other major portions dedicated to brand-specific searches (23.4%) and latent POI discovery (23.0%). To facilitate quantitative evaluation, we further sampled 1,257 route instances as the final benchmark test set, balancing coverage across all metropolitan areas and categories.
3.1 BENCHMARK CONSTRUCTION
We employ a demand-driven pipeline to generate high-quality navigation routes at scale, as summa- rized in Figure By integrating panoramic data, POI filtering, and graph-based distance queries, it yields
routes spanning broad spectrum of real-life urban navigation needs (see Appendix B for details).
Data Collection and Topology Construction. We gather street-level panoramas from both Google and Baidu Street View, covering diverse urban areas (e.g.. Western cities, Hong Kong. Mainland China). To discretize each city's road network, we place node every 20 meters and capture its panoramic imagery with metadata (e.g., latirude/longitude, heading, navigable headings). These nodes and edges represent navigable connections annotated with azimuths and distances, forming graph-based structure stored in Neo4j. This layout facilitates spatial queries and ensures consistent global connectivity. Each collected Points of Interest (POI) is then linked to any vantage node within a 50-meter radius, reflecting the idea of a visible or discoverable place from that viewpoint. This step updates the knowledge graph with triplets of the form (Node) -has (VisiblePOI).
Question Design and Generation. To cover common daily needs, we first identify a set of high- frequency POI categories. These sourced primarily from map providers (e.g.. "restaurant," "coffee shop")) and are supplemented with other visually rich POIs identified from street-view images and manually verified, capturing targets often unannotated on maps. This enables us to design specific templates for different query types. Each question type is manually associated with one or more POI categories; for instance, "T' "I'm feeling thirsty" may map to beverage shops, water fountains, or cafés.
Route Generation and Validation. We generate navigation paths by selecting start and end nodes based on the intended POI category. We determine the starting nodes based on constraints to ensure that the shortest path falls within a controllable minimum radius (c.g., ensuring no other similar POIs lie within an overly small radius), thereby guaranteeing a meaningful navigation distance. The shortest paths between start and target nodes (containing the target POI) are then calculated using shortest-path algorithm (e.g., A*). This process yields effective routes ranging from to 25 steps. Finally, we manually verify each route by confirming the target POI is indeed visible or indirectly visible at the terminal node. To further empirically validate the rationality of our Need-to- to-Pot mappings and mitigate potential designer bias, we conducted cross-cultural human consensus survey (N = 120). which demonstrated a high degree of agreement (83.39%) with our ground truth categories (see Appendix for details). Any route failing this check is refined or discarded.
4

---

Published as a conference paper at ICLR 2026
Implicit Instruction
Muttipla Perspectives
Observation Reasoning
"I need help finding a nearby place with Wi-Fi work."
The image shows street with a Starbucks, some storefronts and trees. Perspective Observation: "This shows a street with Starbucks, likely offer Wi-Fi and good place to wark", "B": shows a bustling street with vendors on the sidewalk. "C" "D": ..)
Overall Observation:
No
Street Panorama
Thoughts: chose Perspective because shows a Starbucks, which likely to offer Wi-Fl...
Stop
Yes
Step 5
VLMs
Time Stup
Visian-Languea Navigation on Graph
Street Panorama
Topological Graph
Choson Perspective
Action: (choose perspective Action Direction: step forward
Score: 0.85
Environment
Figure 4: The CitySeeker Implicit-Need Driven Embodied Urban Navigation Framework.
4 EVALUATION ON CITYSEEKER
4.1 OVERVIEW
Task Formulation. We formulate the VLN task within a navigation graph G = (V,E), where V represents the set of georeferenced nodes and & denotes edges. At time step I, the agent occupies a node Up E V. The agent receives a natural language instruction W and an observation set O2 {or.1, 0r,2 .... )r,n). where n is the number of perspective views at Ut. The agent maintains a state s, = (Ut, O,) representing current environmental context. The agent operates via a policy тe that sequentially generates reasoning rationale followed by an action: (L,, (e, Gt) Tee (W, st). Here, b is the reasoning rationale, and 0₂ E Ar = {1,.... n is the index of the selected perspective view with confidence 6 E [0, 1]. The environment transitions to the next node u1+1 based on the dynamics T(v:, dz). A reasoning process is recorded as: 7 {W, (81, 1 ai, a ........ (8T, ΦT, ar, eT) where the process terminates once the stop condition is met.
Navigation Framework. Figure&Jillustratee the navigation pipeline. At each time step, the panoramic image at the current viewpoint is subdivided into multiple perspective views, each corresponding to a feasible heading. Guided by ReAct-style reasoning procedure (Yao et all] (002)). the VLM processes the current observation (Observe), infers a navigation intention (Think), selects one perspective view to move toward (Act), and finally outputs confidence score reflection (Reflect). This process iterates until the agent deems it has reached the goal or surpasses a maximum step limit (35 steps). To isolate the model's core spatial reasoning ability, intentionally keep each step independent: the agent does not maintain persistent memory or feed previous internal states into subsequent decisions.
4.2 EVALUATION SETUP
Benchmark Models. We comprehensively evaluate 27 multi-image capable VLMSs across diverse model families, encompassing various parameter scales and architecture. For proprietary models, we consider GPT-4o, GPT-4o-Mini, o4-mini, Gemini- 1.5-Pro and Gemini-2.5-Pro (Team) 2024).
For open-source models, we evaluate models from Qwen2 VL series (7/72B Instruct) (Wang et al.
2024a), Qwen2.5-VL series (7/32/72B Instruct) (Bai al. 2025). Intern VL2.5 (8/26/38B) (Chen al. 2024), Intern VL3 (8/ (////////BB)../ Llama-3.2 3.2 (1 1/90B Vision variants) (Dubey et al. 2024), Llama-4 (nsttttuttt,,),, LA derivatives
(Scout-17BB 1677B---E--aa 1aso-///nnee////aaeiieee/////
LLaVA-One Vision Vision-Qwen2-77) (Liu et al. 2023). MiniCPM derivatives (V-2.6/o-2.66 (Yao et al. wtttonnnntttuttt-tttttootoo
(Abdin et al. 2024 Microsoft 2025) and MiniMax-01 (Li et al. 2025). All evaluations are conducted under zero-shot
2024, Yu et al.] 2024), Phi series
settings using unified prompts (see Appendix C.I for full details).
Metrie Design. We adopt comprehensive evaluation protocol that extends standard VLN metrics Qii et al. 2020b) to assess task success and navigation efficiency. Our evaluation includes: Task Completion (TC), measured with metrics of varying granularity: (1) TC-Exact (TCE) requires strict, single-node endpoint match. To account for the fact that a target is often visible from multiple nearby viewpoints, (2) TC-Pioxiiyty (TCP) grants success within geodesic threshold of <50m to address
5

---

Published as a conference paper at ICLR 2026
Table 2: The performance of CitySeeker Framework. For Subeategory evaluations, the TCP score is reported. Top performers are highlighted in bold. while secondary leaders are underlined. For details on the AS metric and more comprehensive evaluation of the models, please refer to Appendix C.3
Overall
Subcutegories
Models
Basic
Brand Specific
Transit
Latent
Abstract Demand
Inclusive
Semantic
GPT-4o
2.4
18.3
6.8
13.3
125.4
136.9
18.9
21.1
19.6
9.7
18.9
14.9
26.0
GPT-4o-mini
202.0
325.2
12.3
21.1
11.8
04-mini
2.6
17.9
6.8
11.6
130.1
156.3
20.2
19.9
12.5
13.1
16.7
9.0
24.0
Gemini-2.5-pro
121.2
20.4
13.5
VL2.5B
1.0
14.6
4.4
118.5
140.4
15.7
14.3
1.8
14.8
18.4
10.4
10.6
In5emV1.2.-2001
1.6
15.3
3.7
7.3
109.5
106.0
2014
12.4
8.9
10.8
17.1
7.5
8.7
122cm8L2..-18B
2.2
18.1
11.2
136.6
169.2
17.6
24.2
14.3
13.1
22.8
10.4
16.4
15.8
118.3
144.5
17.2
15.5
16.7
15,4
IntemVL.1-148
1.7
15.1
6.4
8.5
136.2
170.8
13.8
20.5
17.9
8.0
19.3
10.5
17.3
IntemVL3.38B
128.3
18,9
25.5
23.3
21,2
Qwea2-VL-22 VL-7B
0.7
11.1
1.7
5.2
111.4
114.4
11.4
9.9
14.3
4.6
13.2
13.4
15.4
Qwen2 VL-72B
1.0
113.0
89.5
14.8
8.7
14.5
15.4
Qwen2.5-VL.
0.5
15.8
4.6
119.0
151.8
20.2
17.4
12.5
10.2
13.6
7.5
15,4
Qwen2,5-V1. VVL-325
2.6
21.1
6.2
12.7
122.6
147.0
22.2
30.4
10.7
14.2
24.1
9.0
20.2
Qwea2.5-VL-72B
250,2
15.8
Llama3.2-908
0.9
12.5
3.7
7.3
124.5
123.1
13.6
12.4
1.8
9:7
14.0
10.4
16.4
Llama-4 Scout-
1aama-4-ccott111 17B
1.8
14.1
7.0
6.7
145.0
211.4
12.3
18.6
14.3
8.5
17,5
17.9
14.4
Mlave--Maverickk1125
107.1
110.5
12.7
48
MiniMax 01
1.5
13.6
6.8
8.8
172.2
236.6
15.5
16.2
8.9
6.8
14.9
11.9
13.5
MiniCPM-V-2.6
0.9
11.7
122.2
152.2
12.5
10.6
19.6
13.2
8.7
oi.
1.4
15.5
130.1
176.0
14.4
22.4
10.7
12.5
18.9
11.9
12.5
Phi-4-Multindaal
0.6
9.2
101.
58.1
14.0
6.2
2.8
10.5
2.9
IJava- Jars-JJamn--aa
86.9
14.8
8.7
8.8
1.7
Llava- Qwen2-BB
0.3
6.9
98.1
49.8
12.5
2.5
3.6
6.6
1.0
Human Random Choice
3.7
30.1
13.5
21.2
143.5
178.6
31.8
36.5
34,9
19.7
31.5
16.7
29.8
16.6
14.9
13.5
Forward Dircction
0.2
7.2
100.8
99.3
13.3
3.1
3.6
6.6
1.9
this spatial ambiguity; (3) TC-Category (TCC) evaluates whether the final destination belongs to the same category as the intended target, acknowledging practical flexibility in location-based tasks (e.g., reaching any restaurant rather than the closest one); Path Quality, assessed via (4) Normalized Dynamic Time Warping (nDTW) (Xu et al. 2024) quantify trajectory alignment with ground truth, (5) Success weighted by Path Length (SPL) to rigorously measure navigation efficiency by balancing success rates against trajectory length, and (6) Average Steps (AS) for decision efficiency; Distance-Basd Metrics, (7) Shortest-Path Distance (SPD) for straight-line distance to the goal.
To comprehensively assess model capabilities, we introduce three evaluation baselines. We establish (1) Human Baseline using interactive platform where 10 participants of diverse backgrounds performed the navigation tasks. We also include a (2) Random Choice Baseline, which selects random direction at each step, and (3) Forward Direction Baseline, which always chooses the forward direction as a simple heuristic.
Reference Baselines.
MAIN RESULTS AND ANALYSIS
5.1 OVERALL PERFORMANCE
Table presents the overall results. In general, the models exhibit relatively low success rates, particularly under the stricter criterion TCE. Larger models (c.g.. Qwen2.5-VL--2B. GPT-4o, Gemini- 2.5-Pro) perform slightly better-likke due to stronger internal representations but the gains over smaller models remain modest. Notably, open-source models like the Qwen2.5-VE and Intern series demonstrate competitive performance. This superiority likely stems from specific architectural and training alignments with CitySecker's demands: Qwen2.5-VL. benefits from spatial-aware SFT and efficient regguureooltion processing crucial for identifying street-level details, while InternVL3 leverages native multimodal pre-training and Mixed Preference Optimization (MPO) to enhance the complex cross cross-modal reasoning required for implicit needs. However, model consistently excels, with some even underperforming random baselines. Ablation study further revealed that providing agents with global 2D map surprisingly degraded task completion, underscoring the challenge of

---

Published as a conference paper at ICLR 2026
fusing map-based information with visual grounding (see Appendix Dj. Human participants achieved the best overall performance, outperforming all models on both TCP and TCC. They exhibit a clear advantage across diverse task dimensions, especially in tasks requiring urban commonsense like "Transit Hub" navigation (a TCP of 34.9% for humans versus 10.7% for the best VLM Qwen2.5-VI. 32B) and "Basic POI" finding (31.8% vs. 22.2%
Abstract Demand
ComplttianProxiiily York)
Task Completion-Froximig (Bajjng)
Brant
GPT-4
CPT-45--
20.0
Mobel
Indlusive Intresructurr
Genriti 2.5
15.0
Intammt.3-000
12.5
Bu00ers
7.5
10
Liara 37B
Nireess D.
25
Lotent
Recenta co dven Sconn 11p
ceenti 2. SS
preturnt pae thd the the 110
Clessua 1o clesa Pt disttoons
clens Jay
Qwen2 5-1-78
Senmartic Profurance
Figure 5: Radar Chart: TCP performance of different models across various subcategories. Bar Chart:
TCP performance of models in different cities (left: New York, right: Beijing).
Trajectory lentmmpp York (GPT-40]
Trajectory Heatmap Beijng (GPT-4c)
Rolaton between Predicted Stops nD TW
Regresioe
750
1750
1900
Lompuhado Trijettry Syon:
10.35
200
Trgedtory (pibe 601
Tegeebey Lpte
Trijdda
Trglectry
Stat
Bidt
Pool Pach
Labaac
Proc. Tan
CLA pac
250
Falled
Long Bade
Lengaace
Lenguass
Lengu.ca
Longiids
Predicted Staps
Figure 6: Heat Map: GPT--ooss navigation trajectory distribution in New York (left) and Beijing (right). Scatter Plot: Relationship between model-generatee trajectory steps and nDTW.
Task Category Analysis. Breaking down performance by query category reveals clear divide between tasks requiring direct recognition versus those demanding deeper inference, as shown in Table 2 and radar chart of Figure 5 Models performed best on Brand-Specific navigation, where recogmizable brand names (e.g., Starbucks) serve as strong lexical and visual anchors. In sharp contrast, one of the most challenging categories was Latent POI, which requires agents to reason about indirectly observable targets that are not explicitly signed (e.g., inferring a restroom is inside McDonald's s). This finding highlights a key VLM limitation: while adept at direct recognition, they falter when tasks require nuanced, commonsense inference about the environment.
Variation Across Cities. Performance also varies across different metropolitan areas. As shown on bar chart of Figure interestingly, GPT-4o variants perform poorly in Beijing but achieve highest scores in New York this may reflect biases in the training data or the more grid-like street layouts in the United States. To test for linguistic bias, we conducted a cross-lingual experiment, which revealed that this performance gap is not primarily driven by language factors (see Appendix C.4 for details).
5.2 IN-DEPTH ANALYSIS
As shown in Figure 6 model performance tends to degrade with increased route length. When the number of steps is fewer than 20, nDTW values remain relatively small and correct trajectories are more common. However, around 35 steps, nDTW metrics become highly scattered, indicating that longer horizons require robust sequential reasoning- an ability current VLMs often lack, as their emrrr accumulate without being integrated into a coherent spatial memory.
7

---

Published as a conference paper at ICLR 2026
B. Backtracking Mechanisms
C. Spatial Cognition Enrichment
R. Memory- Baseed Retrieval
Hilstorical Trajectory Sampling
Cognitive Graph
Thoughts.... Contidance Score: U.8
VLM
BPi-4b
Ralantes Poaltion
VLM
0
Ao Hea
Start
Figure 7: Overview of the BCR Approach Backtracking, Spatial Cognition, and Memory- Hemory-Based Retrieval to enhance VLM performance in urban VLN tasks.
Table 3: Performance of different models on TCP and nDTW under three strategies. The best results are highlighted in bold.
Model
Baseline
B1
B2
B3
C1
C2
RI
R2
R3
TCP nDTW
TCP aDTW TCP aDTW TCP
nDTW TCP
nDTW
TCP
nDTW
TCP
aDTW TCP
nDTW TCP
GPT-4o-Mini
12.5
289.5
277.9
18.2 258.3
15.1
239.7
136.6
17.9
19.4
176.9
Vntem
15.0 142.9
16.7 139.9
18.5
137.5
17.2
138.6
17.9
145.2
16.4
197.6
18.6
148.9
16.9
128.4
17.9 134.8
MiniCPM-V-2.6
11.3 159.0
9.1 120.5
13.1
153.1
13.0
137.2
11.6
123.
85
144,4
14.6
143.3
14.8
146.7
15.7 111.5
Dwen2.5-VL-7B-nnnnne Instrrtttt
12.5 122.2
14.0 140.5
14.4
156.7
15.3
1418
15.3
142.5
12.3
135.8
16.1
156.4
15.9
441.4
16.2 141.8
Qwen2.5. 31. Instruct
19,9 167.7
21.9
164.5
22.2
154.6
23.1
159.4
21.2
153.5
19.6
162.1
26.9
173.2
25.9
172.8
25.4
152.8
Trajectory Patterns. In Figure 6 two prominent error modes emerge: (1) Trajectory Deviation. This arises from compounded errors at each sequential decision point, a problem exacerbated by sparse or ambiguous instructions. (2) Oscillatory Detours. Some open-source models exceed the optimal path length by 40-60% (e.g.. trajectories #47 and #153 in New York), likely attributable to fragmented context handling and incomplete global awareness. We also observe that path efficiency correlates weakly with TC: when uncertain, some models veer off course or backtrack repeatedly, leading to wasted steps. Notably, most models demonstrate looping behavior-visiting the same node multiple times (e.g., trajectory #63 in New York and #32 in local decision- decision-making Interestingly, others (e.g.. Blava-Qwen2-11 take relatively concise routes yet still show low accuracy, implying a lack of deeper spatial understanding despite fewer detours.
difficulties with
EXPLORATORY APPROACHES TO EMBODIED URBAN NAVIGATION
To explore strategies for enhancing VLM performance, we propose a three-pronged approach, BCR, involving Backtracking Mechanisms, Spatial Cognition Enrichment, and Memory-Based Retrieval (Figure 7 These methods aim to mitigate cumulative navigation errors, boost global spatial awareness, and enable memory-informed decision-making, We denote these three method series as B, C, and R respectively in the following content. We conducted initial experiments on a mini-size subset of 650 samples; for full technical details, including an analysis of combined BCR strategies, please refer to Appendix
6.1 BACKTRACKING MECHANISMS
We introduce backtracking to correct navigational errors. (B1) Basic Backtracking triggered when the agent's 's internal confidence, averaged over a sliding window of h steps, falls below predefined threshold The agent then reverts to the last trusted node. This mechanism is superviud and does not require external feedback. (B2) -eep-R Backtracking replaces subjective confidence with objective progress metrie: the topological distance to the goal, d, Backtracking initiated this distance increases monotonically for l consecutive steps, i.e., when A k-1 (d+-; > dz-(141) holds. This corrects navigational drift away from the target. (B3) Human-Guided Backtracking enhances B1 by providing directional hint after reverting. This hint guides the agent toward the optimal action a" arg min E(d++1aa = a), which minimizes the expected future distance the
aeA.
target, thereby realigning the agent's trajectory with the shortest path to the goal.
8

---

Published as a conference paper at ICLR 2026
Empirical Performance. As shown in Table 3 B1 generally improves TCP across models, in- dicating that confidence-basee backtracking stabilizes navigation for capable VLMs. However, underperforms smaller models like MiniCPM-V-2.6 (TCP drops from 11.3% to 9.1%) suggesting insufficient spatial reasoning for effective self-assessment. B2 and B3 are more universally effective, consistently improving TCP across all models. For instance, B3 boosts GPTn-o-Mini's TCP 18.2% while significantly improving its path efficiency (nDTW drops from 337.1 to 258.3). These results highlight that while simple self-correction benefits capable models, external signals (like progress rewards or timely hints, in B2 and B3) offer more robust improvements.
6.2 SPATIAL COGNITION ENRICHMENT
To improve the agent's environmental awareness, we generate enriched spatial cues by using GPT-4.1 to synthesize successful and erroneous trajectories from various VLMs (e.g., GPT-4o, Gemini-2.5- pro). This synthesized knowledge is then injected into the agent's prompt in one of two formats. (C1) Topology Cognitive Graph provides a structured graph where nodes represent locations and edges represent actionable transitions. This format forces the agent to ground its decisions in explicit environmental connectivity, discouraging exploration of invalid paths. (C2) Relative Position Map offers more intuitive spatial context, describing relationships between locations with directional cues (c.g., "left", "slightly right") and estimated distances. While lacking explicit connectivity, this method allows the agent construct flexible mental map based on relative positional relationships.
Empirical Performance. Table 3 illustrates that C1 improves TCP across models by grounding decisions in topological structure. For instance, GPT-4o-Mini achieves increase from 12.5% to 17.2%. The impact of C2 is more mixed; while it can improve path efficiency (e.g.. for GPT-4o-Mini, nDTW drops to 239.7), 7), sometimes slightly reduces task completion (e.g., for MiniCPM-V-2.6. TCP drops from 11.3% to 8.5%). Overall, CI appears more reliable for improving task success, whereas C2 supports more flexible and exploratory approach but sacrifices some success rate.
6.3 MEMORY-BASED RETRIEVAL
To overcome fragmented decision-makine making, we implement graph-based memory module, enabling agents to retrieve or append past rationales and actions. (R1) Topology-basee Retrieval aggregates multi-round node and edge metadata based on graph connectivity. At each step, it retrieves local subgraph of h-hop neighbors, accessing metadata like node visitation counts, previous decisions, edge transition success rates and confidence scores. This helps avoid repeating past mistakes and promotes the reuse of successful paths. (R2) Spatial-based Retrieval complements this by retrieving a subgraph of nodes and relationships within a fixed Euclidean radius, which emphasizes geographic proximity. (R3) Historical Trajectory Lookup introduces short-term memory by appending recent intra-round navigation history into VLM's context. This includes spatial data, action choices, and prior decision rationale over sliding window. Unlike RI and R2, R3 does not rely database traversal, making a lightweight method to stabilize reasoning within a single episode.
Empirical Performance. T reveals that the R-series strategies the most powerful overall. For GPT-4o-Mini, R3 yields the highest TCP of all strategies ),,,4%) while RI achieves the best path efficiency with dramatic nDTW reduction 136.6. This demonstrates that different memory- based approaches can simultancously enhance both task completion and navigation efficiency. Across all models, R-series strategies consistently produce some of the highest TCP scores, with R1 pushing the top-performing Qwen2.5-VV ---.e..5-V---2----1 --nt-uue-- to an impressive 26.9% TCP. This confirms that memory-aware navigation is crucial for improving both the reliability and effectiveness of the agent.
Trade-off Analysis and Application Scenarios. The B-Series strategies (especially B2 and B3) reliably improve task completion, making them suitable when success is prioritized. The C-Series is applicable when external spatial information can be provided; CI generally favors accuracy, while C2 may trade success for better path efficiency. Overall, the R-Series is the most robust approach for long-horize navigation, consistently yielding the best task completion rates. Lightweight models like Intern VL3-8B are suitable for time-sensitive tasks, while heavier models like Owen2.55VL-322. Instruct, augmented with R-series strategies, ideal for precision-critical applications.
9

---

Published as a conference paper at ICLR 2026
7 CONCLUSION
CitySeeker establishes standard for embodied urban navigation by systematically evaluating how well VLMs tackle implicit human needs diverse, real-world cityscapes. Our extensive experiments uncover that today's models frequently struggle with the complexities of this task, often deviating from optimal routes due to limited spatial reasoning. To address this, we introduced a spatial
-umann-nspired cognitive framework and outlined three insightful strategies- barregies--bckkrackkkki demonstrably improve performance.
cognition enrichment, and memory-based retrieval-wiin These findings offer a clear pathway toward developing more robust, human-like agents capable of solving the crucial "last-mile" problem in real-world navigation.
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (Grant No. U23B2057), the STIC Shenzhen Natural Science Foundation (No. JCY1202506041913222 191329039), a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. PolyU/252008211, the Shanghai Pilot Program for Basic Research (Grant No. 22TQ1400300), and the Innovation and Technology Fund (Project No. PRP/047/22FX). We also sincerely thank Dr. Yushi Li for insightful comments and revisions, which significantly improved this paper.
ETHICS STATEMENT
The study involved human participants for the establishment of a human baseline. All participants were fairly compensated for their time in line with our institution's 's standard rates for research assistance, and the experimental procedure was clearly explained to them. The street-view imagery used in our experiments was accessed via publicly available APIs (Google Maps and Baidu Maps). To strictly adhere to the API Terms of Service, we do not distribute any raw street-view images or offline copies. Instead, researchers can access the visual data using our provided scripts, which query the official APIs using the released trajectory metadata (e.g., Panorama IDs and coordinates). This approach ensures full compliance while maintaining the scientific reproducibility of the benchmark.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work while complying with data usage policies, we have made metadata, code, and supplementary materials publicly available. The CitySeeker benchmark release includes: (1) the complete trajectory graph data (nodes, edges, and Panorama IDs), and (2) the full implementation code for our evaluation framework and BCR exploratory strategies. The scripts enable re-fetching the necessary street-view imagery directly from the public APIs. This methodology aligns with established practices in the field (e.g.. Touchdown (Chen al. 2019)) to ensure reproducibility without distributing copyrighted images. Furthermore, we have verified that reproducing test set via these seripts incurs no financial cost under standard API usage tiers. All resources can be found at anonymous repository linked in Appendix [A] The main paper and appendix provide comprehensive details on our experimental setup (Appendix C). the exact prompts used for all VLM evaluations (Appendix C.2), and the specifics of our evaluation metrics (Section 4.2).
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint 2xXx:24444.1219, 2024.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and lan--uuuegee navigation: Interpreting visually- grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 3684-3683, 2018
10

---

Published as a conference paper at ICLR 2026
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li. Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar arXiv:2502. 139.2333........ 2025.
Yihan Cao, Jiazhao Zhang, Zhinan Yu, Shuzhen Liu, Zheng Qin, Qin Zou, Bo Du, and Kai Xu. Cognav: Cognitive process modeling for object goal navigation with llms. arXiv preprini arXiv:2412. 10439, 2024.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12538-12547, 2019.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024.
Hao-Tien Lewis Chiang, Zhuo Xu, Zipeng Fu, Mithun George Jacob, Tingman Zhang, Tsang- Wei Edward Lee, Wenhao Yu, Connor Schenck, David Rendleman, Dhruv Shah, al. Mobility vla: Multimodal instruction navigation with cong-context vlms and topological graphs. arXiv preprint arXiv:2407.07775, 2024.
Gregory D Clemenson, Antonella Maselli, Alexander J Fiannaca, Amos Miller, and Mar Gonzalez- Franco. Rethinking gps navigation: creating cognitive maps through auditory clues. Scientific reports, 11(1):7764, 2021.
Harm De Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. Talk the walk: Navigating new york city through grounded dialogue. arXiv preprint arXiv: 1807.83367 2018.
Vishnu Sashank Dorbala, Gunnar Sigurdsson, Robinson Piramuthu, Jesse Thomason, and Gaurav Sukhatme. Clip-nav: Using clip for zero-shot vision-and-language navigation. arXiv preprint arXiv:2211.16649 2022.
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Mohamed Elnoor, Kasun Weerakoon, Gershom Seneviratne, Ruiqi Xian, Tianrui Guan, Mohamed Khalid M Jaffar, Vignesh Rajagopal, and Dinesh Manocha. Robot navigation using physically grounded vision-language models in outdoor environments. arXiv preprint 24Xhv:2409.2444 2024.
Russell A Epstein, Eva Zita Patai, Joshua B Julian, and Hugo J Spiers. The cognitive map in humans: spatial navigation and beyond. Nature neuroscience, 20(1
2017.
Delaram Farzanfar, Hugo J Spiers, Morris Moscovitch, and R Shayna Rosenbaum. From cognitive maps spatial schemas. Nature Reviews Neuroscience, 24(2):63-79, 2023.
Jie Feng, Tianhui Liu, Yuwei Du, Siqi Guo, Yuming Lin, and Yong Li. Citygpt: Empowering urban spatial cognition of large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 591 591-602, 2025.
Peng Gao, Peng Wang, Feng Gao, Fei Wang, and Ruyue Yuan. Vision-language navigation with embodied intelligence: A survey. arXiv preprint arXiv:2402.14304 2024.
Dylan Goetting, Himanshu Gaurav Singh, and Antonio Loquercio. End-to-end navigation with vision language models: Transforming spatial reasoning into question-answering arXiv preprint arXiv:24/ 100755, 2024.
Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint 1rXiv:2310.022277 2023.
11

---

Published as a conference paper at ICLR 2026
Noriaki Hirose, Dhruv Shah, Ajay Sridhar, and Sergey Levine. Sacson: Scalable autonomous control for social navigation. IEEE Robotics and Automation Letters, 2023.
Zhe Hu, Jing Li, Zhongzhu Pu, Hou Pong Chan, and Yu Yin. Praxis-vim: Vision-grounded decision making via text-driven reinforcement learning. arXiv preprint arXiv:2503. 2025.
Gaurav Jain, Basel Hindi, Zihao Zhang, Koushik Srinivasula, Mingyu Xie, Mahshid Ghasemi, Daniel Weiner, Sophie Ana Paris, Xin Yi Therese Xu, Michael Malcolm, et al. Streetnav: Leveraging street cameras to support precise outdoor navigation for blind pedestrians. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pp. 1-21, 2024.
Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.083/33 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-3499 2023.
Xinhao Liu, Jintong Li, Yichen Jiang, Niranjan Sujay, Zhicheng Yang, Juexiao Zhang, John Abanes, Jing Zhang, and Chen Feng. Citywalker: Learning embodied urban navigation from web-scale videos. arXiv preprint arXiv:2411.17822 2024.
Xinhao Liu, Jintong Li, Yicheng Jiang, Niranjan Sujay, Zhicheng Yang, Juexiao Zhang, John Abanes, Jing Zhang, and Chen Feng. Citywalker: Learning embodied urban navigation from web-scale videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 6875-6885 2025.
Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. Zson: Zero- shot object-goal navigation using multimodal goal embeddings. Advances in Neural Information Processing Systems, 55:32340-32333 2022.
Bui Duc Manh, Soumyaratna Debnath, Zetong Zhang. Shriram Damodaran, Arvind Kumar, Yucyi Zhang, Lu Mi, Erik Cambria, and Lin Wang. Mind meets space: Rethinking agentic spatial intelligence from a neuroscience-inspree perspective. arXiv preprint arXiv:2509.09154 2025.
Microsoft. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras, arXiv preprint arXiv:2503.0/743 2025.
Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, et al. Learning to navigate in cities without map. Advances in neural information processing systems, 31, 2018
Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson. Evaluating cognitive maps and planning in large language models with cogeval. Advances in Neural Information Processing Systems, 36, 2024.
Bowen Pan, Rameswar Panda, Sou Young Jin, Rogerio Feris, Aude Oliva, Phillip Isola, and Yoon Kim. Langnav: Language as perceptual representation for navigation. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 950-974, 2024.
Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
9982-9991, 2020a.
Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang. William Yang Wang. Chunhua Shen, and Anton den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
9982-9991, 2020b.
Raphael Schumann and Stefan Riezler. Generating landmark navigation instructions from maps as a graph-to-text problem. arXiv preprint arXiv:20/2. 15329, 2020.
12

---

Published as a conference paper at ICLR 2026
Raphael Schumann, Wanrong Zhu, Weixi Feng. Tsu-Jui Fu, Stefan Riezler, and William Yang Wang. Velma: Verbalization embodiment of Ilm agents for vision and language navigation in street view. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 18924-18933, 2024.
Gemini Team. Gemini: A family of highly capable multimodal models, 2024.
Edward C Tolman. Cognitive maps in rats and men. Psychological review, 55(4):189, 1948.
Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. Talk2nav: Long-range vision-and- language navigation with dual attention and spatial memory. International Journal of Computer Vision, 12:2246-266, 2021.
Hanqing Wang, Wenguan Wang. Wei Liang. Caiming Xiong, and Jianbing Shen. Structured scene memory for vision- languge navigation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pullern Recognition, pp. 8455-8464, 2021.
Hongcheng Wang, Andy Guan Hong Chen, Xiaoqi Li, Mingdong Wu, and Hao Dong. Find what you want: Learning cemaand-onditioned object attribute space for demand-driven navigation. Advances in Neural Information Processing Systems, 36: 16333-166666, 2023.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- laagugge model's ' perception of the world at any resolution. arXiv preprint arXiv:2409.12 1409.12191, 2024a.
Siqi Wang, Chao Liang, Yunfan Gao, Yang Liu, Jing Li, and Haofen Wang. Decoding urban industrial complexity: Enhancing knowledge-driven insights via industryscopegpt. MM '24, pp. 4757-4765, New York, NY, USA, 2024b. Association for Computing Machinery. ISBN 9798400706868. doi: org/10. 1111/3164441.1.11 117447..6661171 3681703
10.1145/3664647.3666 666. 1705.
URL WRLLhttp:://dd
Wansen Wu, Tao Chang, Xinmeng Li, Quanjun Yin, and Yue Hu. Vision-language navigation: survey and taxonomy. Neural Computing and Applications, 36(7):3291-3311 2024a.
Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia. Li Dong, Lei Cui, and Furu Wei. Mind's eye of mmss: Visualization-o--thouuu elicits spatial reasoning in large language models. In The eighth Annual Conference on Neural Information Processing Systems, 2024b.
Yunzhe Xu, Yiyuan Pan, Zhe Liu, and Hesheng Wang. Flame: Learning to navigate with multimodal Ilm in urban environments. arXiv preprint arXiv:2408. 1051, 2024.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprins arXiv:2412.14171 2024.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10632-10643, 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoming and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 01800, 2024.
Baiqiao Yin, Qineng Wang, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, et al. Spatial mental modeling from limited views. arXiv preprint arXiv:2506.21455 2025.
Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao, Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang, Taiwen He, Xiaocheng Feng, Jun Song, Bo Zheng, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Rlaif-v: Open-source ai feedback leads to super gpt-4v trustworthiness. arXiv preprint ar arXiv:2405.1 17220, 2024.
13

---

Published as a conference paper at ICLR 2026
Zhaohuan Zhan, Lisha Yu, Sijie Yu, and Guang Tan. Mc-gpt: Empowering vision-and aasion-andlannguage navigation with memory map and reasoning chains. arXiv preprint arXiv:2405. 10620, 2024.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions Puttern Analysis and Machine Intelligence, 2024a.
Yue Zhang, Ziqiao Ma, Jialu Li, Yanyuan Qiao, Zun Wang, Joyce Chai, Qi Wu, Mohit Bansal, and Parisa Kordjamshidi. Vision-and-languaal navigation today and tomorrow: A survey in the era of foundation models. arXiv preprint arXiv:2407.0m 07035, 2024b.
Yi Zhao, Siqi Wang, and Jing Li. Laf-grpo: In-situ navigation instruction generation for the visually impaired via grpo with im-as-followed reward. arXiv preprint arXiv:2506.04077 043000000077 2025.
Gengze Zhou, Yicong Hong. Zun Wang, Xin Eric Wang, and Qi Wu. Navgpt-2: Unleashing navigational reasoning capability for large vision-language models. In European Conference on Computer Vision, pp. 260-278. Springer, 2024a.
Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 7641-7644 2024b.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A. Data and Code Availability
B. Benchmark Details
B.I Implicit Needs Instruction Generation Process
B.2 Instruction Categories and Examples
C. Experimental Details
C.1 Implementation Details
C.2 Experiment Prompts
Evaluation Results Details
C.4 Cross-Lingual Experiment Linguistic Bias
D. Ablation Study: Impact Global Map
E. BCR Studies Details
Backtracking Mechanisms
E.2 Spatial Cognition Enrichment
E.3 Memory-Bane Based Retrieval
E,4 Performance of Combined BCR Strategies
F. VLM Generation Examples
G. Error Analysis
Limitations and Future Work
LLM Usage Statement
A DATA AND CODE AVAILABILITY
To promote community engagement and ensure reproducibilityy the CitySeeker dataset along with the full implementation code are publicly available at:
https: / https://github.co wwt.:/oo/aaaaooa wttp:///woooroooooooooeeeeeeeeea
B BENCHMARK DETAILS
The CitySecker dataset comprises 6,440 trajectories across 8 diverse urban regions, paired with navigation instructions addressing implicit human needs. To ensure strict adherence to image licensing and usage rights (e.g., Google Maps Platform Terms of Service), our public release strategy dissociates raw visual data from trajectory metadata. Specifically, we release the complete trajectory graphs (including node coordinates and Panorama IDs) and navigation instructions. Rather than distributing offline copies copyrighted street-view imagery, we provide automated scripts that allow researchers to re-fetch the necessary visual data directly from the official APIs. This methodology ensures full compliance with commerclll,,,, research-only usage policies while maintaining the benchmark's reproducibility. Route-may mape visualizations are generated from OpenStreet data (ODbL) and shared under compatible terms.
B.1 IMPLICIT NEEDS INSTRUCTION GENERATION PROCESS
Each instruction type is manually associated with one or more Point of Interest (POI) categories. For Basic POI Navigation, Brand-Specific Navigation, and Transportation Hub Navigation categories, the corresponding POI types are directly indicated in the questions; For Inclusive Infras- tructure Navigation and Semantic Preference Navigation, we leverage POI metadata descriptions provided by map vendors; For Latent POI Navigation and Abstract Demand Navigation questions, we manually define the mapping between the questions and POI categories based on everyday life experience. Table and 5 present mapping examples.
15

---

Published as a conference paper at ICLR 2026
Table 4: Latent POI Navigation Mapping Examples
User Query
Mapped POI Categories
"Please find the nearest cinema."
Movie theater, Shopping mall
"Please find the nearest ATM."
ATM, Bank
"Please find the nearest coffee shop."
Cafe, Coffee shop, Shopping mall
"Please find the nearest gym."
Physical fitness program, Gym, Fitness center, Gym and fitness centre
"Please find the nearest parking lot."
Parking lot, Free parking lot, Barking garage, Public parking space
"Please find the nearest restroom."
Public bathroom, wheelchair-accesiine bathroom, Subway station. McDonald' s, KFC
Public
Shopping mall,
Table 5: Abstract Demand Navigation Mapping Examples
User Query
Mapped POI Categories
"I need go the airport and would like assistance in finding the best way there. Could you help me?"
Subway station, Bus station, Bus stop, Taxi service, Car leasing service, Car rental agency
"I want to bring my child play and need assistance finding suitable place nearby. Could you help me?"
Shopping mall, Park, City park, Museum, Children's amusement
Art museum, center, Playground
"I want to buy fruits and vegetables and need assistance finding suitable place nearby. Could you help me?"
Fruit and vegetable store, Greengrocer, Farmers Farmers' market
Supermarket,
"I want to exercise and need assistance finding a suitable place nearby." Could you help me?
Park, City park, Physical fitness program, Gym, Yoga studio, Swimming pool
"I want rest and read and need help finding a suitable place nearby. Could you assist me?"
Cafe, Coffee shop, Public library, Library, Book store, Park
"I want work with Wi-Fi and need as- sistance finding suitable place nearby. Could you help me?"
Cafe, Coffee shop, Public library, Library, Book store, McDonald's, KFC, Five Guys
"I have run out of phone credit and need recharge it. Could you assist me in finding a nearby place where I can do so?"
Cafe,
Coffee shop, Library,
Public Supermarket,
library, Telecommunications service provider, Cell phone store, Target, McDonald's, 7FC,7-ELEVEN Best Buy
"I'm feeling hungry and would like something to eat. Could you help me find nearby place?"
Convenience store, Supermarket, Market, Dessert shop, Food truck, Food stall, Shopping mall, Restaurant, Diner
"I'm feeling thirsty and would like some- thing to drink. Could you help me find a nearby place?"
Convenience store, Shopping mall, Cafe, Bubble tea store, Water fountain,
Supermarket,
Vending machine,
Dessert shop, Juice shop
"I'm not feeling well and need assis- tance finding a suitable place nearby. Could you help me?"
Clinics, Pharmacies, Hospitals
16

---

Published as a conference paper at ICLR 2026
Enrichment with Visually-Grounded POIs. To enrich our POI beyond standard map data, address the challenge of temporary or purely visual POIs not available from map providers. We employ a powerful VLM identify such facilities (e.g., temporary food stalls vendors) within street-views. To ensure data quality, all identified POIs then undergo a manual verification process. Once confirmed, these visual POIs added to our graph database with their corresponding locations, making them available for the query-to-PO) mapping process. This step is crucial for ensuring our benchmark reflects the dynamic and visually rich nature of urban environments.
Query POI Cypher Statement Example. To retrieve specific POI instances, we construct Cypher queries that consider category attributes, name-based keyword patterns, or descriptive information within the graph database. The queries are adjusted based on the different POI categories provided by various map vendors, ensuring flexibility in retrieving relevant POIs efficiently. For Inclusive Infrastructure and Semantic Preference, POI, which is stored in the retrieved POI metadata. This includes descriptive terms that specify human-centric attributes of the POI, such romantic, upscale, family-friendee cozy, outdoor seating, wheelchair accessible entrance, etc. The Cypher query combines these descriptive words with category speccific terms, shown below.
query the corresponding description information for each
Cypher Query Example Find the nearest group family-friendly restaurant,
MATCH (place:Poines (1:C])]INNNNNNNN -> (vp:
WHERE (ANY (key keys (vp)
WHERE (key CONTAINS 'intrrouctionn OR key CONTAINS result') AND
(vp key] CONTAINS 'Groups' QR vplkeyl CONTAINS "Family-triendly'>> AND (vp.category CONTA 'restaurant/ OR vp. cateerry CONTAINS 'diner')
RETURN COLLECT IDISTINCT place) AS validPO1s
For Abstract Demand, a query such as "I want to rest and read" triggers Cypher query that retrieves POIs from categories like fafe, Co fee Shop, Public Libra ry, Library, Book Store, and Pa Park, while simultaneously considering relevant keywords like "park" and "book store" in the POI names. This dual consideration of category and keywords ensures that the query matches a comprehensive set of potential POIs. The Cypher query is follows:
Cypher Query Example want to rest and read and need help finding suitable place nearby.
MATCH :elace:Point]] CONTAINN]] -> (vp: VisiblePoit))
WHERE vp.category ("Cafe', 'Coffee shop', , 'Public library', 'Library', 'Book store', 'Park']
OR any (keyword IN [ I'park', 'book store']
WHERE toLower (vp. name) CONTAINS toLower (keyword)) RETURN COLLECT (DISTING) NCT place) AS val idPO1s
Manual Trajectory Validation. To mitigate potential mapping incompleteness, we implement manual verification to re-inspect generated trajectories, ensuring no alternative POIs that could fulfill the user's 's request exist along the path except at the designated endpoint. By doing so, we ensure that the generated trajectories are comprehensive and accurate, reducing the likelihood of missing target POIs or introducing redundant POIs that could lead to incorrect ground truth.
To further prove that our common sense mappings are not designer bias but reflect robust human common sense with cross-cultural generalization, we conducted a new supplementary survey. We surveyed 22 participants across four regions (30 North America, 40 Asia, 30 Europe, 20 Other). We presented our 10 abstract queries (e.g.. "I'm feeling thirsty..."). For each question, participants selected from 50 POI categories. These options included our pre-defined mapping categorics mixed into a large, systematic pool of 440 other POI categories drawn from the entire Google/Baidu POI classification system.
The results show massive consensus cliff. Our pre-defined mapping options received Global Average Consensus of 83.39 %. while the unselected other POI categories received a Global Average Consensus of only 1.90% %. This 81 % gap proves that ground truth mappings not designer
17

---

Published as a conference paper at ICLR 2026
Table 6: Full Cross-Cultural Consistency Statistics (Part 1). Survey results (N = 120) validating Standard Deviation.
Need-to-POI mappings. SD indicates
User Query
POI Category
Statistics (Mean SD)
"I'm "I'm feeling hungry and would like something to eat. Could you help me find nearby place?"
Supernarket Restaurant
Mean: 88.89% (SD: 7.70%)
Mean: 88.33% (SD: 7.26%)
Diner
Mean: 85.28% (SD: 8.01%)
Convenience store
Mean: 80.56% (SD: 4.19%)
Shopping mall Market
Mean: 73.89% (SD: 7.88%)
Mean: 64.44% (SD: 21.43%)
Food stall
Mean: 60.56% (SD: 4.19%)
Dessert shop
Mean: 50.83% (SD: 11.27%)
Food truck
Mean: 46.67% (SD: 25.17%)
"I'm feeling thirsty and would like something to drink. Could you help me find nearby place?"
Convenience store Cafe
Mean: 91. 91.94% (SD: 5.02%)
Mean: 87.50% (SD: 4.64%)
Vending machine Supermarket
Mean: 80.28% (SD: 2.93%)
Mcan: 79.17% (SD: 3.63%)
Water fountain
Mean: 16%11% (SD: 18. 18.58%)
Bubble tea store
Mean: 63.06% (SD: 29.58%)
Dessert shop
Mean: 61.94% (SD: 8.83%)
Juice shop
Mean: 55.83% (SD: 8.46%)
Shopping mall
Mean: 45.00% (SD: 5.00%)
"I'm not feeling well and need assistance finding suitable place nearby. Could you help me?"
Pharmacies Hospitals
Mean: 96. 11% (SD: 3.47%)
Mean: 87.50% (SD: 4.33%)
Clinics
Mean: 82.50% (SD: 7.95%)
"I want rest and read and need help finding suitable place nearby. Could you assist me?"
Public library
Mean: 96.11% (SD: 3.47%)
Park
Mean: 94,44% (SD: 3.85%)
Library
Mean: 89.17% (SD: 1.44%)
Coffee shop
Mean: 83.33% (SD: 8.82%)
Cafe
Mcan: 82.22% (SD: 3,85%)
Book store
Mcan: 75.56% 7.70%)
"I want to work with Wi- Fi and need assistance finding suitable place nearby, Could you help me?"
Cafe
Mean: 94.17% (SD: 2.20%)
Coffee shop
Mean: 93.89% (SD: 3.47%)
Library
Mean: 91.94% (SD: 1.73%)
Public library
Mean: 91.39% (SD: 5.55%
Book store
Mean: 66.67% (SD: 3.33%)
McDonald's
Mean: 66.11% (SD: 9.77%)
KFC
Mean: 63.06% (SD: 8.35%)
Five Guys
Mean: 50.56% (SD: 17.35%)
"I run out of phone credit and need to recharge it. Could you assist in finding nearby place where I can do so?"
Cell phone store
Mean: 93.61% (SD: 3.763.)6% 3.76%)
Telecom. service provider 7-ELEVEn
Mean: 90.28% (SD: 2.93%)
Mean: 82.78% (SD: 17.02%)
Supermarket Convenience store
Mean: 77.22% (SD: 2.55%)
Mean: 74.72% (SD: 6.47%)
Best Buy
Mean: 62.50% (SD: 34.59%)
Cafe
Mean: 61.94% (SD: 5.02%)
Library
Mean: 55.83% (SD: 15.07%)
Coffee shop
Mean: 54.44% (SD: 5.09%)
Target
Mean: 51.39% (SD: 42.95%)
Public library
Mean: 48.61 % (SD: 10.55%)
McDonald's
Mean: 46.67% (SD: 11.55%
KFC
Mean: 45.00% (SD: 13.23%)
18

---

Published as a conference paper at ICLR 2026
Table 7: Full Cross-Cultural Consistency Statistics (Part 2).
User Query
POI Category
Statistics (Mean SD)
"I exercise and need assistance finding a suitable place nearby. Could you help me?"
Park
Mean: 95.00% (SD: 1.67%)
Gym
Mean: 91.67% (SD: 6.01%)
City park
Mean: 87.50% (SD: 6.61%)
Physical fitness program
Mean: 76.11% (SD: 6.74%)
Swimming pool
Mean: 73.33% (SD: 11.55%)
Yoga studio
Mean: 64.44% (SD: 7.70%)
"I want to bring my child to play and need assistance finding suitable place nearby. Could you help me?"
City park
Mean: 94.1 14.17% (SD: 2.20%)
Playground
Mean: 91.94% (SD: 8.83%)
Park
Mean: 88.89% (SD: 1.92%)
Museum
Mean: 85.83% (SD: 2.20%)
Children's amusement center
Mean: 81.67% (SD: 2.89%)
Shopping mall Art museum
Mean: 65.83% (SD: 10.64%)
Mean: 65.00% (SD: 1.67%)
"I need to go to the airport and would like assistance finding the best way there. Could you help me?"
Subway station
Mean: 92.22% (SD: 6.94%)
Bus stop
Mean: 80.83% (SD: 8.78%)
Taxi service
Mean: 78.89% (SD: 1.92%)
Bus station
Mean: 78.61% (SD: 10.55%)
Car leasing service
Mean: 67.50% (SD: 4.64%)
Car rental agency
Mean: 65.28% (SD: 6.47%)
"I want buy fruits and vegetables and need assistance finding a suitable place nearby."
Supermarket Fruit and vegetable store
Mean: 97.22% (SD: 2.55%)
Mean: 90.28% (SD: 2.93%)
Farmers' market Greengrocer
Mean: 79.72% (SD: 11.56%)
Mean: 75.00% (SD: 18.33%)
bias but are the clear, emergent human common sense that surfaces from the noise. Furthermore, the consensus was highly consistent on average. The Average Cross-Cultural Standard Deviation (SD) for our ground truth options was only 8.40% (NA, EA, EU). This low SD value quantitatively proves high overall consistency across cultural regions. Tables 6 and present the detailed statistics for questions.
B.2 INSTRUCTION CATEGORIES AND EXAMPLES
To provide comprehensive overview of benchmark, we present additional explanations and question examples from the seven categories in Table 8 These categories were intentionally designed as hierarchical metric to probe different levels VLM's reasoning capabilities. They represent deliberate scaffolding of different cognitive and semantic difficulty, spanning from Direct Recogni- tion (e.g., "Basic POI", "Brand-Specifice and Contextual Inference (e.g., locating an un-signed "restroom"", to Fine-Grained Attribute Reasoning (e.g., an "accessible entrance") and culminating highly Subjective and Abstract Reasoning (e.g., interpreting "I' feeling hungry")).
C EXPERIMENTAL DETAILS
C.1 IMPLEMENTATION DETAILS
We evaluate 27 commercial and open-source VLMs with multi-image input capabilities, selected based on three criteria: (1) recency and popularity, (2) coverage the full parameter spectrum, from lightweight (< 8B) to very large (2 70B), (3) architectural diversity covering both Transformer- based models and mixture-of -exeertss (MoE) designs (e.g.. LLaMA-4 series), and (4) varied training methodologies including reinforcement learning and reasoning enhanneed approaches (e.g., CoT-based 04-mini and Gemini-2.5 2.5-p))... All commercial models are accessed through their official APIs, while open-source implementations leverage the Hugging Face ecosystem. Inference uses vendors' default
19

---

Published as a conference paper at ICLR 2026
Table 8: Navigation Instruction Categories and Examples.
Category
Description
Example
1. Basic POI Naviga-
Request common urban facilities.
" Please find the nearest restaurant."
tion
Please find nearest convenience store."
Please find the nearest shopping mall."
" Please find the nearest bank."
Please find the nearest healthcare facility"
2. Brand-Specific Navigation
Seek specific commercial brand locations.
" Pleasse find the nearest Starbucks."
" "llease find the nearest KFC or McDonald's. s."
" "Please find the nearest 7-Eleven."
"Please find nearest Chase Bank."
"llease find the nearest Apple."
3. Transportation Hub Navigation
Ask for public transit locations.
Please find the nearest subway station."
"Please find nearest bus station."
Latent POI Navi- gation
Indirectly observable tar- gets requiring contextual reasoning.
" Pleass find the nearest restroom."
" "leease find the nearest gym."
" "Please find the nearest ATM." "leass find the nearest parking lot." "Please find nearest cinema." "
5. Abstract Demand Navigation
Express abstract human needs through contextual clues.
"I feeling hungry and would like something eat. Could you help me find a nearby place?"
"Imm feeling thirsty and would like something to drink. Could you help me find a nearby place?" "I have out of phone credit and need to recharge it. Could you assist in finding nearby place where can so?"
"I "n not feeling well and need assistance finding a suitable place nearby. Could you help me?"
"I want bring my child to play and need assistance finding a suitable place nearby. Could you help me?"
"I need to to the airport and would like assistance in finding the best way there. Could you help me?"
6. Inclusive Infras- tructure Navigation
Prioritize inclusive infras- tructure.
" "Please find the nearest restaurant with an accessible entrance."
" "Please find nearest clothing store with an accessible entrance."
"Please find the nearest apartment building with an accessible entrance." "Please find nearest office building with an accessible entrance." "Please find the nearest bank with an accessible entrance."
Semantic Prefer- ence Navigation
Use descriptive language for subjective criteria.
" "Please find the nearest upscale restaurunt." Please find the nearest restaurant with outdoor seating."
"Please find the nearest restaurant with roadside parking."
" "Please find the nearest romantic restaurant." " "Please find the nearest group or family-friendly restaurant."
20

---

Published as a conference paper at ICLR 2026
settings greedy decoding, except for o4-mini, where temperature 1.0. Experiments executed on machines equipped with NVIDIA RTX 4090 and A100 GPUs.
Human evaluation involved 10 participants- 5 undergraduates and 5 graduates-rrerruted to reflect varied cultural and gender backgrounds. They were compensated at $15/hour in line with standard institution. Participants were selected based both availabil- ity and familiarity with navigation tasks; notably, three of them had previously lived in three or more of the cities featured in our benchmark. Our annotation platform shown in Figure and was configured exactly like the model evaluation environment, with a maximum step limit of 35.
part-time research assistant rates
OITYSECKER
Londe
E- Pleate Ind the neamet shcoping mall
Figure 8: Developed Human Annotation Platform.
C.2 EXPERIMENT PROMPTS
CitySeeker mainly employs two structured prompt types. Both prompts require a JSON output. Deterministic parsing applied, and unparseable response automatically falls back action 0. For extended experiments BCR, additional contextual signals (backtrack_ [surrounding_prompp ), and (history_prompp are provided.
Stop prompt: The model inspects panoramic view at the current node and outputs single JSON block that fuses the overall observation with thoughts / rationale. then decides whether to continue (act ion 0) or terminate (action -1), and returns an accompanying confidence score.
The model compares the perspective images generated for each navigable heading and selects the most promising direction. Two constraints apply: (1) (perspect iiee_prompttttttteeeeeepeeee enumerates the available perspectives; (2) ddirection_promee iot_proopppppt reveals the agent's current facing orientation, helping it avoid reflexively choosing backward path.
Choice prompt:
Stop Prompt
#Instruction: You are a helpful robot that analyses images according to question and helps them find the way reach their destination. Given the question, state whether the scene content satisfies the user's requirement. #Output Format:
[format instructions)
#Example:
##Input:
("type": "text", "text": "I am hungry"),
("type": "image url". "image url": "_."]
]
21

---

Published as a conference paper at ICLR 2026
##Output:
"overall observation": "There residential buildings, bookstore, and bus station.", "thoughts": "I am hungry, so should find restaurant. No restaurant here, keep going.".
"action"; 0
If you think you have already arrived at the destination, output action -1 for stop; otherwise 0.
#Now it's your turn. ##Input: {query]
You are now at viewpoint [viewpoint_fllenae (backtrack [hacktrack_prompt
(image_content)
Choice Prompt
#Instruction: are helpful robot that analyses multiple candidate images selects one that best answers the user's ss question. Return its index (A, B, C...) together with confidence score in [0, 1J. #Output Format:
[format instructions)
#Example:
##Input:
("type": "text", "text": "I am hungry"), ["type": "image url", "image u...:" "..."), ("type": "image url", "image url": "...)
##Output:
pperspective observation":
"A": "A narrow side street with shops amenities nearby.".
"B": "A broad avenue lined with numerous office buildings, suggesting higher chance of restaurants other services."
"thoughts":: "I'm hungry and need the route most likely to lead to food. The broad avenue (B) lined with office buildings is much more likely to have restaurants, while the narrow side street (A) lacks any amenities. Therefore, should head toward B."
"action": "B".
'score": 0.78
peerspective prompt). Make sure the number of observations equals the number of perspectives provided. (direction, prompt). Prioritise the FORWARD, LEFT, and RIGHT directions; move BACK only better option exists.
You have just backtracked, please choose image index (index). Historical context from previous rounds: (surrounding_ promppt Visit trajectory far: chistory_nodes_rrone {query)
You are now at viewpoint (viewpoint filename).
C.3 BENCHMARK EVALUATION RESULTS DETAILS
For completeness, report the outcomes of each individual along with all performance metrics providing a more granular view of the models' behavior.
in Tables 10, and [1]
C.4 CROSS-LINGUAL EXPERIMENT ON LINGUISTIC BIAS
To investigate whether the performance disparity between cities like New York and Beijing was due to linguistic bias in English-only prompts, conducted cross-lingual experiment. We selected
22

---

Published as a conference paper at ICLR 2026
Table 9: The overall performance of CitySeeker Framework.
Model
TCE
TCP-40m
TCP-50m
TCP-60m
TCC
SPD
nDTTW
AS
GPT-4o
2.39%
14.96%
18.30%
23.07%
6.84%
125.4
136.97
21.17
GPT-4o-mini
1.11%
9.94%
12.33%
15.43%
7.56%
201.97
325.17
31.62
04-mini
2.63%
14,80%
17.90%
21.80%
6.76%
130.09
156.33
22.96
Gemini-1.5
1.91%
13.13%
15.43%
17.82%
7.48%
157.14
241.86
31.73
Geminn-2255ppr
1.83%
13.84%
17.34%
20.92%
5.01%
121.75
121.15
18.43
InternV1.2.588B
0.95%
10.66%
14.56%
18.85%
4.38%
118.47
140.43
28.88
InternVL2..-2GBB
1.59%
11.06%
15.27%
19.09%
3.66%
109.53
106
23.4
L
2.23%
14.32%
18.14%
22.28%
7.16%
136.55
169.18
25.03
IntemVL3--B
1.27%
11.85%
15.83%
20.29%
4.85%
118.27
144.51
30.13
IntemW1.3-14B
1.67%
12.41%
15.12%
19.17%
6.44%
136.17
170.81
25.66
2.47%
15.04%
19.25%
23.95%
6.68%
115.82
128.34
25.62
Owen2-VL-7B
0.72%
7.96%
11.14%
15.27%
1.67%
111.4
114.36
24.79
Wwen2-VL-7255
0,95%
9.07%
11.93%
16.79%
2.31%
113.01
89.52
13.13
Qwen2.5-VL-7B
0.48%
11.30%
15.83%
21.16%
4.30%
119.02
151.84
Qwen2.5-VL-322B
2.55%
17.66%
21.08%
24.901%
6.21%
122.6
146.95
24.55
Qwen2.5-VIL-72B
1.99%
11.14%
14.56%
17.58%
7.24%
174.92
250.22
28.63
Llama3.2-90B
0.88%
9.15%
12.49%
15.19%%
3.74%
124.51
123.05
18.99
4ama---Scouut178
1.75%
12.01%
14.08%
18.22%
7.00%
145
211.35
31.74
Llama Maver--
0.88%
7.64%
10.82%
15.99%
1.59%
107.14
110,51
30,97
Llama-3.-111--iise
0.48%
9.15%
12.49%
16.79%
2.39%
112.93
112,44
26.07
Lava-IJlama3-88
0.32%
6.92%
10.42%
14.32%
0.80%
104.79
86.85
20.78
Java-Qwen2-77
0.32%
4.38%
6.42%
11.46%
0.40%
98.07
49.82
4.95
MiniMax01
1.51%
11.61%
13.60%
16.47%
6.84%
172.05
236.55
25.7
MiniCPM-V-2.6
0.88%
8.91%
11.69%
15.12%
3,50%
122.23
152.17
32.33
MiniCPM-o-2.6
1.35%
11.38%
15.51%
20.76%
6.44%
130.05
175.96
32.04
Phi-3.5-Vision
0.32%
7.40%
10.90%
14.72%
0.95%
107.11
81.95
15.88
Phi-4-Multmodal
0.56%
6.36%
9.15%
13.44%%
1.11%
101.06
58.05
9.05
Random Choice
0.72%
8.83%
13.92%
19.17%
3.18%
112.41
128.28
35
Forward Direction
0.24%
5.01%
7.24%
13.21%
0.40%
100.83
99.29
35
Table 10: The subcategory performance of CitySeeker Framework-Part!)
Basic POI
Brand-Specifie
Transit Hub
Model
TCE
TCP-
TCC
TCE TCP-
TCC
TCE
TCP-
TCC
SPD nDTW
SPD nDTW
SPD nDTW
(%)
50m({))
(96)
(%)
50m(())
(%)
(%)
50m( (b) (%)
GPT-4o
1.1
18.9
3.0
118.3 99.6
3.7
21.1
8.7
125.7
163.4
5.4
19.6
7.1
114.3 98.2
GPT-4o-mini
0.9
12.3
4.3
203.2
323.8
3.7
21.1
8.7
176.3
286.8
1.8
14.3
7.1
225.1
331.9
04-mini
1.1
20.2
129.1
141.6
6.2
19.9
11.8
134.3
177.6
1.8
12.5
146.3
187.9
1.5pni-1.5-pro
1.9
17.6
157.4
230.8
4.4
20.5
160.6
255.2
0.0
12.5
131.5
138.8
Gomini-22.5prr
1.5
20.4
121.3
3.7
18.0
122.5
150.4
0.0
16.1
109.2
106.8
Intern VB2.5-8B
0.9
15.7
121.7
133.9
1.2
14.3
3.1
105.5
103.7
1.8
1.8
119.3
111.4
Inter LL2.5-22
1.1
20.4
1.9
110.2
105.9
5.0
12.4
6.8
108.7
104.0
0.0
8.9
0.0
120.5
119.9
Iniern
1.5
17.6
2.8
145.3
170.8
7.5
24.2
10.6
115.7
136.8
0.0
14.3
1.8
126.3
145.2
IntemVLL--BB
0.2
17.2
2.8
125.7
153.2
1.9
15.5
5.6
107.3
113.3
0.0
7.1
1.8
118.5
98.0
IntermVT.3-14B
0.9
13.8
147.1
178.5
4.4
20.5
8.1
112.1
128.4
1.8
17.9
136.4
151.6
InternVLL-333
1.7
7.5
25.5
0.0
Owen2-VL-7B
0.9
0.6
102.5
85.0
Qwen2-V1--2B
0.9
14.8
0.4
111.7
72.2
1.9
8.7
2.5
114.6
92.1
0.0
7.1
1.8
110.6
68.6
Qwen2,5-VL-7B
0.4
20.2
1.1
121.0
151.4
0.6
17.4
5.0
112.5
142.3
1.8
12.5
10.7
127.3
153.6
Owen2.5-VL-322
1.7
22.2
2.6
122.7
132.0
3.7
30.4
8.7
115.4
165.0
0.0
10.7
5.4
132.9
127.3
Owen2.5-VL-722
1.5
14.8
3,9
182.9
251.0
2.5
21.7
5.0
158.1
237.4
1.8
14.3
7.1
159.8 206.0
Llama3.2-9UB
0.9
13.6
135.2
140.5
0.6
12.4
115.7
114.6
1.8
124.3 106.3
Lcout--
1.7
153.8
136.8
Maveric--
17BB
1.1
12.7
113.3
116.5
1.2
10.6
103.0
111.1
1.8
8.9
1.8
99.8
84.0
1 1.mma-3.2- BB-Vision
0.9
15.1
1.1
117.2
114.3
0.0
14.3
1.2
107.0
111.4
0.0
7.1
0.0
113.2
94.2
Lava-LLama-388
0.4
14.8
0.7
108.4
90.1
0.0
8.7
0.6
103.4
87.2
1.8
5.4
1.8
113.0
101.6
Llav-Qwee2--7
0.4
12.5
0.2
102.2
51.7
0.0
2.5
0.0
95.5
49.3
0.0
3.6
0.0
92.9
46.9
MiniMax-01
2.2
15.55
171.0
226.3
1.2
16.2
166.2
218.3
1.8
8.9
158.9
148.5
MiniCPM-V-266 V-2.6
0.7
12.5
128.2
153.2
1.2
10.6
120.6
154.7
3.6
19.6
111.5
129.7
MiniCPM-o-2.6
1.3
14.4
137.4
185.7
3.7
22.4
10.6
112.6
144.7
0.0
10.7
132.7
160.1
1ii-3_5-Vision
0.7
14.4
112.0
87.7
0.0
6.8
100.9
72.9
0.0
3.6
1.8
108.3
67.5
Phi-4-Multimodal
0.9
14.0
0.7
102.5
59.4
0.0
6.2
0,0
96.7
54.0
0.0
5.4
1.8
100.8
49.5
Forward Direction
0.7
13.3
0.2
105.0
102.3
0.0
3.1
0,0
97.8
103.0
0.0
3.6
0.0
99.2
93.8
Random Choice
0.7
16.6
1.7
116.9
129.4
0.6
10.6
4.4
110.6
130.1
1.8
5.4
3.6
126.0
138.0
23

---

Published as a conference paper at ICLR 2026
Table 11: The subcategory performance of CitySeeker Framework-Part?
Latent POI
Abstract Demand
Inclusive Infrustructure
Semantic Preference
Model
TCE TCP.
(56 50m(3))
TCC
(96)
SPD
nDTW
TCE
(9)
50m(%)
TCP-
TCC
(%)
sDTW
TCE (5)
50m(S)
TCP-
TCC
(56)
TCE
50m(S)
TCP-
TCC
(5t
GPT-4o
1.3
9.7
10.2
158.1
208.0
4,4
18.9
11A
129.1
155.6
1.3
14.9
1.5
111.1
111.2
1.9
26.01
8.7
108.
139.9
GPT-oommiii
0.0
7.4
8.0
233.5
379.8
0.9
11.8
13.6
215.5
374,4
0.0
7.5
4.5
175.1
282.7
1.0
10.6
8.7
158.3
214.4
1.1
107.5
8.7
Gemini-1.5-p 1.5ppo
155
Gemini-2.5prr
0.6
10.2
3,4
139.2
140.7
3.5
20.6
10.5
118.7
129.6
0.0
9.0
0.0
111.6
81.8
1.0
13.5
6.7
113.1
103.2
1.1588112.5-.88
1.7
14.8
4.6
125.8
167.8
0.9
18.4
888
113.9
146.5
0.0
10.5
1.5
115.8
156.7
0.0
10.6
39
122.8
171.9
IntemVL2..526B
0.6
10.8
2.6
107.7
107.2
IntemVLL2.53388
0.6
157.2
218.4
3.5
22.8
14.0
132.1
115.3
127.6
10.6
167.6
IntemVi.3-88
2.8
14.2
744
125.8
167.7
1.8
16.7
7.5
1109
145.8
1.5
16.4
30
115.4
138.1
1.9
15.4
5.8
107.
141
IntemVL.3-14B
1.3
8.0
6.3
149.3
195.1
1.8
19.3
10.5
134.3
184.0
1.5
10.5
6.0
110.0
99.4
1.0
17.3
13.5
122.9
188,
1.32mmV133388
0.6
Qwen2-V1.78
1.8
111.0
1.9
Ww-22-VL-28B
0.6
7.4
2.8
111.8
81.1
1.8
14.5
5.3 114.
120.5
0
1.5
1.5
112.0
78.9
0.0
15.4
3.9
117.6
127
Sww-2.5-V1-7B
0.6
10.2
5.7
129.4
163.7
0,4
13.6
7.5
117.2
153,9
0.0
7.5
4.5
117.4
156.6
0.0
15.4
4.8
103.1
139.9
Owm225-V1-328
1.3
Owm225-VVL-7BB
0.0
217.0
319.7
2.6
15.8
11.0
160.7
231.2
9.0
11.9
16.4
139.3
185.8
14.4
12.5
156.7
256.3
1lam33.2-908
0.6
9.7
3,4
125
117.4
0.9
14.0
6.1
1164
109.5
D.O
10.5
3.0
104.9
83.7
1.9
16.4
8.7
119.2
131,
1lama--Scout-12
0.6
8.5
6.8
160.8
239.0
2.2
17.5
11.8 132.8
201.8
0.0
17.9
6.0
119.9
123.0
1.0
14.4
6.7
127.5
188.7
Lamma4-Maerrck-177
1lama-32-111-
0.0
107.9
114.7
4-5
108.1
29
110.5
Llava-Llammis
0.0
8.D
1.1
102.1
80.4
0.4
1.3 103.5
83.7
D.O
4.5
0.0
100.6
88.5
0.0
7.7
0.0
96.8
80.7
Java-Qwen2-77
0.0
2.8
0.6
94.5
51.4
0.9
1.3 94.6
47.4
0.0
90.1
45.5
0.0
1.0
0.0
96.7
49.1
MiniMax-411
1126
1148
MiniCPM-o-2.6
1.1
12.5
8.5
143.7
190.0
1.3
189
10.5
123.5
175.6
0.0
11.9
4.
118.3
169.3
0.0
12.5
8.7
121.7
170.7
Phi-3 S-Vision
0.6
9.1
107.8
830
0.0
11.8
133
1034
76.8
0.0
9.0
3.
110.2
106.6
0.0
7.7
010
994
71.7
Muliimall
0.0
58.4
1.3
Forward Dircttin
0.0
99.4
93.2
0.8
101.0
104.1
0.0
938
95.0
00
94.7
85.6
Random Choice
0.0
10.8
1.7
114.9
131.3
1.3
14.9
6.6
1089
126.3
0.0
16.4
30
96.1
113.9
1.0
13.
29
101.7
123.7
representative models and them tasks in both Beijing and New York, comparing performance when using our standard English prompts versus fully localized Chinese prompts and outputs.
Table 12: Cross-linges performance in Beijing and New York. Localizing prompts to Chinese did not yield consistent improvements, suggesting linguistic bias not the primary performance driver. Model
City
Language
TCP (%)
TCE (%)
TCC (%)
nDTW
English
25.95
4.32
10.81
122.40
Beijing
Chinese
23.91
3.44
8.88
120.36
Qwcn2.5-VL-32B
English
15.42
3.48
3.48
144.78
New York
Chinese
14.93
2.99
4.98
133.13
English
14.63
0.00
3.25
143.41
Beijing
Chinese
15.45
1.63
1.63
148.51
Intern VL3-8B
English
15.42
2.49
6.97
136.13
New York
Chinese
9.45
1.49
2.99
140.07
English
7.32
0.81
3.25
345.36
Beijing
Chinese
7.32
2.44
2.44
217.31
GPT-4o-mini
English
13.43
1.00
9,45
332.95
New York
Chinese
14.93
2.49
8.96
251.84
The results, detailed in Table 12
show that localizing the prompts to Chinese does not provide consistent performance benefit, and the impact varies significantly by model and city. For instance, the top-perforning Qwen2.5-VL-
saw its TCP decrease in both cities when using Chinese. Con- versely, GPT-4o-mini's performance increased New York with Chinese prompts, while In3ernVL3- 8B's performance dropped sharply. This high degree of variability strongly suggests that the observed performance gaps between cities are not primarily driven by linguistic factors but are likely rooted in deeper visual and geographic biases within the models' training data.
24

---

Published as a conference paper at ICLR 2026
"Question", : "Please find the nearest convenience store."
Navigation map each time step
Street panorama at each time step
Time Step
Time Stop
Time Step
Figure 9: Illustration of the agent's input in the map-augmented setting. At each time step, the VLM receives both global navigation map, which updates with its current position, and the corresponding first-person street panorama.
D ABLATION STUDY: THE IMPACT OF GLOBAL MAP INFORMATION
primary design choice in CitySeeker is the focus on VLM's ability to navigate based on its first-person visual perception and intrinsic world knowledge. To better isolate and understand these intrinsic reasoning abilities, we conducted an ablation study to quantify the impact of providing explicit global map information. This study explores alternative, nuu-auugentedd setting where, every step, the VLM was provided with both its first-person street view and an interactive 2D global map. This map dynamically updated with the agent's s current position and heading, mimicking the experience human using a modern navigation application.
D.1 EXPERIMENTAL SETUP
In this setting, the agent's prompt augmented at each step with both dynamically rendered 2D map and the corresponding first-person street panorama, as illustrated in Figure This interactive map always showed the full planned route while centering on the agent's current position and heading, providing continuous global context just as user would see in navigation software. The VLM was explicitly instructed first analyze this map to determine the geometrically optimal next step and then select the corresponding ttreet-view perspective. This setup was designed to test if providing such real-time, human-like map guidance would simplify the task and improve performance.
D.2 RESULTS AND ANALYSIS
We evaluated two top-performing models, GPT-4o and ----22..VL--32-111 on the full bench- mark test set under this map-augmented condition. The results were counter-intuitive and revealing, summarized in Table B
Table 13: Performance comparison between the map-free and map-augmented navigation settings on the full test set. The new results reveal a trade-off where map guidance improves path following (nDTW) but degrades task completion (TCP/TCE).
Model
Map-Free
Map-Augmented
TCE (%)
TCP (%)
nDTW
TCE (%)
TCP (%)
nDTW
GPTT-4o
2.4
18.3
136.9
0.9
11.7
75.7
Qween25-VV-S2D-IInsmmc
2.6
21.1
147.0
0.2
7.6
54.4
Contrary to expectations, providing a global map at each step led to a surprising trade-off: while fath-following efficiency improved dramatically (as shown by lower nDTW scores), the models'
25

---

Published as a conference paper at ICLR 2026
VLM: Qwenn2.--V-332--Inniti
Question: "Please find the nearest convenience store.". Overall Observation: Map: Reached the final destination (red marker). Street View: A convenience
store (NYC Deli) is visible.".
Thoughts: "StopiContinue Thought: "Step 1: Map Analysis. ** The map shows my blue dot right top of the red target marker, which marks the end of the blue line. This indicates that I have reached the intended destination. "Step 2: Street View Analysis." * The street view shows building with sign that reads "NYC Deli," which suggests a convenience store or similar establishment. This aligns with the goel finding a convenience store. "Step Decision." ** Since map indicates that have reached destination and the street view confirms the presence of a convenience store, I must stop.",
Action: -1 Action Direction: Stop
Score: 0 0.98
Figure 10: An example of a typical reasoning failure from Qwen2.5-VLL322--Imse Instruct..
ability complete the actual task collapsed. For both models, Task Completion (both TCP and TCE) dropped significantly. For instance, the TCP for Qwen2.5-VL- 1--.55----32B--nttee fell from 21.1% % to just 7.6%, even as its path alignment score (nDTW) improved from 147.0 54.4. This suggests that while the map helps the agent follow geometric route, it severely hinders its ability to perform the core task of semantic discovery. Our analysis of the models' reasoning traces, an example of which is shown in Figure I0, revealed primary failure modes:
The models demonstrated weak understanding of direction and distance on the 2D map. They frequently failed to perform correct "mental rotations": for instance, when their heading was North but the route required moving South, they struggled to select the "back" perspective. Furthermore, they showed little awareness of distance, often hallucinating that they were near the destination when still at the starting point.
(1) Poor Cognition of 2D Map Geometry:
The VLMs tended to fixate on the 2D map's abstract geometric instructions, failing to properly align them with the labeled, first-person street views. For instance, after inferring "left turn" from the map, model would still frequently select the "front" perspective, despite being explicitly told which perspective corresponded to the "left" direction. This demonstrates deep failure in grounding an abstract command, even when provided with all necessary information.
(2) Over-reliance and Failure to Align:
(3) Trivialization of the Core Challenge: The map effectively turned the task from one of discovery and semantic reasoning into a simpler, but more brittle, geometric path-following exercise. The core challenge of our benchmark- innferrrng that "I'm thirsty" means looking for a café and then visually identifying one-was often ignored. The models would become 'map followers". focusing only on the blue line and failing perform the crucial visual exploration needed to identify the actual POI that satisfied the user's implicit need.
D.3 CONCLUSION
This ablation study demonstrates that simply providing a global map does not necessarily solve the core challenges of implicit-need navigation and can even distract the VLM from the essential task of grounding abstract language in the visual world. Furthermore, the map's limited information and potential for positional deviation can introduce additional sources of error. Therefore, isolate and rigorously evaluate the agent's 's intrinsic spatial cognition and commonsense reasoning abilities, our primary experimental results are reported under the map-free setting. This approach forces the VLM to build and rely on its own internal mental model of the environment, providing a truer measure of its embodied intelligence. While developing agents that can effectively fuse map and
26

---

Published as a conference paper at ICLR 2026
visual information is valuable direction for future research, work focuses first establishing this crucial baseline for the VLM's intrinsic, perception driivn capabilities.
E BCR STUDIES DETAILS
E.1 BACKTRACKING MECHANISMS
We propose three distinct backtracking strategies to mitigate error accumulation over long trajectories in scrge-scale VLN tasks in urban environments: (B1) Basic Backtracking, (B2) Step-Reward Backtracking, and (B3) Human-Guides Backtracking.
B1. Basic Backtracking
B2, Step-Reward Backtracking
B3. Human-Guided Backtracking
Thoughts:...
Hint: Steps
Hint
Guide: You should
choose Perspective
VLM
VLM
VLM
e. n comoonten
6-8.n=5
n= mononnoe Bocktrack
Backtrock
Backtrack
Start
End
Start
End
Start
End
Staps to Destination
azgamin E H'deeltee ma)
k1l
A (4:
Hint
Figure 11: Backtracking Mechanisms: (B1) Basic Backtracking, (B2) newa-d Backtracking, and (B3) Human-Guided Backtracking.
Basic Backtracking (B1). In this basic backtracking strategy, the agent reverts to the last "trusted" node when its internal confidence falls below predefined threshold over several consecutive steps. The model tracks its confidence scores during each step, and if the average confidence score over number of steps drops below a threshold, the agent will backtrack the last trusted node. This simple strategy operates independently of any external signals. The confidence metric, stored in deque, is updated at each step with the current score of the model's prediction.
The implementation process maintains a sliding window of prediction confidence scores st/=n-k over A consecutive steps (default k: 3). Backtracking triggers when:
3
8
(0 0.75)
1+
The agent reverts the last node where the confidence 8; meets or exceeds the threshold The agent reverts A: steps when the average confidence ø falls below 0.
Step-Reward Backtracking (B2). This mechanism evaluates progress toward the goal by replac- ing subjective confidence scores with objective topological distance as the backtracking criterion. Specifically, it calculates the shortest path steps d, from current viewpoint to target Uraget in the graph topology G.
During each step, the agent checks whether the distance to the goal has increased. Backtracking triggers when distances increase monotonically over consecutive steps:
A
(d d(-(+1))
i=0
This condition ensures that if the agent's progress toward the goal stagnates or worsens, it will backtrack.
27

---

Published as a conference paper at ICLR 2026
Human-Guided Backtracking (B3). This strategy extends basic backtracking B1 with correc- tive guidance, providing minimal external "hint" that suggests the best action to take next after backtracking. The optimal action a" maximizes path consistency with the shortest path to Vtarget through:
arg min B(d++1 la = a]
eEAt
where Al denotes available actions backtracked node uz, and de+1 represents the expected graph- theoretic distance after taking action a.
The guidance integrates by dynamically aligning the shortest path directions with navigable headings
through topological analysis. The action that maximizes path consistency is selected via:
Φ(a) = Tua E Opptimal cOs - 8ptth)
(4)
where On is the heading direction for action a, and Opptimal is the set of optimal path headings. This external guidance helps ensure that the agent's ' action maximizes path consistency and progress toward the goal.
E.2 SPATTAL COGNITION ENRICHMENT
In this section, we explore the potential of providing richer spatial cues to an agent to enhance global awareness and reduce fragmented decision- makiin during navigation tasks. Specifically, we compare two distinct methods of presenting spatial information: Topology Cognitive Graph (C1) and Relative Position Maps (C2). The goal is to investigate how these different representations influence the agent's s ability to understand its environment and effectively plan routes.
We investigate whether supplying richer spatial cues can bolster global awareness and reduce frag- mented decision-making by synthesizing multi-model trajectory data through GPT-4.1 as cognitive summarizer. Our methodology processes ground truth paths and generated trajectorics from GPT-4o and Gemini2.5-pro (containing both correct segments and historical errors) for each navigation task. GPT-4. analyzes these heterogeneous trajectories using specialized prompts to produce structured spatial representations, which then integrated into the initial input for downstream VLMs.
Itttorrial Thajectory Sampling
Tepuotagy
Ct. Topotogy Cognitiiee Graph
C2. Relative Position Map
Goundiuu
Tapsaay
Gream
Paltiin
LUM
Piceoc
Figure 12: Spatial Cognition Enrichment: (CI) Topology Cognitive Graph, and (C2) Relative Position Maps.
(C1) Topology Cognitive Graph. In this approach, the VLM is provided with topological graph of recently traversed segments, which explicitly defines the connectivity between various locations. The graph consists of nodes representing locations and directed edges that signify possible actions or transitions between those locations. Each trajectory is annotated with explicit relationships between nodes, forming a clear and structured map of the environment. This graph can be seen as an abstracted representation of the agent's route, where the agent is instructed to focus the connectivity between locations. This forces VLM to ground decisions in connectivity patterns, reducing exploration of invalid paths.
An example of how this spatial information is represented is shown in the prompt:
28

---

Published as a conference paper at ICLR 2026
Topology Cognitive Graph Generation Prompt
Given the following trajectories with the same starting point for the same question, connect them and represent the relationships between the nodes in graph format.
Example:
#*# Node: 00002929.jpg
**Relationships:**
*00002929 jpg - acttio 0)* '000029330.jp''
*00002929, .rg + **cttion 14* '00002931.jpg"
### Node: 00003210.jpg
** *Relationships:***
000003210.jp5 - action 0 "00003211.jpg
.00003210.jpg + action *** "00003212.jpg'
### Node: 00001111.1
* **Rclationhhips**
'00001111.jpg'
** **action 0 '00001112.jp''.
(C2) Relative Position Map. In contrast to the topology-basek approach, the Relative Position Map emphasizes the spatial orientation of locations without directly specifying connectivity. Instead of relying explieit graph structure, the VLM receives approximate directional cues that describe the relative positions recently visited nodes. These cues include terms such as "front", "back", "left", "right", and "slightly left/right", along with approximate distances between nodes. This approach provides the VLM with a more intuitive understanding of the space, allowing it to navigate with flexible mental model of its surroundings, albeit with less precision in terms connectivity.
An example of the relative directional information provided to the VLM is:
Relative Position Map Generation Prompt
Given the following trajectories with the starting point for the question, describe the spatial relationships between the key nodes. Emphasize relative directional orientation (such as front, back, left, right, slightly left, slightly right, etc.) and relative distances. Note that, not all nodes are directly connected, so focus on the relative spatial relationships rather than direct connections. Example:
#*# 10001199.jpg 00001192.jpg
** Direction Right
Relative *o Distance: 20-30 meters
#*# 00001192.jpg - 00001204.jps
** # Slightly right
** Relative *0 Distance: 30-50 meters
#$# 00001204.jps 00001539.jpp
Direction Slightly left
Relative "osition**; Distance: 50-70 meters.
This method captures the VLM's internal spatial sense, where perceives locations in terms of relative proximity and directional alignment rather than distinct, connected points. The benefit of this approach is that fosters a more adaptable navigation strategy, particularly in dynamic or complex environments where exact node connectivity is not always available necessary.
E.3 MEMORY-BASER RETRIEVAL
To address the issue of "fragmented memory", we introduce memory- retrieval mechanism based on the Neo4j graph database. This mechanism consists of three core components: (R1) Topology- Retrieval, (R2) Spatial-based Retrieval, and (R3) Historical Trajectory Lookup. This architecture enables memory formation across multiple reasoning iterations while mitigating error propagation.
Topology-based Retrieval (R1). Each navigation query undergoes n sequential rounds of execution. During rounds to 1, the agent explores the environment without memory retrieval, with all trajectories and node metadata (c.g.. observations, thoughts, decisions, and confidence scores) stored in Neo4j. In the final round n, the agent activates memory retrieval mode, dynamically accessing
29

---

Published as a conference paper at ICLR 2026
Memory Topological Graph
R1. Topology- Retrieval
Spatial-based Retrieval
Historical Trajectory Looloup
50mm
Ao
Currenty Waating
Camenty
Qumerrty Visting
Mode
Figure 13: Memory-Baa -Baed Retrieval: (R1) Topology-basd based Retrieval, (R2) Spatial-basede Retrieval, and (R3) Historical Trajectory Lookup.
historical cross-round information. At each navigation step, the agent queries a local subgraph spanning ht-opp topological connections (default h=1) through Neo4j's graph traversal operations. The returned subgraph contains:
Nodes with cumulative visitation counts and n round metadata (thoughts, previous/next actions, directional choices, confidence scores); Edges annotated with historical transition patterns across multiple rounds (successful/falled attempts, directions). For example, a node might reveal that previous rounds chose "action 1" (right turn) when transitioning specific neighbor. This topo- logical memory enables the agent to prioritize frequently successful paths while avoiding recurrent error-prome branches.
Spatial-based Retrieval (R2). Complementing topological constraints, this mechanism retrieves localized subgraphs based on Euclidean spatial proximity. At each navigation step in the final round n, the agent queries all nodes and relationships within configurable radius (default: 50 meters) from the current position using Neo4j's geospatial index. This dual focus on geographic density and historical usage patterss allows the agent to weigh physically closer options while avoiding previously failed paths. By jointly modeling metric space and graph topology. R2 mitigates the "tunnel vision" of pure topologiaal approaches, particularly in irregular urban layouts where physical proximity often outweighs structural connectivity.
Historical Trajectory Lookup (R3). This mechanism introduces short-term working memory by dynamically appending the agent's recent navigation history within the current round. At each time step I, the mechanism automatically appends the trajectory segment from the preceding n steps (n by default), including: (1) spatial context filenames, coordinates, and headings of visited nodes; (2) action traces movement directions (e.g., "turn left", "proceed straight") and associated confidence scores; (3) decision rationale- preserved observations and reasoning from prior steps.
Unlike cross-round memory in R1/R2, R3 operates through a sliding temporal window that exclusively tracks intra-episode navigation patterns. At step t, the 1 to t entries get injected into the VLM's input context through template-based natural language formatting, preserving recent decision logic. This short-term memory gets purged upon round completion through automated attribute pruning in Neo4j, ensuring no residual traces affect subsequent trials.
E.4 PERFORMANCE OF COMBINED BCR STRATEGIES
To explore the synergistic potential of our proposed strategies, we conducted preliminary experiment where we combined several effective mechanisms (B2, B3, C1, R3). The results, shown in Table [4] indicate a positive but not strictly additive effect. For instance, the combined strategies boosted Qwen2.5-VL-32B's TCP from 19.9% to 27.38%. This suggests complex interactions between the different cognitive tools. full exploration of optimal strategy combinations a promising direction for future work.
F VLM GENERATION EXAMPLES
30

---

Published as a conference paper at ICLR 2026
City: London Category: Basic POI
"Question": P find the nearest restaurant."
VLM: GPT-40
Current Position: www.Pnnlootooolllnnaaaaaoo 0000
Overall Observation: scena conta resice buildings longitude 165933522 52882 latiude 48 1679)
small alley, but no visible restaurants
Perspective Observation A "This pe repective shows now alley with a van in front, which could load a more open erea."." "B" "This clive shows dead-end with bins and no vieible pell forwaird
Thoughts: I am looking for a restaurant. This image shows residentia area with no visible restaurants. sho keep moving find restaurant.
Action Direction: Step Forward
Stop
Current Positton: Weew (Postoon(llanaass 10000
buildings, longitude =-0 165809952, lattude 4889 28)
Observation: imaae shows
Perspective Observation: ('A' "This perspective is your FRONT, showing a emall street leeding foraadd with possible stores: "g": "This serapecoive is on your BA CK, showing continuation on the street with visible tulliinggs "}
Thoughts: am trying to find restaurant urant, but this image dbes not see to show any signs restaurant. sshuuld continue looking.
Action:
Action Direction: Slep Forward
Score:
Step
Overall Posifion: www./innassllnnilleennaa
jos', longitudes (Waterstones)
48.8632251
buildings visible imege.
restaurants.
Pempective Observation: ("A *This pe perspcttive on your FRONT with visible shops and poto mial restaurant options.", "B": "hii cliveeelive is on your BACK and shows quieter street with fower visible ameritec rities..
Thoughts: am lookkigg for the nearest restaurant, but there is no restaurant visible in this image. I should continue searching to find suttable location.
Action: 0
Action Direction: Step Forward
Step 10
Current Position: View PnnoaPPPotortlleenaaaa. =0000=0002222 Joo', longitude =-0, 165657411 lstlude 48873373)
Oveiall Observation: There restaurent thi image.
Perspective Observation: ('A': T "his perspective perspective
RIGHT and shows with retail stores pedestrians." ", "B": reetaurents seeting visible.. "This options,
per perspectiv
your
Thoughts: looking restaurent. this image, there restaurant visible, so have arrivee at destination can stop here.
Actioni -1
Action n Disection: Slop
Score:00
Step 17
Figure 14: Further illustrative examples drawn from the CitySeeker benchmark.
31

---

Published as a conference paper at ICLR 2026
City: London Category: Brand-Specific
"Question": Please find the nearest Starbucks."
VLM: GPT-4o
Current Position: wewPdnnoosttonfliiili 00:000mm00000000000000000 66 jog), longltude 16780781 latitue==11 888
Overall Observatio:: There buildiigs under de development con construcion, but visible Starbucks or coffee shop this
image.
Parspective Observatin: (AA: "This perspective is your BACK. K It shows a dlased area with los ding docks and no visible signs of Starbucks "B": "hhs perape ctive on your FRONT. It shows pathway with building a and a potertia way forward.
Thoughts:
looking for the nearest Sta rbucks, but do not see any Starbuccks coffee shop in this Image. This area construction bu huildings, but Starbucks visible.
Action.
Directio::
Forward
Current Position: MewPannFostorrtllee amm 86 jpg longrtude=00 1666 12021, Ielitude=51 4888-4625)
Overal Observatio: Ther are various shops, including dlahing stores and a Five Guys restaurant, but Starbucke visible, Perspective Observation: ('A': This perspeciive is on your BACK with vieible shops and calm street" "B": "This perspective FRONT with visible shops. bus, and more sctivity: ") Thoughts: am looking for Startucks. From image, can see stores and shops, but Starbucks visible. should continue searching for the destinatin.
Directon:
Step 26
Currant Position: ViowP
longitudes 1684 70288, tattuue==11 48840884)
Overil Observalion: There
including Five Starbucks visible this
"B":
Starbucks direction.]
Thoughts: looking
rearest
image, stores Five Guys shops,
no visitle Starbucks. should continue searching to find Starbucka. Actio n: 0
Action n Direction: Step Forward
Scorm: 0.96
Stop 27
wineenimmm Ovemil Observation: There Starbucks this image. Perspcctive Observation: ( perspectiv your FRONT clearty shows Starbucks the right side strest". "Tiis perspectivg on your pe rapaciive shows Starbucks appears futther mway.")
Current Position:
longhude== 1663288556 latitude=51 48846744 43)
your
Thoughts: am loaking for Starbucks, Them is a Starbucks visitle this image, so have arrived at the destination and do not need to conlinue furher.
Actio n:
Action Direcion: Stop
Score: 0.95
Figure 15: Further illustrative examples drawn from the CitySeeker benchmark.
32

---

Published as a conference paper at ICLR 2026
City: New York Category: Latent POI
"Question": "Please find nearest restroom."
VLM: GPT-40
Current Pisiion:: wwwpaaoaooootoofiaaeeeeec
ing', langitue== = 73 9887 89 latitud==:0) 74 44274025)
Overall Obsarvalion: There are various commercial buildings, s, shops, and vahicles visible in this image, but no dleer indication of restroom.
Parspectiv Observation: ('At: "This porspe clive is BACK. The view shows street with buildings and parkod
vahicles." "B": "This perspective is your FRONT. The view shows street with shops, parked cars, and some visitle
storefronts.7 ]
Thoughts:
nearest
image, observe any signs indications
need continee searching.
Action Diraction: Step Fonward
Current Position: ViewPPin Prostion(fienamme 000003 38 josl,
9900 7471. Iatiudend0 74598407)
Overall Observation: The image shows urban streat commerciaa buildings, stores, and McDenald's, restroom
visidle
signs.
Perspective Observation: ('A "This perapective shows street with ried cars and some buildings, but clear ind indicaaion perspective shows busy infersection
signage Thoughts: am looking for the nearest restroom. There is no visible restroam in this image, so should continue searching to find one.
by restroe mn options.)
Action:
Action Direction: Slop Forward
Score: 0
Stop
D
Current Posilion: ViewPcinPostion/ PonnPeoonflllaameee wwwnnaleeee00000e 000001 ins', longitude= 73 99.99900111 lafitud 40 74604502)
Overall Sesevatioo: This image shows urban intersection with buildings, vehicles, and pedestrian but visible restroom
signs facillies.
Perspective Observation: ('A': "This perspective
FRONT LEFT and shows
buildiigs cars.
"B" This perspective
FRONT and highights McDonaldss which may restrooms. "C* "This perspective perspective
on
directio.."
on your FRONT RIGHT and dippaass McDonald's which may have ms Thoughts: need
imago, abserve any spe signs indicating presence restroom. Therefore, should contrue searching. Agtion:
restroom.
Action Direction: Step Forward
Score: 0.9
Position: ViewPainPosstoryyleee
longitude
(af 74880222)))
Observatio::
image, might restroom
several building,, including possible stores resta urantss 'B": This perspectio shows narlower street with McDonald's visible, which might have restrooms
Persective Observatio:: ('Al: perspective
avallable.
Thoughts: am looking for the nearest restroom. From this image, I see McDonald's, which might have restroom inside. should check out as could fulfil requirement.
Action: -1
Action Direcionn: Stop
Scom:: 0.:8
Stap
Figure 16: Further illustrative examples drawn from the CitySeeker benchmark.
33

---

Published as a conference paper at ICLR 2026
City: Shanghai Category: Abstract Demand
"Question": "I want to work with Wi-Fi and need assistance finding suitable place nearby. Could you help me?"
VLM: Qwwn2.5-VL.
Current Po siton: PoPoinntttiinlllaaooao 0000 40 loo'. longltude 121 5061 697, lellude 31 27 52
Overall Observation: This image shows wide strect with vanous buildings, including commercial estabishments and bank. Wi-Fi availab specific place like cafes libraries typically known for Wi-Fi street", goal buildings, insluding appe buildings commercial structure. place would be best to contrue exploring the area.
There visible
Perspective servatioon: (AA
residential
right
Thoughts:
places image,
Astion:
Astion Direction: Step Forward
Score: 85
Stop
Position:
ViewP (osttbo(tliiaaame 1000000000000000000000000 lng longltude=11
places 57888777
Onservation:
with buildings, vehicles,
street older buildinns and less commercial actilty making less likely to have public Wi-FI", ngr. "The right side features modern building with sign, suggesttig commercial area like mail office spa who public Will more common- ")
Perspective Observation: (AA The side
Thoughts: The scena shows busy street with ngis cars, and bus. There are no visible signs WiFi.enab led place
like café library. Since the user looking for placce to work with Wi-Fi, it's best continue searchingg
Action:
Action Direction: Stop Forward
Step
Current Position: NeewPinnPsstinnffeeeee filename (000mm==000000025500000000000000 00004024 loo', longitude - 121 506507 lelitude 31 27 27302746)
Ovemll Observatio: There is widh road with multiple lanes, bardered by trees and modiern building's s. There are no visible signs indicating the gresence Wi-ihhotsp Fiholspats
cafes.
Pe Perspective Observation: ('A" TThis repective shows clear forward with buildings and shops either side, which could be potential places with Wi-Fi "B% This perspe ctive shows backwvrrd view. which less likely to leed place Wi. since itss alroady passed
Thoughts:
public zanes,
exploring
by moving
find place.
Action:
Action Directon: Step Forward
Score: 0.96
Step
Overall ViewPointPoslla
Starbucke shop long
lafitude=31
image, typically
Pe rspective Observatio:: (A FRONT image displays wide stree with bulldinge and pille spots ahead
"g": "Tii perspecive is on your LE and showis street with some restaumnts and seating area visible " "The
BACK imaige not vieible, stepping BACK discoureged""
Thoughts: Since the goel is to find place with WAFI, and tarbuckk Coffee shop is suitable place for this ne reed, should stop here.
Action:
Action Direcion: Stop
Figure 17: Further illustrative examples drawn from the CitySeeker benchmark.
34

---

Published as a conference paper at ICLR 2026
Table 14: Performance with combined BCR strategies.
Model
Baseline TCP %
Combined BCR TCP %
Intern VL338B
15.0
18.96
Qwen2.5-VI-332B
19.9
27.38
G ERROR ANALYSIS
Error Type Distribution Comparison
Wies Ooservanis
UndenGyethhiing Oriacal Cecuse Orere
waw.saaaeeeiiiiiii Instraciio Paniing Stiees
Cowrraann Hurach
Owen2.5-VL-322 Error Distribution
InternVL3-88 Error Distribution
6.3%
10.8%
22.2%
17.
10.1%
4.4%
22 8M
32.9%
27.8%
Figure 18: Error Distribution of Primary Error Types in Qwen2.5-VL-32B and Intern VL3-8B for Embodied Urban Navigation.
To identify the main bottlenecks in VLMs for Embodied Urban Navigation with implicit human needs, thoroughly examine and analyze the primary error patterns observed in the 300-sample mini-size subset of the CitySeeker benchmark. By manually inspecting model outputs - including observations, rationales and actions. with corresponding street-view images, we categorize failure modes into six distinct types (visually exemplified in Figure 19):
(I) Visual Observation Errors: These arise from failure recognize target or misidentification of critical visual cues (e.g., overlooking signage while fixating on irrelevant street objects).
Even when the VLM correctly observes the target, the rationale diverges from the observation. For example, the model visually identifies a target restaurant but fails to prioritize advancing toward it in its rationale.
(2) Observation-Thime Mismatch:
Underthinking is particularly evident when the model lacks knowledge of non-primary functional affordances of POI. For example, while recognizing Starbucks, the model ignores its potential to provide Wi-Fi, leading to missed target. Overthinking might draw unfounded conclusions from limited evidence, leading to incorrect conclusions, such as assuming store labeled "CVS Pharmacy'
(3) Underthinking or Overthinking:
convenience store.
(4) Critical Decision Errors at Waypoints: Incorrect routing choices at pivotal junctions, leading to irreversible deviations. This occurs when the target is observed within the right field of view, yet the model erroneously selects the leftward path, or when perceptual inaccuracies lead the choice of non-viable route that cannot reach the target.
(5) Premature Termination or Overshooting: Over-anticipatory stopping before reaching targets or passing by the target leads to not reaching missing the target.
(6) Instruction Parsing Errors: Generation of malformed JSON outputs violating predefined action schemas (e.g., missing "action" fields), rendering trajectories unexecutable.
35

---

Published as a conference paper at ICLR 2026
VLM: GPT-4o City: New York
Visual Observation Errors
Trajectory (qidx: 63)
Start
A End
Real Path
Visual
Eftors
Pred. Path
Optimal path
Detour path
Failed path
Question: "Please nearest parking lot.",
Overall Observation: The scene depicts busy city intersection with tall buildings. There is an open parking
vehicles parked, which appears Perspective Observation: ('A': "The BACK view shows a row of commercial buildings." "B" "The RIGHT view shows straight road with urban buildings on both sides.", 'C"; "The FRONT view dlearly shows blue and suggesting yellow bus parked designated paved interaection.")
parking lot.", "D": 'The LEFT view
the leading away
Thoughts: The visual informatioon from perspective C shows designated area with parked vehicles, including bus. identify the target parking Therefore, have reached the destination.
Action:
Action Direction: Stop
Score: 0.95
VLM: GPT-4o City: Beijing
Underthinking or Overthinking
Trajectory (qidx: 32)
Start
A
Path
Pred. Path
Undorthinking Oveithinking
Optimal path
Detour path
Failed path
Question: 'Please find the nearest Apple Huawel." Observation: Image
city intersection with large, buildings on both sides. There is prominent building with large sign.
Perspective Observation: ('A': The FRONT
features large, modern building with glass windows shows leading
sign partially visible, resembling
store style.",
cluster commercaal buildings in the distance.")
Thoughts: The building on the FRONT has modern design often associated with tech companies, and the partially visible sign could related to Huawei. will that is the correct direction.
Action:
Action Direction: Step Forward
VLM: GPT-4o City
Beijing
Trajectory (qidx: 46)
Decision Errors at Waypoints
O Start
A
Real Path
Waypoint
Pred. Path
Optimal path
Delour path
Falled path
Question: "Please find the nearest Adidas or Nike
Overall Observation: The is complex urban intersection with multiple diverging roads. Tall
modern buildings surround the area, and there is a mix commo ercial and office structures.
Perspeclive Observation: ('A': "The LEFT view shows narrower side street lined with smaller shops and with heavy traffic large distance." 'C" "The BACK view leeds down wide evenue. The buildings the distance appear to become
FRONT leads
buildings
more sparse.) )
Thoughts: I need find a sportswear store like Adidas or Nike. The Perspective B seems lead towards major commercial district with large buildings, which Is likely location for flagship stores.
Action:
Action Diraction: Step Forward
Figure 19: Qualitative examples of primary VLM failure modes in City Seeker.
36

---

Published as a conference paper at ICLR 2026
Figure 18 compares two representative VLMs Qwen2.5-VL-32B and InternVL3-8B revealing distinct error profiles. Both exhibit significant critical decision errors (Type 4: 22.8% vs. 24.7%). highlighting persistent challenges in spatial reasoning navigation waypoints. The larger Qwen model demonstrates stronger visual perception (Type 1: 14.6% vs. 22.2%) but suffers more under- thinking/overthinking errors (Type 3: 32.9% vs. 27.8%). This discrepancy likely stems from the model's 's lack of human-lite urban living experience, which limits its ability to discern primary/sec- ondary affordances of POIs. Additionally, the model occasionally draws unfounded conclusions from sparse visual evidence potentially due to overcompensation for missing contextual knowledge. Conversely, Intern VL3-8B struggles with instruction parsing (Type 10.8% vs. 6.3%) and basic visual recognition, with more misidentifications of street POIs and hallucination. This contrast reveals a scale-driven tradeoff: larger models develop nuanced visual grounding but introduce complex reasoning failures, while smaller models face fundamental perception and instruction-complianc challenges.
Deeper Dive: Human vs. VLM Failure Modes Through analysis of error trajectories and participant interviews, a deeper analysis of failure modes reveals a crucial distinction between human and VLM bottlenecks. As summarized in Table 15 VLM failures are predominantly cognitive, stemming from a lack of commonsense knowledge. In contrast, the dominant failure mode for humans is strategic. Faced with unfamiliar environments and strict step limit, humans possess the necessary world knowledge to understand the task, but often falter in devising an optimal exploratory plan. intoratioo-leadiin
This manifests inefficient, intuition-driven loops- or poor management of the step budget, such as overshooting a valid target.
forgotten paths and repeated
Table 15: Detailed comparison of primary failure modes between humans and the top-performing VLM (Qwen2.5-VL-322B)
Failure Mode Cate- gory
Human Error Profile & Rationale
Model Error Profile & Rationale (Qwen2.5-VL-32B)
Strategic & Naviga- tional
60.7% (Primary Human Failure) Humans understand the goal but strug- gle with devising an optimal ex- ploratory plan. Failures intuition- driven and stem from:
40.5%
The model's errors here are less strate- gic. It does not suffer from imperfect memory but still misjudge distance to a visible target or fail to recognize it, leading to premature stops or passing the goal.
(e.g.. Termina- tion/Overshooting)
1. Inefficient Exploration: Partici- pants noted that poor signage visu- ally similar streets often led to confu- sion and forgetting previously explored paths, causing inefficient loops that ex- haust the 35-step limit.
2. Overshooting: A human might find valid target but continue walking past it "just to see what's around the next cor- ner," reasonable exploratory behavior that fails under strict step budget.
Cognitive Failures (e.g.. Under/Over- thinking)
19.1%
32.9% The model's key bottleneck. strug- gles to infer non-obvious functions of POIs and lacks the real-world experi- ence to make flexible logical lcaps, re- vealing a critical gap in commonsense reasoning.
Humans rarely fail basic inferences. Fr- rors occur on nuanced tasks, such as failing consider convenience store's secondary function of recharging phone credit, or overthinking whether a restau- rant is truly "upscale".
Visual & Execution Errors
20.2%
26.6% %
In an unfamiliar city and under pressure, human perception can falter. This is exacerbated by language barriers (e.g., non-native participants processing En- glish storefront signs) and cultural un- familiarity with local brands, leading overlooked or misidentified cues.
The model prone mix of errors: misidentilying visual cues (14.6%), fail- ing to adhere to the output format (6.3%), or having disconnect between its visual observation and textual ratio- nale (5.7%).
(c.g., Visual, Parsing, Mismatch)
37

---

Published as a conference paper at ICLR 2026
This contrast reveals fascinating trade-off. An Al agent has theoretical advantage with its perfect memory, which should prevent the inefficient looping that humans are prone to. However, this is counteracted by its lack of deep, intuitive commonsense. While humans possess the necessary real-world knowledge to understand the implicit needs, they are hampered by imperfect memory and intuition-driven approach to planning that can be inefficient in unfamiliar environments. This suggests that future work should focus not just on improving VLM perception, but on endowing them with more robust, human-like intuitive and strategic reasoning capabilities.
H LIMITATIONS AND FUTURE WORK
Despite the advancements enabled by CitySeeker, our study highlights several limitations of current VLMs in embodied urban navigation, which in turn point to promising directions for future research.
First, generalization and adaptability to specific contexts remain challenges. Physically, while models perform modestly in familiar urban layouts, their effectiveness declines in irregular road networks due to training data biases. Semantically, our framework currently relies on foundational spatial consensus and does not yet account for personalized user preferences or historical behavioral patterns, limiting the model's ability to provide tailored assistance.
Second, long-con contect reasoning and memory retention are key bottlenecks. Long- Long-horizon naviga- tion requires processing extensive sequences of visual and textual information, which consumes large number of tokens. This poses a challenge for models with context length limitations and can lead to the forgetting of crucial information from earlier steps, resulting in inefficient paths with loops and repeated errors.
Third, our ablation study on map-augmented navigation (Appendix D revealed a fundamental weak- ness in aligning 2D maps with and distance on the 2D map often led to confusion and hallucinations, degrading performance rather than improving it. This highlights critical gap in the models' ability to fuse these two distinct modes of spatial information.
street views. The models' poor cognition of orientation
Lastly, real-time adaptability is limited by computational inefficiencies. The significant latency in VLM decision- makinng stemming from redundant visual processing, currently restricts practical deployment in real-time applications.
Addressing these limitations requires a multi- faceted approach. Future work should focus on:
Enhancing Commonsense and Affordance Reasoning: key area for improvement is the model's understanding of functional affordances (e.g., that a café provides Wi-Fi), weakness particularly evident in smaller models. New training methodologies could focus enriching models with this kind of real-world, commonsense knowledge.
Integrating Personalized Behavioral Priors: While CitySeeker focuses on universal common sense, real-world navigation is often personalized. A promising future direction is to incorporate behavioral priors derived from GPS traces or check-in data. Once VLM demonstrates robust spatial common sense, these empirical signals can be used to rank valid candidates, moving from general capability to personalized assistance.
Improving Geometric and Orientational Understanding: Future research should ex- plicitly target the model's ability to understand orientation and perform "mental rotations". bridging the gap between abstract 2D map representations and the first-person visual world.
Developing More Efficient Architectures: Innovations in persistent memory architectures and more efficient inference frameworks are needed tackle the challenges of long- long-horizon reasoning and real-time performance.
I LLM USAGE STATEMENT
In preparing this manuscript, LLM was used as a general-purpose assistive tool. Its application was confined to the final stages of writing, specifically for minor polishing and refinement of the English prose to improve clarity and readability. The LLM did not contribute to the research ideation, experimental methodology, data analysis, or the generation of results and conclusions in this work.
38


Reviews:
[{"review_id": "tSPAMmKrEs", "reviewer": "Reviewer_agkw", "summary": "This paper introduces CitySeeker, a comprehensive benchmark designed to evaluate the spatial reasoning and decision-making capabilities of vision-language models (VLMs) in the context of embodied urban navigation for addressing implicit user needs. Extensive evaluations across a wide range of VLMs reveal key limitations in long-horizon reasoning, and the authors propose effective strategies to improve model performance.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Clear and Well-Structured: The paper is well-organized, with thorough explanations of the data collection process, benchmark design, and task formulation.\n\n- Novel and Interesting Setting: The paper proposed the task of embodied urban navigation guided by implicit human needs. This task is currently not widely explored and has significant potential for real-world deployment.\n\n- Extensive Evaluations: A wide range of VLMs are evaluated on the curated benchmarks, accompanied by comprehensive analysis and discussion.\n\n- Actionable Insights: The authors propose concrete strategies to enhance VLM performance, offering insights for real-world deployment scenarios.", "weaknesses": "I don't find significant weaknesses in this submission. However, I do have some concerns as follows. Therefore, I give a conservative score of borderline accept. I may consider increasing the rating if the authors adequately address these concerns.\n\n- Open-Source Model Superiority: The paper observes that open-source VLMs (such as Qwen) occasionally outperform the proprietary VLMs. The submission would benefit from a deeper analysis of the underlying reasons behind this phenomenon.\n\n- Presentation Issue: Some figures in the submission (e.g., Figures 5 and 6) require further refinement to improve readability and visual clarity.\n\n- The manuscript would benefit from including illustrations of typical failure cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hBolT0N2zq", "reviewer": "Reviewer_Xv86", "summary": "This paper introduces CitySeeker, a framework that leverages VLMs to understand and predict human urban mobility behaviors based on natural-language queries. The system maps user needs to POI categories using multimodal embeddings and evaluates performance on multiple cities. The goal is to test VLMs’ ability to align semantic and spatial reasoning in real-world city environments.", "questions": "How is user intent variability modeled, are there multiple valid POIs for the same need, or just one ground truth?\n\nHow does the model handle ambiguous or multi-intent queries?\n\nIs the need-POI mapping empirically validated with real mobility data (e.g., GPS traces, check-ins)?\n\nCould the framework integrate behavioral priors (e.g., time-of-day, personal preferences) to better capture real-world decision patterns?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Addresses an interesting interdisciplinary question.\n\nThe dataset construction across multiple major cities, combining geospatial and textual information.\n\nThe evaluation is systematic, covering both semantic matching and spatial reasoning tasks.\n\nThe idea of connecting natural language intent to spatial decision-making is novel and potentially impactful.", "weaknesses": "The mapping from need to POI type assumes a fixed, deterministic relationship, which may not hold in practice, human intent is subjective and context-dependent.\n\nThe model implicitly assumes people choose the shortest path or most direct POI option, which is unrealistic, behavioral factors like preference, familiarity, and accessibility play major roles.\n\nCross-cultural generalization is a concern: the same “need” may imply different POIs across societies.\n\nThe paper lacks an analysis of cultural and linguistic bias, despite using mixed data from Beijing and New York.\n\nThe need-to-POI mapping seems to reflect designer bias rather than emergent patterns from real user behavior.\n\nNo mention of whether the system can adapt to multi-intent or ambiguous queries.\n\nIt’s unclear whether the evaluation reflects real mobility choices or just semantic alignment accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZkLfuYJnl2", "reviewer": "Reviewer_1jEc", "summary": "As the ability of vision-language models (VLMs) to interpret *implicit human needs* in dynamic urban environments remains underexplored, this paper proposes CitySeeker, a novel benchmark designed to evaluate VLMs’ spatial reasoning and decision-making capabilities in *embodied urban navigation* tasks that involve implicit objectives. CitySeeker includes 6,440 trajectories across 8 cities, capturing diverse visual characteristics and implicit needs within 7 goal-driven scenarios. Extensive experiments demonstrate that even top-performing models (e.g., Qwen2.5-VL-32B-Instruct) achieve only 21.1% task completion, revealing fundamental weaknesses in long-horizon reasoning, spatial cognition, and experiential recall. To further diagnose these issues, the authors explore a set of human-inspired strategies—Backtracking Mechanisms, Enriched Spatial Cognition, and Memory-Based Retrieval (BCR)—reflecting iterative observation-reasoning cycles and adaptive path optimization in human navigation. The analysis provides valuable insights into developing VLMs with more robust *spatial intelligence* for tackling “last-mile” navigation challenges.", "questions": "1. Please clarify the ethical compliance issue.\n\n2. The paper claims that these navigation tasks are highly challenging even for humans (e.g., “humans achieve only 5.7% accuracy”). However, tasks such as finding a restaurant or locating a place with Wi-Fi do not seem inherently difficult for human participants. The paper does not adequately explain why humans perform so poorly, or how the tasks are defined, constrained, or evaluated. This undermines the interpretability and credibility of the benchmark’s human baseline.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The motivation of this paper is well-grounded. Identifying that VLMs’ ability to interpret implicit human needs in dynamic urban environments remains underexplored is both timely and significant. It provides a new angle for examining VLMs’ world knowledge and decision-making capabilities.\n\n2. The paper is clearly written and visually appealing. The figures effectively illustrate the framework and experiments, helping readers understand the design and reasoning process.\n\n3. The authors conduct extensive experiments involving 27 different VLMs, and their findings are deep and thought-provoking, revealing critical gaps in current models’ embodied spatial reasoning.", "weaknesses": "1. Overall, this is a clear accept-level paper in terms of novelty, clarity, and experimental depth. However, there is a **serious ethics concern**. The paper states:\n   *“CitySeeker dataset was sourced from publicly available APIs (Google Maps and Baidu Maps) and is used in accordance with their terms of service for non-commercial research purposes only.”*\n   After reviewing Google Maps’ Terms of Service[1], it explicitly states:\n   **“Downloading Street View images to use separately from Google services (such as an offline copy) is prohibited. These restrictions apply to all academic, nonprofit, and commercial projects.”**\n   This implies that **the dataset collection may violate Google’s ToS**, introducing a significant **ethical and legal issue**. Consequently, the dataset **cannot be publicly released**, which severely limits the reproducibility and extensibility of this research. This is a **veto-level weakness** for a top-tier venue.\n\n[1] [Brand Resource Center | Products and Services - Geo Guidelines](https://about.google/brand-resource-center/products-and-services/geo-guidelines/)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1fI8TZbHKs", "reviewer": "Reviewer_4hXJ", "summary": "This paper introduces *CitySeeker*, a novel benchmark for evaluating Vision-Language Models (VLMs) in embodied urban navigation driven by implicit human needs. Unlike prior VLN benchmarks that rely on explicit instructions, *CitySeeker* focuses on abstract, functional, and semantic goals (e.g., “I’m thirsty”), spanning 6,440 trajectories across 8 cities. The authors propose a framework for evaluating spatial reasoning and decision-making and identify key bottlenecks in current VLMs. They further introduce three human-inspired strategies—Backtracking, Spatial Cognition Enrichment, and Memory-Based Retrieval (BCR)—to enhance navigation performance.", "questions": "**Questions**: See weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- **Benchmark Contribution**: *CitySeeker* fills a critical gap in VLN research by targeting implicit human needs in dynamic, real-world urban environments. It is the first large-scale benchmark to do so across multiple cities and diverse task categories. \n- **Realism and Diversity**:  The benchmark includes diverse urban layouts and visual characteristics, making it highly relevant for embodied AI applications. \n-  Comprehensive Evaluation： \n- **Comprehensive Evaluation**: The paper presents extensive empirical results across 27 VLMs, including proprietary and open-source models. The analysis spans task categories, cities, and trajectory patterns. The authors also  proposed BCR strategies are well-motivated and show measurable improvements in task completion and path efficiency.\n- **Clarity and Presentation**:  The paper is generally well-written, with a clear logical flow and a well-motivated problem statement.", "weaknesses": "1.  **Problem Setup**:  \n\n    - The task formulation leans more toward object navigation than traditional VLN. In partially observable environments, defining the agent’s state solely as the current observation $o_t$ is insufficient.\n    - A more appropriate formulation would involve belief state estimation using historical observations $\\{o_0, o_1,...,o_t\\}$, which is a regular practice in POMDP-based problem.\n\n2. **Symbol and Notation Clarity**:\n\n    - The definition of $v$ as a graph node and $v_t$ as the location is inconsistent.\n    - Subscripts such as $o_i$ (different views at a single node) and $o_t$ (timesteps) are used ambiguously.\n    - Lacks a clear explanation of how reasoning $\\phi$ and action $a$ are computed -- what inputs are used ($O_t$ or $S_t$ or something else?).\n\n4. **Evaluation Metrics Misalignment**:\n\n    - Metrics like **nTCE**, which measure trajectory deviation from ground truth, are more suitable for instruction-following tasks (in previous VLN tasks, the agents are asked to follow the excact GT trajectory given a detailed user instruction).\n    - In implicit-need-driven navigation, multiple valid trajectories may exist. Metrics like SPL (Success weighted by Path Length) or goal proximity might be more appropriate.\n\n\n5. **Low Task Completion Rates**:\n\n    -  Even the best-performing models and human baselines achieve low success rates under strict metrics (e.g., 5.7% TCE for humans), raising concerns about benchmark difficulty or metric suitability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)