Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?
Authors: Pingyue Zhang
Zihan Huang
Yue Wang
Jieyu Zhang
Letian Xue
Zihan Wang
Qineng Wang
Keshigeyan Chandrasegaran
Ruohan Zhang
Yejin Choi
Ranjay Krishna
Jiajun Wu
Li Fei-Fei
Manling Li
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Mode
Vision-Language Model
Spatial Reasoning
Spatial Agent
Active Exploration
Primary area: datasets and benchmarks

Abstract: Spatial embodied intelligence under partial observability requires agents to actively acquire missing information rather than passively consume complete observations.
While multimodal foundation models excel at passive perception and reasoning, their ability to support active, self-directed exploration to build and maintain a coherent spatial belief remains unstudied.
We therefore propose Theory of Space, defined as an agent's ability to construct, revise, and exploit a spatial belief through self-directed active exploration under partial observability.
We implement Theory of Space using a benchmark with textual and visual environments. Rather than solving specific tasks, the goal is curiosity-driven exploration to build a complete, accurate spatial belief. 
A core innovation is spatial belief probing: we prompt it to reveal its internal spatial belief as a cognitive map at each step, letting us measure the quality of its underlying spatial belief.
Our evaluation of state-of-the-art models on a suite of downstream tasks reveals critical bottlenecks: (1) \textbf{The Active-Passive Gap}: Performance degrades when agents must autonomously gather information (e.g., \textsc{GPT-5.2}: $57.1{\to}46.0$); (2) \textbf{Inefficiency}: Models explore in an unsystematic way and with high redundancy, failing to match the efficiency of program-based proxies while producing no better results.
Through belief probing, we diagnose that perception acts as an initial bottleneck, yet global beliefs suffer further from \textbf{instability} that causes spatial knowledge to degrade over time. 
Finally, a false belief paradigm reveals \textbf{Belief Inertia}: agents fail to overwrite obsolete priors, an effect especially severe in vision-based models.

Published as a conference paper at ICLR 2026
THEORY OF SPACE: CAN FOUNDATION MODELS CON- STRUCT SPATIAL BELIEFS THROUGH ACTIVE EXPLORATION?
Pingyne Zhangl.*,, Zihan Huang", Yue Wang"*, Jieyu Zhang*,, Letian Xue', Zihan Wang', Qineng Wang', Keshigeyan Chandrasegaran', Ruohan Zhang?, Yejin Choi", Ranjay Krishna', Jiajun Wu', Li Fei-Fef", Manling Li
Northwestern University Stannord University
3 University of Washington 'Corell University pingyuezhangêu www.oowwwtttteeeeeeeem edu, manl manl.ngg.i www.g......oteeeeeeeeeeeeee ii@northwestern....
"Equal contribution
CCorresponding author
ABSTRACT
Spatial embodied intelligence under partial observability requires agents to actively acquire missing information rather than passively consume complete observations. While multimodal foundation models excel at passive perception and reasoning, their ability to support active, self-directed exploration to build and maintain a coherent spatial belief remains unstudied. We therefore propose THEORY OF SPACe, defined agent's ability construct, revise, and exploit spatial belief through self-directed active exploration under partial observability. We implement THEORY OF SPACE using a benchmark with textual and visual environments. Rather than solving specific tasks, the goal is curiosity-driven exploration to build complete, accurate spatial belief. A core innovation is spatial belief probing: we prompt it to reveal its internal spatial belief as a cognitive map at each step, letting measure the quality of its underlying spatial belief. Our evaluation of state- of-the-art models on suite of downstream tasks reveals critical bottlenecks: (1) The Active-Passive Gap: Performance degrades when agents must autonomously gather information (e.g., GPT-5.2: 57.1-46.0)); (2) Inefficiency: Models explore in an unsystematic way and with high redundancy, failing to match the efficiency program- based proxies while producing no better results. Through belief probing, we diagnose that perception acts as an initial bottleneck, yet global beliefs suffer further from instability that causes spatial knowledge degrade over time. Finally, a false belief paradigm reveals Belief Inertia: agents fail to overwrite obsolete priors, effect especially severe in vision-based models!.
INTRODUCTION
Spatial embodied intelligence relies on active exploration. Unlike disembodied systems that pas- sively process fixed observations, an embodied agent could take actions to alter its position in the environment as exploration, selectively acquiring observations needed to construct spatial knowledge for various spatial tasks. Cognitive science shows that such active exploration leads to substantially better spatial understanding than passively receiving the same information, even when observations are identical (Held & Hein, 1963; Chrastil & Warren, 2012; 2013). But exploration isn't simply about collecting more observations. is about efficiency, acting under uncertainty to target what is unknown ambiguous in the agent's 's spatial belief and maximize information gain.
We propose THEORY OF SPACE as a framework that explicitly treats exploration as a first-class decision decision-making problem, decoupled from any single downstream task, focusing opening the box of the agent's 's internal spatial belief. Just as Theory of Mind (ToM) measures how agents model the hidden mental states of others, THEORY OF SPACE assesses an agent's 's ability to model the hidden physical structure of the world. We define THEORY OF SPACE as an embodied agent's 's ability to actively construct, revise in a dynamic environment, and exploit internal spatial belief formed through active exploration. Beyond end-task evaluation, THEORY OF SPACE directly probes what the agent knows, what remains uncertain, and how effectively its actions reduce those uncertainties,
'The code and data are publicly available at ttttps://theoryoofffi space. . gi thub. 1o/

---

Published as a conference paper at ICLR 2026
Environment Topdown
Active Exploration
Spatial Belief
90; Ooserve
Evaluation (Palrwisc):
Where reletvv
mont-right, mid distance
Northwast, sightly for
Turn 90; Observe
Evaluation (Rotation): Object onder rotate dioclaite
Biatront, mic distance Turn 180; to R
Observe
Representation
abject 20
front, near
90;
IT front-left, mid distence
vision observation
frone,
observation
Turn 180; Observe
predicted 2D coord
front, mid distance
TL 90; Obseree
agent action
Tront-lft, mid distancc
Figure 1: THEORY OF SPACE: active exploration, belief probing, and evaluation. Left: Agent trajectory under partial observability. Middle: Action loop across text/vision environments with egocentric observations. Right: Evaluation via spatial task exploitation and cognitive map probing.
measured by the number of exploration steps and the uncertainty resolved per action. Figure 1 provides overview of THEORY OF SPACE'S active exploration, belief probing, and end-task evaluation.
We apply THEORY OF SPACE to evaluate multimodal language models, which promising can- didates for embodied agents. By integrating vision and language, they support unified perception, reasoning, and action over time, yet existing foundation-modee benchmarks offer little insight into these capabilities. Most current benchmarks fall into two categories: passive (Weston et al., 2015; Shi al., 2022; Yang et al., 2025c; Gholami et al., 2025; Yang et al., 2025a), where the agent is only asked to reason over given observations, and task-driven (Gordon et al., 2018; Shridhar et al., 2020b; Li al., 2025; Yang et al., 2025b), where the agent must achieve specific goal (e.g.. "find the red chair").
We propose to systematically evaluate the active process of spatial belief construction. Unlike passive benchmarks, our THEORY OF SPACE benchmark requires agents to actively explore via moving, rotating, and observing to build coherent global beliefs. We implement scalable environment using ThrleDWorld (Gan et al., 2021) and Objaverse (Deitke et al., 2022) with text-based and vision-based worlds to localize perception versus reasoning failures. After exploration, we evaluate along two axes: belief exploitation via spatial downstream tasks probing route-level and survey-level knowledge (Siegel & White, 1975; Montello, 1998), and exploration efficiency via steps taken and information gain time. We further design scripted proxy agents disentangle exploration from reasoning. Our evaluation of state-of-the-art foundation models reveals promising capability in the text world but striking limitations in the vision world. Active exploration remains primary bottleneck: models perform reasonably well passively but degrade when actively gathering information (c.g.. GPT-5.2: 57.1 + 46.0; GEMINI-3 PRO: 60.5 57.3). Moreover, we also find a major efficiency gap: rule-based proxy agents reach target coverage in ~9 steps, whereas foundation models explore redundantly, requiring& 2 14 steps without improving belief accuracy.
Beyond downstream task scores, a core contribution of THEORY OF SPACE is explicit cognitive-map probing, which directly exposes the agent's 'ss evolving spatial belief during exploration rather than treating the model as black box. This enables fine-grained diagnosis of how models represent space, revealing that while perception acts as an initial bottleneck, global beliefs further suffer from instability, causing knowledge to degrade over time. We also introduce False Belief paradigm that alters the environment after initial exploration, uncovering spatial belief inertia: agents persistently retain obsolete spatial priors even after directly observing the updated configuration, revealing critical failure in spatial memory revision.
An important direction for future work is to extend THEORY OF SPACE beyond single-agen settings to multi-agent exploration, where additional challenges arise around coordination and aligning (or sharing) spatial beliefs across agents.
2

---

Published as a conference paper at ICLR 2026
2 THEORY OF SPACE
To build agents with spatial intelligence, we argue for evaluating not merely passive reasoning, but the active, self-directed construction of spatial belief from partial observations. We introduce THEORY OF SPACE, a conceptual counterpart to Theory of Mind (ToM). While ToM models hidden mental states of others, THEORY OF SPACE models uncertain, currently unobserved structure of space.
E
Definition: THEORY OF SPACE
Ability to construct, revise, and exploit an internal spatial belief.
Here, an internal spatial belief is a mental model (Taylor & Tversky, 1992) of spatial layout and relations maintained in working memory and updated from partial observations. We provide the formal definition of THEORY OF SPACE in Appendix A.I.
2.1 A PARADIGM FOR ASSESSING THEORY OF SPACE OF LARGE FOUNDATION MODELS
We propose new paradigm for Assessing THEORY OF SPACE of large foundation models, which consists of three essential components below.
Task-Agnostic Active Exploration. Evaluating THEORY OF SPACE requires a shift from downstream task completion to exploration, where the agent decides what to see next. We place the agent in a partially observable environment and challenge to actively select actions (moving, rotating, observing, and terminating) to build a general-purposs. internal spatial model through self-directed exploration. This process encompasses both Belief Construction and Belief Revision. Inspired by the false belief paradigm in Theory of Mind (Wimmer & Perner, 1983) and spatial belief revision (Knauff et al., 2013), we evaluate whether agents can detect environmental changes and correctly revise their internal beliefs during exploration. The agent must identify what remains uncertain and terminate only upon acquiring sufficient evidence to form an accurate internal map.
Belief Exploitation Assessment. To translate THEORY SPACE into concrete evaluation tasks, we draw insights from the development of spatial representations (Siegel & White, 1975; Montello, 1998) and define two tasks to measure an agent's ability to exploit its internal belief for goal-directed behavior: (1) Belief on Route evaluates path-based understanding of space organized around landmarks such as pairwise spatial relationships along a egocentric navigation; (2) Belief on Survey assesses map-like "bird's-eye view" that represents space allocentrically, allowing for the inference of global relationships.
Explicit Probing of the Internal Spatial Belief. Behavioral success cannot directly reveal the quality of an agent's internal spatial model. We therefore require the agent to explicitly externalize its belief by probing its cognitive map at each turn. Cognitive maps are structured allocentric representations of space, well-established in neuroscience (Tolman, 1948; O'Keefe & Dostrovsky, 1971; Hafting et al., 2005), and serve our canonical representation of hidden spatial structure. We evaluate map correctness and diagnose reasoning with dynamic signals capturing how reliably observations are integrated, tracked over time, and kept coherent across local and global structure. We further probe uncertainty modeling by requiring the agent to identify unobserved regions. This shifts evaluation from behavioral success direct assessment of representational competence.
3 BENCHMARKING THEORY OF SPACE ABILITY FOR FOUNDATION MODELS
Unlike task-driven benchmarks that only test task completion, we aim to answer "can the agent form a global environmental belief through exploration?". We structure the benchmarking into two phases. the Exploration Phase the agent interacts with the environment to construct spatial belief by selecting and executing actions in the action space in & 3.1, and gather sequence of local observations to integrate them into a unified spatial belief. In the Reasoning Phase II, the agent is asked to conduct spatial tasks (detailed in 3.2).
3.1 SPATIAL ENVIRONMENT CONSTRUCTION
To ensure controlled experimentation, we procedurally generate multi-room indoor layouts on an N M grid. Each scene is populated with indoor objects, each assigned 2D integer coordinate
3

---

Published as a conference paper at ICLR 2026
and a cardinal orientation from (N, S, E, W). The agent begins at a random position, is informed of the total number of rooms and the names of all objects in the scene, and then starts exploration. Following the Gym-style interface (Brockman et al., 2016), define highly scalable environments in which each random seed deterministically instantiates a distinct multi-room layout.
Action Space in the Environment. The agent's interaction with the world is designed focus on high-level decision makinn rather than low-level motor control: Goto to move directly to a currently visible object; Rotate to turn in place by 90", 180", or 270"; Observe perceive visible objects in the 90" field of view; and Query to obtain a visible object's absolute 2D coordinates. We additionally assign a cost of to Observe and Query, encouraging Query to be used only when necessary to resolve ambiguity. However, across all models Query is rarely invoked, so we only focus on Observe and measure exploration efficiency by step count instead of cost.
Observation Feedback from Text- Vision Parallel Environment. We offer both text- and vision- based environments, enabling diagnostic analysis of spatial reasoning. Each Observe action returns both textual and visual feedback from 90 field of view. The Text World provides symbolic observations with discrete bins for direction and distance (e.g., "chair is front-left and near", detailed below), isolating pure spatial reasoning. The Visual World instead supplies ego-centric RGB images rendered in ThreeDWorld (Gan et al., 2021) with Objaverse assets (Deitke et al., 2022), requiring perception to recover spatial relations. To calibrate perception in the visual setting, we provide two reference images, indicating unit distance (1 grid unit) / angle (a 22.5 angular cone), and showing all objects with their "front" orientation respectively. Details are shown in Appendix B.1
Spatial Relation Representation. To ensure that agents perceive and communicate about space using consistent language across tasks and modalities, we discretize spatial relationships for directions and distances. For allocentric direction, we discretize into eight 45 bins aligned with the four cardinal and four intercardinal directions, denoted compactly as {N, NE, E, SE, S, SW, W, NW). Each bin spans 45 around its heading (e.g., N = 22,,5% 22.5")) )) For egocentric direction. within a 90º forward field of view (FOV), we use five labels: front-
[-45% -22.5%) frent- -s1 light- -efft (0, 22.5), and front-right (22.5", 45%). For distance, measured map units independent of direction, we define six bins: same = 0, near (0, 2]. mid (2, 4), slightly far (4, 8). (8, 16), and very far (16, 32].
22.5", 0), front 09, tront-sl. wellihht-rr
3.2 DOWNSTREAM SPATIAL TASKS
We use open-ended questions rather than multiple-choice questions to reduce the risk of knowledge leakage. Drawing prior work (Siegel & White, 1975; Montello, 1998), define tasks to evaluate an agent's Route and Survey knowledge. Route belief captures how an agent encodes paths and spatial relations from an egocentric step-by-step perspective. Survey belief is map-like, allocentric representation. Under route belief, the static task requires reporting direction and distance between two objects (direction); forward dynamics tasks ask the model to adopt object's perspective (persp.take) or predict the final observation after executing a Got o/Rot at sequence (act2view): backward dynamics tasks infer which perspective is currently adopted (perc.dec) or recover the action sequence yielding target observation (view2act). Under survey belief, the static task reconstructs global coordinates for all objects (alloc.map); forward dynamics tasks predict tront-facing objects during full self-rotation (ment.rot) or infer the observation from a given global pose (loc2view); the backward dynamics task localizes the agent from target observation (view2loc). Details of these downstream tasks are shown in Appendix A.2.
3.3 ASSESSMENT DIMENSIONS
We define assessment dimensions that align with the core THEORY OF SPACE abilities: construction and revision are evaluated via exploration efficiency and belief quality, while exploitation is evaluated via task success. The detailed definition of assessment dimensions is shown in $A.3
(D1) Belief Construction Efficiency. Measures how efficiently the agent collapses spatial uncertainty. Quantified via normalized information gain (ratio of eliminated to total possible object positions, ranging from 0 to 1).
Belief Representation and Quality Assessment. We decompose the cognitive map into: (D2) Cognitive Map (Observed), measuring fidelity and coherent integration of observations, evaluated
4

---

Published as a conference paper at ICLR 2026
via Correctness (composite of positional, directional, and facing accuracy) and dynamic diagnostics; and (D3) Uncertainty Map (Unobserved), measuring how well the agent identifies unobserved regions, evaluated via Fi of selecting unexplored over candidate positions.
(D4) Belief Revision. Measures the agent's ' ability to revise spatial beliefs under covert environ- mental changes. Evaluated via the False Belief task ($5.2), scored by detection F1 (identity and transformation type), with Belief Inertia quantifying bias toward obsolete priors.
(D5) Belief Exploitation Success. Measures task success when the agent must utilize its spatial belief. Relation- tasks (direction, perspective-taking, action2view) score direction and distance separately (0.5 each); coordinate-output tasks (view2loc, alloc.map) use coordinate similarity.
3.4 EXPLORATION STRATEGIES
To rigorously evaluate spatial cognition, we distinguish between two capabilities: the ability to acquire information (exploration) and the ability to synthesize it (reasoning). We present two evaluation settings: (i) Active Exploration, where the agent must plan actions to reduce uncertainty, and (ii) Passive Comprehension, where the agent reasons over standardized logs generated by scripted proxies.
Unnertainty-Driver On-Policy Exploration. We conduct active evaluation to understand agent ability in exploring the environment to gather necessary information in building spatial belief. In this setting, the evaluated agent must plan and execute its own informationngatthree each step, the agent selects an action based on its observation history and current objective, then receives new observations (text or image). Exploration continues until the agent issues an exploration termination reaches the step budget. Success requires balancing two goals: maximizing coverage of unknown relations while minimizing action cost. This setting directly reveals whether the agent can recognize what it does not yet know and actively reduce uncertainty through exploration.
policy. At
Passive Exploration via Scripted Proxy Agents. Evaluating THEORY OF SPACE also requires disentangling exploration quality from spatial reasoning ability. To isolate the latter, we introduce proxy agents that supply fixed observation stream to evaluated models, eliminating variance from exploration failures and enabling fair comparison of core reasoning across architectures. We design two scripted proxies: SCOUT (visual environments) rotates exhaustively at each location to ensure full object coverage, and STRATEGIST (text environments) follows belief-driven viewpoint selection policy to maximally reduce relational ambiguity. Implementation details appear in Appendix B.2.
4 EVALUATION AND ANALYSIS
We evaluate set of state-of-the-ar proprietary and open-source foundation models. They are evaluated on both passive and active settings described in 3.4. Unless otherwise specified for ablations, all experiments use three connected 6 x 6 rooms with 4 objects in each (total 12 objects). To enable a like-for- comparison between the text and vision settings, we instantiate identical room layouts across modalities. We use 384 x 384 images in the vision setting. We generate 100 scenes and create three questions per task per scene, yiclding 3 x 9 X 100 = 2700 questions per setting. We mainly evaluate six foundation models: GPT-5.2 (OpenAI, 2025), GEMINI-3 PRO (Google, 2025), CLAUDE-4.5 SONNET (A (Anthropic, 2025), GLM-4.6V (Zhipu AI Team, 2025), QWEN3- VL. (Bai et 2025) (235B-A22B- -20B--22TTThhi and INTERNVL-3.5 (Wang et al., 2025) 241B-A28B) For closed-source reasoning models GPT-5.2, GEMINI-3 PRo, and temperature and the maximum number of tokens 32768. For all other models, we set the temperature to O. INTERNVL-3.5 supports at most 10 images, so we omit it for the vision-based world setting.
we set the
Active Exploration Results. We evaluate models as active agents, where they must autonomously explore the environment to build their spatial belief and terminate the exploration process by their own This setting tests the full THEORY OF SPACE pipeline, requiring the agent to simultaneously ceermaationgathhrriie
plan an efficient cognitive map under uncertainty. The agent's performance is measured by its Exploration Efficiency as shown in $ 3.3 and its final accuracy on the downstream spatial tasks. The agent has a maximum of 20 exploration steps. Table presents the active performance of the models, providing holistic view of their ability to translate curiosity into knowledge. Figure illustrates information gain over the course of the exploration turns. GPT-5.2 acquires substantial information early on, but its rate
trajectory, integrate observations, and maintain a coherent
5

---

Published as a conference paper at ICLR 2026
direction
persp take
perader.
act
view Bect
alloc.map masp
ment.y
locvveew
view2loc
Static (5)
Dynamic
Static (S)
Dynamic (D)
Methods
Avg.lep
Route
Survey
Avg.
Passi Arg
Viston-based World
Proprictury Models
GPT-5.2
17.2
40.0
36.7
56.2
43.8
40.3
43.4
59.7
56.9
37.8
46.0
57.1
GEMINI-3
13.6
68.2
47.2
63.5
73.0
52.2
57.3
60.5
CLAUDE-4.5 4.5 SONNET
19.6
23.7
23.3
18.7
33.3
10.7
37.4
34.7
33.7
50.9
29.6
43.1
Open source Models
GLM-4.6V
15.0
18.5
3.3
14.4
OWEN3-VL
16.3
16.8
23.3
13.4
24.8
5.7
25.8
16.3
21.5
43.7
21.3
24.9
HUMAN
9.8
94.5
100.0
100.0
100.0
93.4
93.4
100.0
100.0
86.7
96.4
HUMAN TOOL?
11.1
100.0
100.0
100.0
100.0
97.8
100.0
100.0
100.0
93.4
99.0
Text-based World
Proprietary Models
GPT-5.2
80.3
77.9
GUMINI-3 PRO
13.5
78.0
79.2
90.6
75,3
76.3
81.0
94.0
83.3
76.2
81.5
86.5
CLAUDE-4.5 SONNET
18.7
65.3
65.3
79.0
62.7
51.7
68.8
76.3
57.0
67.0
65.9
73.6
Open Models
GLM-4.6V
14.5
20.8
19.7
12.7
21.8
3.7
13.9
9.3
22.7
26.2
16.8
23.4
IN7ERNVL-3.5
15.0
28.8
44.8
26.0
36.8
7.3
31.0
27.7
33.8
38.9
30.6
37.4
OWEN3-VL
36.8
HUMAN
10.8
87.8
82.1
100.0
85.5
86.8
06.6
100.0
95.6
75.8
86.7
HUMAN WITH TOoL"
12.8
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
91.2
99.0
Table I: Exploitation Performance (%) of Belief Construction via Active Exploration. Models autonomously plan actions and are evaluated on exploration cost and reasoning across text- and vision-based environments. GEMINI-3 PRO leads almost every task. Humans outperform in both settings, especially in vision. "Humans can use instruments such as protractors and compasses to infer object positions precisely. We also include passive (pass.) results here for comparison.
of gain slows in later turns, resulting in lower cumulative information gain than GEMINI-3 PRO and CLAUDE-4.5 SONNET. Moreover, none of the models achieves full coverage relative to the proxy agent. We benchmarked three human subjects across five text and five vision scenes. Humans consistently outperformed foundation models in both domains, particularly in vision. Intuitively, humans scored higher in vision than as visual information is easier process. With tools, they achieved near-perfect accuracy.
Passive Exploration Results. We evaluate models trajectories generated by rule-based proxy agent to understand a model's core spatial reasoning ability regardless of its exploration strategy. The last column of Table 1 reports the average performance of each model both text-based and vision- based environments. The detailed experimental results are shown in Appendix C.I. As evaluated, the results show clear separation: GPT-5.2 and GEMINI-3 PRO lead by wide margin over other systems, particularly open- sourcs models. A substantial modality gap persists, with text performance far better than vision performance for all models.
Key Findings: Modality Gap
Modality Gap Exists: text significantly outperforms vision.
Text-based World
Accumulated Information Gain
Methods
2-room
4-room
1.0
pass.
steps
pass.
steps
GPT-5.2
92.3
77.8
6.2
86,5 66.0
16.4
Information Gain
0.8
GEMINI-3 PRO
86.7
80.6
6.2
81.2
77.7 19.7
Vision- based World
0.6
Methods
2-room
4-room
0.4
poss.
act.
steps poss.
act. steps
GPT-5.2
59.3
51.5
52.6
40.3 23.2
0.2
GEMINI-3 PRO 58.3
57.8
6.6
56.2
$1.5
19.7
06%
1
3
7
9
13
15
17
19
21
Table Exploitation Performance (%) for Multi-Room Settings (2-room and 4-room). pass. for passive avg acc, act. for active avg acc, steps for average steps.
Steps
Figure 2: Accumulated information gain over exploration steps in the text world.

---

Published as a conference paper at ICLR 2026
Overall, active accuracies underperform the passive setting. As shown in Figure 2, models such as GPT-5.2 gather information quickly but terminate prematurely, leaving unresolved uncertainty. Compared to the STRATEGIST proxy, which achieves full certainty, autonomous models remain less thorough. A second disparity efficiency. In the vision domain, the ScoUT proxy reaches target coverage in = 9 steps, whereas autonomous models expend significantly more actions with no performance benefit. In the text domain, following the concise SCOUT trajectories (= 9 steps), GPT-5.2 and -EMINI-3 PRO achieve accuracies of 83.9 and 86.7, surpassing their active exploration scores (72.0 and 81.5, Table 1). This confirms that models perform better when guided by efficient proxy paths than when exploring autonomously.
Different Room Seftings, For the two best- best-performing models, GPT-5.2 and GEMINI-3 PRO, we further evaluate performance under multi-room configurations (one four-room and two three-room settings). As shown in Table 2, performance declines and the active passive gap widens as room count increases. Notably, GEMINI-3 PRO requires nearly the same exploration steps text-only and vision-based environments. Detailed results are in Appendix C.2.
Key Findings: Active Exploration as the Bottleneck
Performance and Efficiency Deficit: Active agents score lower than reasoning on rule based program histories, and explore less efficiently than the program.
Incomplete Coverage: Active agent fails to achieve complete information coverage.
Complexity-Wii Widened Gap: The active versus passive difference grows with environment scale; GEMINI-3 PRO degrades least.
Exploration Pattern. Manual inspection reveals distinct behavioral patterns. GPT-5.2 explores unsystematically, prioritizing newly discovered doors and leaving rooms partially unexplored, result- ing object omission and path redundancy. GEMINI-3 PRO adopts methodical "rotate-and-scan" strategy before transitioning between rooms, mirroring the SCOUT proxy agent. Further examples are provided in Appendix D.2.
5 How DO FOUNDATION MODELS MANAGE INTERNAL SPATIAL BELIEF?
In this section, we use the THEORY OF SPACE belief-probing mechanism (as proposed in $2. 12 to diagnose how MLLMs manage internal spatial beliefs and move beyond treating the agent as a black box. Figure 3 shows the example of how we probe the belief of agent at each exploration step.
5.1 COGNITIVE MAP PROBING
Instead of treating the spatial belief as a black box, we probe the agent's internal state to distinguish verifying known facts from hypothesizing about the unknown. The agent externalizes its belief via structured JSON containing a Cognitive Map, which records observed objects.
Representation. For cognitive map, the agent presents its belief map serialized in structured JSON. The map maintains (i) a global layout anchored to the agent's s initial pose, and (ii) local snapshot that records only the currently visible objects with the current pose as origin to diagnose immediate perceptual errors.
single, allocentric cognitive
Metrics. We evaluate cognitive maps along three axes: positional accuracy (Euclidean similarity between predicted and true coordinates, scaled by object coverage), directional accuracy (pairwise directional relation accuracy), and facing accuracy (fraction of correctly predicted object orientations). Using global and local belief representations updated at each turn, we compute five diagnostic scores: Correctness (composite accuracy of the final-turn global map); Perception (accuracy of local map interpretation for newly observed objects in the current FOV); Self-tracking (agent pose estimated from the predicted global map vs. ground truth); Locals-Global consistency (agreement between local and global predictions within the same turn); and Stability (whether beliefs about previously observed objects remain non-degrading across turns). Full metric definitions are provided in Appendix A.4.
Table 3 reveals a substantial modality gap: performance drops markedly in the vision setting across all metrics. Self-tracking is not primary bottleneck, as models generally maintain accurate pose
7

---

Published as a conference paper at ICLR 2026
Exploration History
Spatial Belief Probing
generate cognitive map
Turn 90; Observe
O
front-right, mid distance
Turn 90; Observe
front, mid distance
Turn 180; Go to
Observe
select unexplored points
front, near
Turn 90; Observe
||||
front-left, mid distance
Go ||||
Turn 180; Observe
C, unexplored
front, mid distance
starting point
Agent pose
front-left, mid distance
possible position
predicted position
Terminate
observed point
unobserved point
Figure 3: Internal Spatial Belief Probing. At each step, the agent executes an action, receives an observation, and updates its spatial belief. We probe this belief by prompting the agent (i) output ISON-struetured cognitive map of all observed objects and (ii) select unexplored positions from a top-down view with labeled candidate points.
ori-
pas.
overall
ori.
pas.
orl.
pos.
ori.
pos.
ori.
pas.
Locale
Self- tracking (%)
Uncertainty (%)
Methods
Correctness (%)
Perception (%)
Stability (%)
Viston-based World
GPT-5.2
20.2
42.0
32.2
33.5
72.4
57.9
58.7
65.4
56.4
93.3
53.7
GEMINI-3 PRO
32.2
62.5
52.1
68.5
52.9
68.3
61.8
62.0
98.8
73.9
70.2
Text-based World
GPT-5.2
75.1
GEMINI-3 PRO
92.5
75.5
81.4
99.9
88.2
91.6
84.8
90.8
67.7
99.9
85.2
79.2
Table 3: Spatial Belief Quality via Cognitive Map Probing. We measure final map correctness and turn-level perception, local global consistency, stability, self-tracking, and uncertainty in text- vs. vision-wond worlds. ori. for orientation and pos. for position. Across models, vision lags text on all metrics, with the largest drop orientation and stability.
beliefs. Perception remains key limitation, particularly for facing direction. This aligns with Table 1, where agents perform poorly on perspective-takinn tasks (~ 36% accuracy). Stability & Decay: While Perception scores suggest models can capture local spatial details with reasonable accuracy, this initial fidelity fails to translate into final Correctness. Spatial beliefs are highly brittle: even when objects are correctly perceived, agents frequently overwrite verified facts with incorrect predictions in later turns. Low final Correctness thus stems not only from perceptual errors, but from the cumulative degradation of spatial memories over the episode.
Key Findings: Cognitive Map Failures (Orientation, Stability, and Belief Drift)
Orientation Gap: Vision perception is bottleneck. especially for object orientation.
Unstable Map: Beliefs about previously observed objects degrades over time.
Belief Drift: New updates corrupt earlier correct perceptions, lowering final correctness.
We also validate the utility of the probed cognitive map. When prompted to explicitly output the cognitive map before answering, performance degrades slightly, revealing the explicit map lossy
8

---

Published as a conference paper at ICLR 2026
all
red.
ori
pos.
on.
pos.
ori
pos.
Belief Correctness (%)
Belief
Methods
Avg. Steps
Identification (%) f
Inertia (%)
Text-based World
GPT-5.2
6.92
0.55
97.9
98.4
89.5
69.7
5.5
12.5
GEMINI-3 PRO
7.79
0.18
98.7
98.8
91.8
72.9
7.9
5.7
Vision- based World
GPT-5.2
13.06
6.20
14.3
68.0
16.7
42.9
68.9
34.7
GEMINI-3 PRO
10.29
3.23
23.9
82.5
30.3
63.1
51.1
Table 4: Belief revision results. We relocate/reorient k=4 objects and evaluate change identification, re-exploration cost (including redundancy (red.)), and belief correctness/iterriia in text- vs. vision- worlds. Vision agents take more redundant steps and fail to overwrite obsolete facing beliefs.
compression the model's internal spatial belief. Nonetheless, map Correctness significantly and positively correlated with downstream performance (Pearson up 0.645, all < .001), validating as a diagnostic proxy for failure analysis (Appendix C.4).
Key Findings: Maps as Diagnostic Proxy
Lossy but Diagnostic: Though lossy compression, map correctness correlates signifi- cantly with downstream success, making a strong diagnostic signal.
We additionally include uncertainty probing, where the agent identifies previously unobserved locations from set of candidate points overlaid on an object-free top-down map, evaluated via Fi score. Results show that GEMINI-3 PRO models uncertainty more effectively than GPT-5.2 in both text- and vision-based settings, which aligns with its stronger cognitive map accuracy. Further analysis is provided in Appendix C.3.
5.2 BELIEF REVISION TASK
Spatial intelligence requires not only mapping static environments but also maintaining beliefs under non-stationarity. Inspired by false belief protocols developmental psychology (Wimmer Perner, 1983; McronnCohen al., 1985) and spatial belief revision (Knauff et al.. 2013), introduce dynamic perturbation task to probe the agent's 's ability to discard obsolete priors and reintegrate new evidence.
Task Protocol. Following the initial exploration phase, we introduce a discrete environmental shift: subset of & 4 objects stochastically relocated reoriented. The agent, retaining its memory (exploration history), must actively re-explore the environment to identify the state changes. This requires the agent detect conflicts between its internal belief state and sensory observations.
Metrics. We evaluate performance along four axes: Identification Accuracy (Fi) (detect which objects changed position orientation); Average Steps (Total Steps identify all changes, and Redundancy Steps, i.c., steps taken after the last changed object is observed, ideally + 0); Belief' Correctness (map correctness as 1,.1, restricted changed objects); and Belief' Inertia (whether updated beliefs remain systematically biased toward obsolete priors). For inertia, we measure positional inertia s!0= as the tendency revised beliefs to remain pulled toward their obsolete prior location, and orientation inertia s as the failure rate overwrite obsolete facing directions. Under unbiased updating, both scores should be near zero. Full definitions are in Appendix A.5.
Key Findings: Vision Deficiencies & Belief Inertia
Vision-based Revision Failures:
Vision agents suffer from excessive exploration redun- dancy and poor accuracy in identifying object shifts.
Belief Inertia: Agents, especially vision-based ones, persist in obsolete spatial coordi-
nates despite new observations.
Table 4 corroborates the modality gap observed in previous sections: vision-based agents significantly underperform their text-based counterparts. This performance drop is characterized by increased exploration redundancy and lower accuracy in identifying changed objects. Notably, while belief inertia persists across both modalities, is markedly more severe in vision-basce agents, particularly regarding object orientation. Vision models frequently fail to overwrite their initial spatial memory,

---

Published as a conference paper at ICLR 2026
persisting with obsolete facing estimates despite new visual evidence. This also suggests that fine-grained orientation estimation remains a critical bottleneck for visual spatial reasoning.
RELATED WORK
Passive Spatial Reasoning, Early paradigms treat spatial reasoning as static inference: given a textual description, agents answer relational queries (Weston al., 2015; Shi et al., 2022; Mirzace et al., 2021; Li et al., 2024). Other benchmarks probe understanding from a single image, asking for relative directions, topological relations, or metric attributes (Ma al., 2024; Deng al., 2025; Cheng et al., 2024; Chen et al., 2024; Liao et al., 2024; Kamath et al., 2023). Multi-view and video benchmarks raise difficulty by requiring crews-view integration, egocentric- allocentric conversion, and temporal consistency (Yang et al., 2025c; Xu et al., 2025; Wu et al., 2025; Yeh et al., 2025; Gholami et al., 2025; Zhou et al., 2025b). Recent works explicitly adopt cognitive maps, showing consistent improvements in spatial reasoning across video QA (Yang et al., 2025a) and multi-view settings (Yin et al., 2025). While informative, these benchmarks remain disembodied, as agents reason only over pre-collected trajectories.
apocentrii-allocentroo
Active Exploration for Spatial Understanding. Research has also examined agents that actively explore, but their exploration is usually tied to task-specific goals rather than building a general spatial belief. Embodied question answering benchmarks evaluate agents by whether they can gather evidence to answer questions (Das et al., 2018; Gordon al., 2018; Majumdar et al., 2024; Ginting et al., 2025; Ren et al., 2024). Instruction-follooinng settings extend household tasks to long horizons and realistic scenes, often with dialog or language grounding (Shridhar al., 2020b; Kim et al., 2024; Shridhar et al., 2020a; Puig et al., 2018; Padmakumar et al., 2022; Gao et al., 2022). Navigation benchmarks stress path execution and generalization across diverse environments (Anderson et al., 2018; Jain et al., 2019; Ku et al., 2020; Krantz et al., 2020; Nguyen & III, 2019; Wang et al., 2024; Zhao et al., 2025). Spatial reference tasks focus on grounding natural-language descriptions in embodied search (Qi et al., 2019; Zhou et al., 2025a), and manipulation (Jiang et al., 2023; Mees et al., 2022: Srivastava et al., 2022: Wu et al., 2023). While existing benchmarks rely on task-driven exploration, they often conflate exploration efficiency with task performance and can foster brittle spatial representations (Bonawitz et al., 2011). EXCALIBUR (Zhu et al., 2023) considers task- agnostic exploration but requires RL training. risking goal leakage and encoding maps implicitly in policy weights. In contrast, we study zero-shot foundation-model agents with no environment-specife training, emphasizing uncertainty reduction over coverage, and evaluating explicit belief construction via belief probing.
CONCLUSIONS
We introduce THEORY OF SPACE, which asks whether foundation models can function as spatial agents under partial observability: not merely answering questions from fixed views, but actively acquiring information through self-directed exploration to construct, revise, and exploit an internal spatial belief. We contribute an evaluation paradigm centered on task-agnostic active exploration, downstream spatial tasks for belief exploitation, and explicit probing of internal beliefs via cognitive map externalization, implemented in a multimodal environment with parallel text- and vision-based worlds. A key strength of THEORY OF SPACE is that it makes spatial belief measurable rather than implicit. By requiring models externalize evolving cognitive maps, THEORY OF SPACE reveals the correctness, internal consistency, and temporal dynamics of belief formation, and quantifies how localized mistakes propagate into global map corruption. Empirically, active exploration is major bottleneck: end-task performance drops relative to passive viewing, with the gap widening as room complexity increases. Belief probes make error sources explicit: in vision, perception error often appears early, and models exhibit belief instability where correct information is overwritten or forgotten, cascading into global inconsistencies. When environments change, models exhibit strong belief inertia, failing to overwrite obsolete priors. THEORY OF SPACE reframes spatial evaluation from "can the model answer?" to "can the model build and maintain a coherent, revisable spatial world model through efficient information gathering?" We hope this benchmark provides foundation for developing models with uncertainty-awar aware exploration policies, robust belief maintenance under long horizons, and reliable belief revision when the world changes.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work proposes a benchmark for evaluating spatial reasoning in large language and vision- language models. All environments are procedurally generated and rely on publicly available assets without involving human subjects or private data. For involved data curation and assessment were compensated at rates well above their local minimum wage. While our benchmark can advance embodied AI research, it may also be applied to domains such as surveillance or autonomous control. We highlight this dual-use potential and encourage responsible application that aligns with ethical guidelines for Al deployment.
evaluation, all contributors
REPRODUCIBILITY STATEMENT
We are committed ensuring reproducibility of our results. Detailed environment specifications, task definitions, prompts, and examples are included in the main sections $3, $4, and Appendix B. The code and data used in our project is publicly available at https: https://theory: of space. github.io.
ACKNOWLEDGMENTS
This work in part supported by the Stanford Institute for Human-Centered AI (HAI), ONR YIP N00014-24-1-2117, ONR MURI (00014-22-1-2740, ONR MURI N00014-24-1-2746 ONR MURI N00014-21-1-2801, and DSO National Laboratories Agreement DSOCO25017.
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauff Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-languuge navigation: Interpreting visually- grounded navigation instructions in real environments, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3674-3683, 2018
Anthropic. System card: Claude sonnet 4.5. m/12f21 4e fec2 F457a/orrroo haa/rriiiaa///// September 2025. System card (PDF).
https::/assee http://.aasttts..t.ppppp.p. anthropi c. com/ pdd,
Shuai Bai, Yuxuan Cai, Ruizhe Chen, and Ke Zhu. Qwen3-vl: The next generation multimodal 11m from qwen alibaba cloud. arXiv preprint, 2025. URL https::/aaaaeiie //tp///avvvvv//////////////a.av/ org/abs /2511. 21631.
Simon Baron Baron-Cohen, Alan M Leslie, and Uta Frith. Does the autistic child have "theory of mind"? Cognition, 21(1):37-46, 1985.
Elizabeth Bonawitz, Patrick Shafto, Hyowon Gweon, Noah D Goodman, Elizabeth Spelke, and Laura Schulz. The double-edged sword of pedagogy: Instruction limits spontaneous exploration and discovery. Cognition, 320(3):32--330, 2011.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 10Xxvv666660054000 2016.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing languaee models with spatial reasoning capabilities. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465, 2024.
An-Chieh Cheng, Hongxu Yin. Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision- language models. Advances in Neural Information Processing Systems, 37:135062-135099 2024.
Elizabeth R Chrastil and William Warren. Active and passive contributions to spatial learning. Psychonomic bulletin & review, 19(1), 2012.

---

Published as a conference paper at ICLR 2026
Elizabeth R Chrastil and William H Warren. Active and passive spatial learning in human navigation: acquisition of survey knowledge. Journal of experimental psychology: learning, memory, and cognition, 39(5):520,, 2013.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-10, 2018.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: universe of annotated 3d objects, 2022. URL https:
Nianchen Deng, Lixin Gu, Shenglong Ye, Yinan He, Zhe Chen, Songze Li, Haomin Wang, Xingguang Wei, Tianshuo Yang, Min Dou, et al. Internspatial: A comprehensive dataset for spatial reasoning in vision-language models. arXiv preprint arXiv:2506. 2025.
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf. James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund, David Cox, Antonio Torralba, James J. DiCarlo, Joshua B. Tenenbaum, Josh McDermott, and Daniel L. K. Yamins. Threedworld: A platform for interactive multi-modal physical simulation, 2021. org/ abs 2007 04954.
URL
Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gaurav S. Sukhatme. Dialfred: Dialogue-enabled agents for embodied instruction following. IEEE Robotics and Automation Letters, 7(4): 2022. Also available as arXiv:2202.13330.
Mohsen Gholami, Ahmad Rezaei, Zhou Weimin, Yong Zhang, and Mohammad Akbari. Spa- tial reasoning with vision-language models in ego-centric multi-view scenes. arXiv preprint arXiv:2509.06266, 2025.
Muhammad Fadhil Ginting, Dong-Ki Ki Kim, Xiangyun Meng, Andrzej Reinke, Bandi Jai Krishna, Navid Kayhani, Oriana Peltzer, David D. Fan, Amirreza Shaban, Sung-Kyun Kim, Mykel J. Kochenderfer, Ali akbar Agha-mohammadi, and Shayegan Omidshafiei. Enter the mind palace: Reasoning and planning for long-term active embodied question answering. 2025. URL https: //arxiv.org abs 2507 12846.
Google. Gemini 3 pro: Model card.
https://storage googleapis.... com/ deepmind nd -nd Mtodel-Cards /eemmiimmnnnnneeeeeeeeeeieiiieie Meniini--Prreee ---rro-Moeel-------------------- pdf, Novem- ber 2025. Model card (GA/preview update for Gemini Pro).
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering interactive environments. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4089-4095 2018.
Torkel Hafting, Marianne Fyhn, Sturla Molden, May- Buy-Britt Moser, and Edvard I Moser. Mierostructure of a spatial map in the entorhinal cortex. Nature, 436(7052):801-866 2005.
Richard Held and Alan Hein. Movement-producee stimulation in the development of visually guided behavior. Journal of comparative and physiological psychology, 56(5):872, 1963.
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. Stay on the path: Instruction fidelity in vision-aand-language navigation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023. arXiv preprint arXiv:2210.03094.
Amita Kamath, Jack Hessel, and Kai-We Chang. What's" up" with vision language models? investigating their struggle with spatial reasoning. arXiv preprint arXiv:2310. 197855 2023.
12

---

Published as a conference paper at ICLR 2026
Taewoong Kim, Cheolhong Min, Byeonghwi Kim, Jinyeon Kim, Wonje Jeung, and Jonghyun Choi. Realfred: An embodied instruction following benchmark in phato-realistic environments. In ECCV, 2024.
Markus Knauff, Leandra Bucher, Antje Krumnack, and Jelica Nejasmic. Spatial belief revision. Journal of Cognitive Psychology, 25(2):147-156, 2013.
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: vision-and-language navigation in continuous environments. In European Conference Computer Vision, pp. 100- Springer, 2020.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Multilingual aision-and-language navigation with dense spatiotemporal grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
room:
Fangjun Li, David C Hogg, and Anthony G Cohn. Reframing spatial reasoning evaluation in language models: A real-world simulation benchmark for qualitative reasoning. arXiv preprini arXiv:2405. 15064, 2024.
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. Embodied agent interface: Benchmarking Ilms for embodied decision making, 2025.
URL https:tt///// //apps:///xxivvv org/ abs/2410. 07166.
Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models. arXiv preprint arXiv:2409. 09788, 2024.
Wufei Ma. Haoyu Chen, Guofeng Zhang, Yu-Cheng Chou, Celso M de Melo, and Alan Yuille. 3dsrbench: A comprehensive 3d spatial reasoning benchmark. arXiv preprint arXiv:2412.07825, 2024.
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra. Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering the era foundation models. In Proceedings of the IEEE/CVF conference 2024.
on computer vision and pattern recognition, pp.
Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditionee policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3), 2022. Also available as 12.0:2212.003227.
Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjmashidi. Spartqa:: A textual question answering benchmark for spatial reasoning. arXiv preprint arXiv:2/04.0588. 2021.
Daniel R. Montello. A new framework for understanding the acquisition of spatial knowledge in large-scale environments. Spatial and temporal reasoning in geographic information systems, pp. 143-154, 1998.
Khanh Nguyen and Hal Daumé III. Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraginng imitation learning. In arXiv preprint (XXX:1199.0017/ 2019.
John O' Keefe and Jonathan Dostrovsky. The hippocampus spatial map: preliminary evidence from unit activity in the freely- rat. Brain research, 1971.
OpenAl.
Gpt-5.2 system card. Gee3--acchde44aredde
ht /dn. http:///..........//////// cwm/com//////oo www.ayeeeeaaaaeeeeeocc.
1a4115cc--c443 August 2025. System card.
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan- Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur: Teach: Trikedriven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2017-2025, 2022.
13

---

Published as a conference paper at ICLR 2026
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang. Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2018. URL https: arxiv. org/abs 118160077000 07011.
Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. arXiv preprint arXiv: /904./09000//// 2019.
Allen Z Ren, Jaden Clark, Anushri Dixit, Masha Itkina, Anirudha Majumdar, and Dorsa Sadigh. Explore until confident: Efficient exploration for embodied question answering. arXiv preprint arXiv:2403. 15941, 2024.
Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. Stepgame: A new benchmark for robust multi- hop spatial reasoning in texts. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 11321-1188 11329, 2022.
Mani Shridhar, Roozbeh Mottaghi, Yonatan Bisk, Luke Zettlemoyer, and Dieter Fox. Alfworld: Align- ing text and embodied environments for interactive task learning. arXiv preprint arXiv:2010.03768, 2020a.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020b.
Alan W. Siegel and Sheldon H. White. The development of spatial representations of large-scale environments. Advances in Child Development and Behavior, 10:9-55, 1975.
Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martin-Martín, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 477-490 PMLR, 08-11 Nov 2022.
Holly A Taylor and Barbara Tversky. Spatial mental models derived from survey and route descrip- tions. Journal of Memory and language, 31(2):66-222. 1992.
Edward C Tolman. Cognitive maps in rats and men. Psychological review, 55(4):189, 1948.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, al. Internv13. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508. 18265, 2025.
Zhaowei Wang, Hongming Zhang, Tianqing Fang, Ye Tian, Yue Yang. Kaixin Ma, Xiaoman Pan, Yangqiu Song, and Dong Yu. Divscene: Benchmarking Ivlms for object navigation with diverse scenes and objects. arXiv preprini arXiv:2410.027300, 2024.
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv: 152.055698... 2015.
Heinz Wimmer and Josef Perner. Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children's 's understanding of deception. Cognition, 13(1):103-128, 1983.
Haoning Wu. Xiao Huang, Yaohui Chen, Ya Zhang, Yanfeng Wang, and Weidi Xie. Spatialscore: Towards unified evaluation for multimodal spatial understanding. arXiv preprint arXiv:2505.1 17010, 2025.
Yue Wu, Xuan Tang, Tom M. Mitchell, and Yuanzhi Li. Smartplay: A benchmark for 1lms as intelligent agents. arXiv preprint 10.0022310 2023.
14

---

Published as a conference paper at ICLR 2026
Runsen Xu, Weiyao Wang, Hao Tang, Xingyu Chen, Xiaodong Wang, Fu-Jen Chu, Dahua Lin, Matt Feiszli, and Kevin Liang. Multi- suatiilppai Multi-frama spatial understanding with multi-modal large language models. arXiv preprint arXiv:2505.17015, 2025.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10632-10643, 2025a.
Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang. Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision- driven embodied agents, 2025b. URL
org/ abs 2 22022. 09560.
Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang. Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, et al. Mmsi-bench: A benchmark for multi-image spatial intelligence. arXiv preprini arXiv:2505.23766 237644 2025c.
Chun-Hsiao Yeh, Chenyu Wang, Shengbang Tong. Ta- Ying Cheng, Ruoyu Wang, Tianzhe Chu, Yuex- iang Zhai, Yubei Chen, Shenghua Gao, and Yi Ma. Seeing from another perspective: Evaluating multi-view understanding in milms. arXiv preprini urXiv:2504. 15280, 2025.
Baiqiao Yin, Qineng Wang, Pingyue Zhang, Jianshu Zhang, Kangrui Wang. Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, al. Spatial mental modeling from limited views. arXiv preprint arXiv:2506.2 1455, 2025.
Yong Zhao, Kai Xu, Zhengqiu Zhu, Yue Hu, Zhiheng Zheng, Yingfeng Chen, Yatai Ji, Chen Gao, Yong Li, and Jincai Huang. Cityeqa: A hierarchical 11m agent on embodied question answering benchmark in city space. arXiv preprint arXiv:2502. 125322, 2025.
Zhipu AI Team. Glm-4.6v: Native multimodal foundation model. Technical report / model release, 2025. URL https:::...... http://aaa//bbl ai/blog / ggm-mmm4444 4. 6v. Available online; native multimodal vision + reasoning model (128k context) among open-source LLMs.
Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong, Chi Zhang, Pengwei Wang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, and Shanghang Zhang. Roborefer: Towards spatial referring with reasoning in vision-language models for robotics. arXiv preprint arXiv:2506. 2xXx:2500600308, 2025a.
Shijie Zhou, Alexander Vilesov, Xuchai He, Ziyu Wan, Shuwang Zhang. Aditya Nagachandra, Di Chang, Dongdong Chen, Xin Eric Wang, and Achuta Kadambi. Vlm4d: Towards spatiotemporal awareness in vision language models. arXiv preprint arXiv:2508. 02095, 2025b.
Hao Zhu, Raghav Kapoor, So Yeon Min, Winson Han, Jiatai Li, Kaiwen Geng, Graham Neubig, Yonatan Bisk, Aniruddha Kembhavi, and Luca Weihs. Excalibur: Encouraging and evaluating embodied exploration. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14931-14942, 2023.
15

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
Details of THEORY OF SPACE
17
A.I Formal Definition
17
A.2 Downstream spatial tasks
17
A.3 Assessment Dimensions
17
A.4 Cognitive Map Probing
19
A.5 Belief Revision Metrics
19
Technical Details
21
B.1
Benchmark Construction
21
B.2
Proxy agents.
22
B.3 Prompts
23
C Extra Results
26
C.1 Additional Results for passive setting
26
C.2 Additional Results for multi-room settings
26
C.3 Uncertainty Map Probing .....
27
C,4 Cognitive map Validation & Correlation
28
Additional Visualization Examples
D.1 Cognitive map output by models
31
D.2 Exploration pattern examples models
ce..............................
D.3 Analysis Platform
31
LLM Usage Statement
38
16

---

Published as a conference paper at ICLR 2026
A DETAILS OF THEORY OF SPACE
A.1 FORMAL. DEFINITION
E
Definition: THEORY OF SPACE
Ability to construct, revise, and exploit an internal spatial belief.
We formalize THEORY OF SPACE within partially observable framework over a spatial structure S E S. The agent interacts with S to generate history h2 = (Ouas du:z). where and denote observations and actions. We define THEORY OF SPACE as the capacity to manipulate a probabilistic belief B, through three core operations:
1. Construct: To form a globally consistent internal spatial belief by actively seeking out and integrating partial observations. Formally, the agent integrates le to approximate the true posterior, denoted as B,( R P(S h.).
2. Revise:
To dynamically update the internal belief by using new information acquired through.urtter exploration to resolve conflicts with prior beliefs. Upon an environmental shift S 3 S²,, the agent utilizes exploratory actions Ah to minimize the divergence from the new ground truth, i.c., Bu+ + P(S') hetae).
3. Exploit: To utilize the current belief to support spatial tasks. The agent utilizes policy x conditioned on the belief, π [a; B²). to perform a downstream task T. a benchmark
context, we measure the value of belief by the performance metric I achieved by this policy: I (=((IB)) T)
A.2 DOWNSTREAM SPATIAL TASKS
Drawing prior work (Siegel & White, 1975; Montello, 1998), we define tasks to evaluate an agent's Route and Survey knowledge. An overview of the tasks is present in Figure 4 and Table 5.
A.3 ASSESSMENT DIMENSIONS
Here show the detailed definition of five assessment dimensions.
(D1) Belief Construction Efficiency. Measures how efficiently the agent collapses spatial uncertainty during exploration. We quantify this using normalized information gain metric, E. Let M be the number of possible positions for any object at the start of exploration (a uniform prior), and let C, be the number of positions for object that remain consistent with all observations gathered by the agent (calculated by AC-3 algorithm, check Appendix B.2). The efficiency is calculated as
= EN log, m)x(1,C.). This score ranges from 0 (no information gained, C2 = MM) to (all N logg Mr
objects perfectly localized, C; = 1). Note that it can also be used to calculate the accumulated information gain each step. Information gain is mainly used in text-based environments, since bsiionbbssed environments have direct access to scenes without such ambiguity. Therefore, for vision-based environments, we directly use node coverage to measure exploration efficiency.
Belief Representation and Quality Assessment. A core contribution of THEoRY OF SPACE is disentangling spatial memory from spatial inference. We structurally decompose the probed cognitive map into two components:
(D2) The Cognitive Map (Observed): Measures fidelity and coherent integration of observations over time. We evaluate using two criteria: (1) Correctness, alignment with ground truth, computed as a composite of positional, directional, and facing accuracy: and (2) dynamic reasoning diagnos- ties, including Perception quality, Self-tracking, Stability, and Local <+ Global Consistency, reflecting internal coherence such as the absence of contradictions within the relational graph and between maps and relations.
(D3) The Uncertainty Map (Unobserved): Measures how well the agent models plausible hypotheses about unobserved regions. We assess Uncertainty Modeling by providing a candidate set of positions formed by randomly sampled points from both observed and unobserved areas, and measuring the agent's ability identify valid locations via Fi.
17

---

Published as a conference paper at ICLR 2026
Dynamic Group
Belief on Route
Belief on Survey
Statie
Pairwise Relation (direction) report allocentric direction and distance from A to B.
Allocentric Mapping (alloc.map)
predict global coordinates (and headings) for all objects.
Forward Dynam-
Perspective Taking (persp.take) output the observation from specified object's perspective.
Mental Rotation (ment.rot)
ies
predict the sequence of front-facing ob- jects during 360" self-rotation. Location2 View (loc2view)
Action-to-Vics (ac(2view)
given sequence of Goto/Rotat ete, pre- dict the final observation (one object in FOV with ego direction/distance bins).
given global pose, predict the observa- tion (onc object FOV with ego bins/dis- tances).
Backward Dynam- ics
Perspective Decision (perc.dec) infer which object's ' perspective the agent is currently adopting.
View2Location (view2loc)
localize agent (and optionally orienta- tion) from a target observation under the map.
View-to-Action (view2act) recover an action sequence that produces a target observation.
Table Task suite comparison: Route belief emphasizes egocentric, step-by-step path reasoning; Survey belief emphasizes allocentric mapping and novel view inference.
Route
Survey
Belief Route: Reasoning about paths and landmark relations
Bellef on Survey: Building an allocentric map
Pairwise Direction
Allocentric Map
What's spatial direction
What's 2D coordinate
between
Perspective taking
perspective,
Mental Rotation
object sequence rotate clockwise
Perspective Determine
You observe at your front-left, which object's
perspective it?
Location2View
Action2View
If you move (0, 2), where
Aftar actions
where |MI
View2Action
View2Location If you see at your right front, where are
||| your
front-right, generate action sequence
Figure 4: THEORY OF SPACE exploitation task suite: it covers route-level egocentric reasoning and survey-level allocentric mapping. Route tasks evaluate path-based inference and egocentric observations. Survey tasks test global mapping, geometric transformation, and perspective conversion. Together they cover both local navigation reasoning and global spatial abstraction.
This separation lets us diagnose whether failures stem from misestimating the observed world or from insufficient reasoning about what remains unobserved.
(D4) Belief Revision. Measures the agent's 's ability to revise its spatial belief under latent envi- ronment changes. We evaluate this using the False Belief task ($5.2), where objects covertly
18

---

Published as a conference paper at ICLR 2026
manipulated (translated rotated) following the initial exploration. The agent must re re-explore to detect these discrepancies; we measure the accuracy of these identified changes (both object identity and transformation type) using the Fj score. Furthermore, we introduce Belief Inertia quantify whether belief revision remain biased toward obsolete priors.
(D5) Belief Exploitation Success. Measures task success when the agent must utilize its spatial belief. For tasks involving spatial relations (direction, persp.take, action2view), we score direction and distance separately, awarding 0.5 for each correct component. For tasks that output coordinates (view2loc, mapoc we compute a coordinate similarity score.
A.4 COGNITIVE MAP PROBING
We give detailed definition of metries and scores defined for cognitive map probing here.
Metries. We evaluate cognitive map using three complementary metrics. Positional accuracy (pos.acc) is the Euclidean similarity between predicted and true object coordinates: (K/N) RMSB/L where RMSE is the root mean squared error between predicted and ground-trut object positions, L is the RMS l -n-m of the positions of all objects in the scene, and K / is the coverage (the ratio of the number of predicted objects K the number of ground-truth objects N). Directional accuracy (diracc)) is the accuracy of directional relationship between each pair of objects. Facing accuracy (facing.ace acc) is the fraction of objects whose predicted facing matches the ground truth.
Using global and local belief representations, we compute set of diagnostic scores at each turn t (all per-turn except Correctness, which is computed only at the final turn after termination). Unless noted, scores are averaged over turns and scenes:
Correctness (final): Measures the accuracy of the agent's 's terminal global spatial belief. At the last turn, we evaluate the predicted global map and report a composite score given by the (equally weighted) mean of the three metries defined above, with weights 1/3 each. We compute diracc only for correctness, since the global cognitive map prioritizes consistent pairwise spatial relations.
Measures how accurately the agent interprets newly observed local structure. We compare the predicted local map to the ground-trute local map for the current field of view (FOV), counting only objects that appear in the FOV for the first time.
Perception:
Self-tracking:
Measures how well the model estimates its own pose over time. We infer the agent's 's pose from the predicted global map and compare it against the ground-truth agent state.
Local ++ Global consistency: Measures whether new local evidence incorporated into the global belief coherently. Within the same turn, we compare local and global predictions to verify that newly perceived structure integrated without contradictions.
Stability: Measures whether beliefs about previously observed objects remain non-degrading over time. For each previously observed object, at every subsequent turn check that its predicted state does not worsen; the per-check score is 1 if the prediction is no worse than in the previous turn.
A.5 BELIEF REVISION METRICS
We provide detailed definitions of the metrics used to assess belief revision here.
Metrics. We evaluate performance along four complementary axes:
Identification Accuracy (Fi): How precisely the agent pinpoints which objects changed. We compute the Fi score for detecting the subset of objects whose position or orientation shifted.
Average Steps: How efficiently the agent revises its beliefs to completion. We report Total Steps needed identify all changes, and Redundancy Steps, defined as the number of steps taken after the last changed object has been observed. Ideally, Redundancy - 0, indicating the agent recognizes when updating is complete.
19

---

Published as a conference paper at ICLR 2026
How accurate the updated beliefs the changed subset. We compute correctness as in $5.1, but restrict evaluation to changed objects to isolate the fidelity of re- exploration.
Belief Correctness:
Belief Inertia: Whether updating remains systematically biased toward obsolete priors. To quantify attraction back to pre-shift beliefs, we test whether the residual error of the updated belief aligns with the direction of the old belief. For each shifted object i, let b denote the pre-shift belief, b the post-revision belief, and g the post-shift ground truth. Define the
prior-offset and post-revision enror vectors: V, b g e; positional inertia
b gree. We define
| new boldd 12
pos
V V
exp
e
+
20
alignment (cos
Proximity weight (ws)
Here cos 0g is large when the remaining error after updating still points toward the obsolete location, while w, downweights such alignment when the belief has moved far from bodd, We a to dynamic noise scale: the RMS localization eтrоr on the first re-observed unchanged objects during re-exploration; c ensures numerical stability. Under unbiased updating, E[s whereas o > 0 indicates systematic pull toward the obsolete prior. For orientation shifts, we measure inertia via s?ri Il (opeew oold). where & denotes the predicted orientation. It flags failures overwrite the obsolete facing direction.
20

---

Published as a conference paper at ICLR 2026
B TECHNICAL DETAILS
B.1 BENCHMARK CONSTRUCTION
We expose the ToS world as Gym-like interface (Brockman et al., 2016): agents interact in discrete steps under partial observability at a resolution of 384 x 384 to construct and revise an internal spatial belief, which we later exploit in evaluation tasks. Scenes layouts on an Nx M grid with named indoor objects (each with integer (æz, y) and heading in (N.E,S,W) and randomized agent spawn pose. We restrict multi-room layouts tree topology: the room-adjacency graph is connected and acyclic (no loops).
procedurally generated multi-room
Text-based World At each step, OBSERVE returns a symbolic snapshot of objects in the current room within 90" forward FOV. For every visible object provide discretized egocentric direction front-efollleff)) and distance bins (e.g., nearlmidffar), plus object identity and facing when deter- minable. Egocentric observations into angular and distance bins as specified in Figure 5a. Visibility is room- bounded; doorways act as transparent portals only when the agent stands in them, enabling dual-room visibility. Optional noise modules perturb bins for ablations.
rendered with 90-degree field of view (FOV), discretized
Vision-based World We procedurally generate scenes in 3D simulator with controllable parameters: the level (number of rooms) and the object count room. Objects are drawn from library of 293 distinet 3D models, grouped into 6 categories and 37 subtypes, primarily everyday household items (see Figure 5b). To ensure diversity, each object type appears at most once in a given scene.
Model Overview
Far 20
Furnitue
Transportation
Utamals
Apslunnss Toys
Glohing
front Joft 45* -2255
min th
front-siggt left (-22.5
front (22)
10%
near (0. 21
for (5. 10]
(a) Field of view (FOV) specification the agent our tasks. The FOV spans 90% front of the agent and is divided into angular bins (e.g., front, front-slight left, front-left) and distance ranges (near [0.2], mid [2.5]. far [5,I This egocentric perception de- fines how spatial relations are observed and reported.
(b) Distribution of all 3D models used our vision tasks.
Figure Demonstration figures for FOV and model distribution
For task setup, we additionally generate instructional (Figure 6) and orientation (Figure 7) images that serve references for the agent in vision-world. We include both images in the vision prompt. Object placement follows validity constraints (e.g., collision avoidance, minimum spacing), and random seeds control reproducibility across environments.
21

---

Published as a conference paper at ICLR 2026
Figure 6: Example of distance cues in the vision prompt. The colored cylinders illustrate objects placed different distances from the agent: yellow at m, blue at m, red 2 m, and green at 3 m, providing calibration for mapping visual observations to discretized distance bins.
Figure 7: Object appearance and orientation cues in the vision prompt. Objects with facing direction are shown from both the front and side views, while objects without inherent orientation are displayed only from the front view. This provides the agent with consistent visual references for recognizing shape and facing.
B.2 PROXY AGENTS
Information Gain Calculation We use the AC-3 are-consistency algorithm to maintain, for each object, domain of feasible grid cells. Initially, every object's domain spans the entire 20 20 map.
22

---

Published as a conference paper at ICLR 2026
Each new observation is compiled into unary and binary constraints (e.g., egocentric direction/distance bins, room visibility/occlusion,, and ALLDEEEEENT to prevent collisions). When a constraint is added, AC-3 iteratively prunes any cell in one object's domain that is unsupported by the domains of related objects, propagating revisions along incident ares until a fixed point is reached (all ares are consistent). While AC-3 alone does not guarantee global consistency, in our setting all constraints derived from a valid trajectory; therefore the ground-truth assignment remains supported and is never pruned, ensuring that domains stay non-empty throughout propagation.
Scripted Proxy Agents We implement two scripted proxies to provide strong, reproducible base- lines.
SCOUT. From its spawn pose, the agent performs 360 sweep (four cardinal ROTATE+OBSERVE actions) to capture all views at the initial location. It then follows a fixed room- visi-iiitatioon order: upon discovering doorway, it enters the adjacent room, executes the same sequential sweep, and repeats this "weep aadi--seeep-aavacce" advance" routine until every room has been observed at least once.
STRATEGIST. The first stage mirrors SCOUT: panoramic sweep to register all currently visible objects. Thereafter, within the current room the agent maintains, for each object, a set of feasible positions ("domain") induced by accumulated observations. each turn it: (i) selects the object with the largest remaining domain (highest positional uncertainty); (ii) moves to a viewpoint that best constrains this object (e.g., near along a sightline that intersects the most candidate cells); (iii) that viewpoint, orients to test pairwise relations: it computes unresolved pairwise directions between the target object and all others in the room, identifies the direction bin with the highest outstanding count, and OBSERVES Es in that orientation first. The procedure iterates until all objects in the room are resolved (domains shrink to singletons), then proceeds the next unvisited room and repeats.
B.3 PROMPTS
We show the detailed designs of our prompts for exploration in Figure 8, evaluation prompts in Figure 9, cognitive map prompts in Figure 10, and top-down view for uncertainty modeling in Figure 11.
Poind duse wiluding
CVV wddn
janonn)
and chape sipat
Buang
MeT Seach tie
(/2 aos real Eaary
moniiyy CAV
6107
Tou a0o0
rostrand Lofiggh midh
1/00 Cieerci
Figure& 8: Exploration prompts
23

---

Published as a conference paper at ICLR 2026
Route
Survey
Pairwise Direction
Allocentric Map
You return to your starting position and face north. From Top-Down map, describe where (obj name) relative to (anchor __na_)).
Treat your starting position as the origin (0, 0) while facing north. Report allocentric coordinates using (x right/east. y up/north).
Objects: (object list).
Answer format: <cardinal direction>, <distance>
Answer format: (x0, y0); (x1, y1); in the same order.
Example: north-west, near
Example: (1. O); (-2, 3); (0.-1)
Perspective Taking
Now you jump fanchor namel's direction, facing its direction. Describe where (abj_naaee name) is relative to you.
Mental Rotation
Answer format: <ego direction>, <distance>
You return to starting position and face north.
will perform full 360-degree rotation by continuously turning (lum direction) in place. Assume all walls are removed (you can see through walls), every object visible.
Example: front-left, near
Perspective Determine
Focus on this set of objects: (object _oolc___ooo.._____o_______ List them the exact order they appear directly ahead while you rolate. two objects share bearing, place the nearer one first.
Now you jump to object's position, facing its direction. You observe that (observation). Which object are you standing at?
Answer format: <object name>
Answer format: <object_ name1>, <object =object_name2>
Example: lamp
Example: mug. sofa, plant
Action2View
return your starting position and face north. You will execute the following action sequence: (actions)
Location2View
(origin_instruction]) instruction)Youu move to (loc) and face (direction).
After executing the actions, what is the ego relation of (target) relative to you?
What is the egocentric relation (largel)?
Answer format: <ego direction>, <distance> Example: front, near
Answer format: <direction>, <distance>
Example: front, near
View2Action
You return to your starting position face north. Then you have executed an action sequence and changed to location and facing direction. observe following: (final obs)
View2Location
move location and face (orientation). (observations)
What action sequence led to this final view? The action sequence must valid and only contain move actions.
(origin_ineel jir_situcttonnWnattttttt is your new 2D coordinate (x, y)?
Answer format: (x. y) Example: (2, -1)
Answer format: <sequence of actions> Example: Jump To(lamp), Rotate(9O)
Figure Evaluation prompt design. We show the prompt for each evaluation task.
24

---

Published as a conference paper at ICLR 2026
Kagiia tee LOONE
Reprsemn Dee tore LSON
[Sdhra jahande
Now (s. ieteeer
ppallios magers
fuding Yoattadd tpiia
"ogainve"" Jocal hocmme
Inducc obaeveed eijecis.
iellde Ducing Ciy the sijject
frcnng buoing obadd
I2hbe Cog
oegett your pollion
ane roltca is
celtle giotal map hield goi.
inbil
bratie dul poies phanty
Contant incuud all reberreed cijeca and
keep Butthiee rignii curvert
gittagert Ascpps
Corkent jrckus oijicts and grass
raggir cound makin
usa ahjuss
contial onlly
hourt "orlttoueeattivv
'Glis paint Vour caroor posiiion.
porlo ohfunng cnbaii foon..
NOTe mest eapecs
[[mmpp))
inber thrt locdledis
(boatton Noolgg
wing use "reethee Booe doens
Tearf)
(beation pt Taeingg
'ouar Cooa don't P. 4 lading
"Toceg" (wert)
CLM 7900 Sor P. "saing (x4)
"100" (poation 3, "oraa"" "wor
Figure 10: Belief probing prompt design. We use these prompts to ask the model to output cognitive map or select unobserved points.
Symbol Map (text)
Image Map (vision)
wall
emppy abject
empty oell
A K: object
aggnnn position
FA......
agent position
t.B.
F.C.G.
B.b....
Figure 11: The symbol map and the image map provide parallel representations of the same environment for text and vision settings uncertainty probing prompts.
25

---

Published as a conference paper at ICLR 2026
C EXTRA RESULTS
To enable a like-for-like comparison between the text and vision settings, we instantiate identical room layouts across modalities. Concretely, we generate 100 evaluation instances with IDs 0-99; for each ID, we use the ID itself as the random seed to drive task sampling in both environments, This seed tying guarantees deterministic layouts and bit-for-bit reproducibility across modalities.
C.1 ADDITIONAL. RESULTS PASSIVE SETTING
The performance of various models both text-based and vision-based environments is summarized in Table 6.
direction
perspiinke
pere doc
act2view
view2act
alloc map
mentrot rot
loc2view
view2hoc
Static (s)
Dynamic
Statie (S)
Dynamic
Methods
Survey
Avg.
Vision-hased
Proprietary Models
G.T
47.3
35.0
63.9
54.5
49.3
64.8
83.3
50.3
65.6
57.1
GEMINI-3 PRO
63.8
36.3
57.5
49.0
58.0
67.2
85,3
70.4
57.0
60.5
CLAUDE-4.5 SONNET
47.3
33.5
37.7
40.8
15.7
54.8
58.3
44.7
54.8
43.1
Open-soures Models
GLM 4.66V
11.5
24.5
4.7
19.0
2.7
22.9
11.7
20.0
33.6
16.7
QWEN3-VL
20.8
28.3
22.7
16.7
4.7
33.2
21.7
27.3
40.8
24.9
rest-
Proprictury Models
GPT-5.2
84.5
88.2
97.0
76.0
98.3
94.8
89.2
GEMINI-3 PRO
82.7
92.7
97.0
87.5
75.7
86.2
91.3
85.7
80.0
86.5
CLAUDE-4.5 SONNET
73.0
80.7
90.7
77.7
59.0
76.9
74.3
59.2
70.7
73.6
Open source Models
GLM-4.6V
22.3
39.8
25.0
25.3
4.7
21.2
9.0
27.0
35.7
23.4
IL-ERNVL-3.5 QWEN3-VL.
36.7
67.8
42.7
41.2
8.7
37.3
19.3
38.7
43.8
37.4
40.8
69.3
42.5
54.6
Table 6: Exploitation Performance (%) of Belief Construction via Passive Observations. Models evaluated as passive comprehension agents on Route- and Survey-level reasoning using standard- ized observation logs from scripted proxy explorers, decoupling exploration from belief construction across text- and baseed environments. GEMINI-3 PRO leads most tasks in the vision-base based world and achieves the best overall average, while GPT-5.2 leads the text-based world and attains the best overall average.
C.2 ADDITIONAL RESULTS FOR MULTI-ROOM SETTINGS
We show detailed results for different room settings including two-room and four-room layouts. In both the two-room and four-room settings, we use the same room size and the same number of objects per room as the three-room setting. For the four-room setting, we connect the main room with all the others. We evaluate GPT-5.2 and GEMINI-3 PRO, the two best pest-pperforing models. Additionally, we tested higher resolution, but found no performance gain. Table 7 and 8 report passive and active performance of the two-room setting. Table 9 and 10 report passive and active performance of the three-room setting. As the number of rooms increases, exploration cost rises accordingly. The results also underscore the importance of efficient exploration: in the four-room setting, which demands more strategic exploration, the gap between active and passive performance becomes substantially larger.
26

---

Published as a conference paper at ICLR 2026
direction
perqptake
perc dec
act2view
view Zart
alloc.map map
ment.rot
loch
view
Static (S)
D Dynamic (D)
Static
Dynamic (D)
Methods
Route
Survey
Avg.
Vision-hused World
Proprietary Models
GPT-5.2
39.2
63.3
53.8
58.3
92.7
52.3
68.6
59.3
GEMINI-3 PRO
57.8
33.9
53.8
48.5
58.7
64.6
83.3
54.7
69.8
58.3
Text bused World
Proprietary Models
GPT-5.2
99.0
83.0
99.7
GEMINI-3 PRO
88.2
86.7
91.7
87.3
79.3
90.1
92.7
81.5
82.9
86.7
Table 7: Exploitation Performance (%) via Passive Observations under two rooms settings.
directien
perspiake
perc dite
aed aet2 jow
view view Zact
alloc map
mentrot
Loc veiwv
view2loe
Static
Dynamic (D)
Static (5)
Dynontie
Methoos
Avg.cost
Route
Survey
Avg.
hissd World
Proprictury Modelss
GPT-5.2
56.9
72.0
GEMINI-3 PRO
6.6
51.7
36.3
63.0
47.2
56.0
63.4
85.0
50.3
67.5
57.8
Text-based World
Proprietory Models
GPT-5.2
68.7
67.3
90.0
76.8
64.0
83.4
92.7
73.7
83.7
77.8
GEMINI-3 Pro
6.2
76.0
68.3
89.0
77.2
72.7
83.1
96.0
77.5
86.2
80.6
Table 8: Exploitation Performance (%) via Active Exploration under rooms settings.
direction
persp take
pore dee
aci2view
view2act
alloc.map
Ioc2view mentroo rot
View
Statie
Dynamicc (D)
Statie
Dynomic
Methods
Survey
Avg.
Visson bused World
Proprietary Models
GPT-5.2
47.0
37.7
59.7
38.3
40.3
60.1
73.7
50.5
65.9
52.6
GEMINI-3 PRO
35,5
58.7
42.8
43.0
81.7
48.8
67.4
562
Text-based World
Proprietary Models
GPT-5.2
83.8
88.2
94.3
86.8
62.7
94.8
93.7
82.0
92.5
86.5
GEMINI-3 PRO
81.2
91.3
96.7
82.2
68.3
76.8
81.3
74.2
79.0
81.2
Table 9: Exploitation Performance (%) via Passive Observations under four rooms settings.
direction
perspaaae Lake
perc.dec
View Juct
alloc map
memurox
loc2view
view
Stutie
Dynamic (D)
Stonic (S)
Dynomic Survey
Methods
Avg.cost
Ronte
Avg.
Visson based World
Priprletory Models
GPT-5.2
23.2
41.2
33.2
49.0
30.8
30.7
32.5
49.7
40.5
55.4
40.3
GEMINI-3
Text based World
Proprietory Models
GPT-5,2
16.4
65.3
69.0
74.3
62.8
44.3
66.6
76.3
57.5
77.8
66.0
GEMINI-3 PRo
19.7
76.3
77.2
91.7
73.3
61.3
77.0
83.7
74.0
81.9
77.7
Table 10: Exploitation Performance (%) via Active Exploration under four rooms settings.
C.3 UNCERTAINTY MAP PROBING
To probe an agent's ability model uncertainty, we provide with a top-down view of the scene in which all objects removed, and overlay set of candidate points. These points sampled randomly and include both previously observed and unobserved locations. The agent's task is to identify which candidate points remain unobserved, thereby revealing its belief over unseen regions.
Representation. The agent receives an empty top down map that shows only the candidate points and its current position, with objects present. The agent must select the points that have not yet been observed. the text based world, the top down map is represented as an N x M symbolic grid, where different symbols denote the agent, gates, and candidate points. In the vision based world,
27

---

Published as a conference paper at ICLR 2026
Accumulated Info Gain & Cognitive Map Correctness
1.0
0.8
Cognitive Map Correctness
0.8
Info Gain
0.6
0.4
0.4
gpe-32 MGgg Teat)
0.2
2pt-12 (DegMap
0.2
gerxzi-3prr
1po May Waner
1
5
7
11
13
15
17
19
21
Steps
Figure 12: Accumulated Information Gain and Cognitive Map Correctness steps.
all objects removed and the agent instead receives top down image of the environment, check examples Figure 11. We use F1 to evaluate selected points.
We report Uncertainty scores in Table 3. GEMINI-3 PRO models uncertainty better than GPT-5.2 in both text- and vision-based settings. These results help explain the information gain and cognitive map trends in Figure 12. GPT-5.2 achieves higher initial information gain (i.e., it ramps up faster), likely because it quickly commits to an explore-the-dee
strategy. However, it generalizes poorly to unobserved regions, reflected by the subsequent plateau in Figure 12: additional steps yield little marginal gain. In contrast, although GEMINI-3 PRO improves more slowly at the beginning, its cognitive map accuracy continues to increase with exploration, suggesting it keeps collecting useful evidence and progressively resolving uncertainty.
C.4 COGNITIVE MAP VALIDATION & CORRELATION
Cognitive Map Validation & Correlation. To validate the utility of the probed cognitive map and investigate whether faithfully reflects the agent's reasoning process, first conducted two ablation studies:
Sufficiency Test (Oracle Map): We conditioned the model on the ground-truth cognitive map before generating answers for evaluation. Performance rose to near-perfect levels (~2 95% for both models in both worlds). This confirms that our cognitive map representation captures all necessary information for the tasks; performance bottlenecks stem from the agent's inability to accurately construct the map, not the representation format itself.
Alignment Test (Explicit Reasoning): We prompted the model to explicitly generate the cognitive map before answering the evaluation questions. This resulted in slight performance degradation compared to direct answering.
These results reveal an externalization gap: belief is richer While it is a lossy compression of the agent's true internal state, the explicit map remains a strong diagnostic signal. We support this claim by computing the Pearson correlation between the agent's cognitive map Correctness and down- stream task performance. To ensure robust cor- relation, we calculate the average performance across five independent cognitive map runs for each sample. As shown in Table 11, belief correct- ness is consistently and positively correlated with downstream success both modalities, with all correlations significant (p < .00)). The association
the model's latent internal spatial
more accurate than the
discretized JSON output produces.
Methods
Text (%)
Vision (%)
GPT-5.2
41.8
57.0
GEMINI-3 PRO
46.6
64.5
Table 11: Pearson correlation (r) between be- lief correctness and downstream performance. All correlations are significant (p .001).
28

---

Published as a conference paper at ICLR 2026
is stronger in vision (r=0.370/0.66) than in text (0=00.41/0.4666. The stronger vision correlation suggests that perception-drite -eriiinn-rivennnne mapping emrors and unstable belief updates more directly translate into task failures. Thus, we establish map probing While acknowledging that correlation does not imply causality, we treat the explicit map as a robust, albeit conservative, signal for diagnosing reasoning breakdowns rather than definitive evidence.
validated diagnostic proxy for failure analysis.
29

---

Published as a conference paper at ICLR 2026
D ADDITIONAL VISUALIZATION EXAMPLES
We include concrete examples of task formats and answer styles with open-ended, format-constrained outputs in Figure 13.
Pairwise Direction
Allocentric Map
You return starting positinn and face
Treat your starting poeition as the origin (0, while facing north. Report allocentric coordinates using righteast, upinorth). Objects: shellf, truck, lamp.
From a Top- Top-Down map, describe where shelf relative to tnack.
A: south east, mid distance
[112, -1] [10, 1). [0,
Taking
you jump backpack's direction, Desctbe where chair is relative you.
Montal Rotation
A: front-left, mid distance
Q: You retur to your starting position and face north. You will perform a full 360-degree rotation by continuuully tumning counterdlodkwise place. are removed (you see through
Perspective Determine
walls), sa every abject is visibla. Focus on this sat objects: bike, television. List them in the exact they appear directly while you objects bearing, place
Q: Now you jump an object's position, facing its direction. You observe that truck front-left, facing backward:
distance. Which object are you standing at?
first.
A: laptop
[television', "penn, 'bike'
Action2View
Q: You retum starting position and face north.
You will executie the following action sequence:
Location2View
Jump to the onject frant-night, mid distance. Rotale(-SO)
Treat the green door new 'origin' (0, 0).
Jump objest frant-right, Rotiale-1180
You move to (2, -5) and face north. What the egocentric relation of
After executing the action,, what relation
bike relative you?
A: front, mid dietance
front-right,
Q: You
starting position and face
Vlow2Location
sequence and changed to a new location and facing direction. You abserve the following: pan at front-riht, slightly facing backward:
Q: You move to new location and face north. You observe: front-riht,, door at front front-sligti-ler slightleft. slighly far, on wall Treat the green door the new lorigin' (0, O).
front, distance, facing
front-right,
green
dietance. facing forward: leptop is et front, distance: facing backward What action sequence led final view?
[2,
A: [[rotate', 90), [jumptt'', 'green door). [Jumpto', 'sholf), (rrotatel, 180]]
Figure 13: Examples of task formats and answer styles used. Each block illustrates a spatial reasoning task type in our suite (Route-level and Survey-level), including the corresponding input context and an example open-ended answer that must follow strict output format. In the vision setting, textual scene descriptions the questions are replaced by rendered observation images.
30

---

Published as a conference paper at ICLR 2026
D.1 COGNITIVE MAP OUTPUT BY MODELS
We visualize the tur-by-turn cognitive maps (in Figures 14 and 15 of GPT-5.2, comparing them against ground-truth maps. The performance is noticeably stronger in text-based environments than in vision-based ones.
D.2 EXPLORATION PATTERN EXAMPLES BY MODELS
We include representative trajectories from each model to illustrate the active exploration patterns identified in our analysis, shown in Figure 16, 17, 18, 19, and 20 These examples highlight how different models manifest recurring exploration behaviors: for instance, GPT-5.2 often adopts a "finding-gate gaatingggtte"" strategy, rotating until doorway is detected before moving toward it, while other models more frequently repeat redundant checks. All figures mark the agent's ' position and orientation explicitly, with actions annotated beneath each frame and shared legend provided for each trajectory.
D.3 ANALYSIS PLATFORM
We also include some demonstrations in Figure 21, 23, 22, 24, and 25 of our designed platform for better analysis
Texn Reod
Tere Pree
Turn Piid
Tors Frod
Then Pood
Ture Frod
Turn Pood
Tele
Taln 2'177
ST
Figure 14: GPT-5.2's turn-by-tun cognitive map in text world during exploration.
Tex Aer
T Aey
Yext Aed
Text
EB
Figure 15: GPT-5.2's turn-by-tum cognitive map in vision world during exploration.
31

---

Published as a conference paper at ICLR 2026
grben char
14
Dlack
DACSDACK
AOUTT
Stop
Stop 2
Stop 3
Stop
2+
Figure 16: Example trajectory illustrating GPT-5.2's door-finding strategy and systematic sweeping
patterm: Upon detecting a door, the agent navigates toward it and executes a strategic rotation maximize environmental coverage. The process terminates all target objects have been successfully identified.
scooter
shoppingcart
green
truck
airplane
teddybear
plant
laptop
toilet
television
gata
agent
Figure 17: Example trajectory illustrating GPT-5.2 5.2's omission pattern: Observing the door too early may lead the agent to skip the rest the exploration, causing incomplete environmental discovery.
32

---

Published as a conference paper at ICLR 2026
shoppingoat
groan chai
truck
tod0yoow
plant
laptop
Delevidoon
gata
Stop
Step Jamn
Stop
Step
Step
Step 7
Step 0
Step 9
Stop 10
Stop 11
Figure 18: Example trajectory illustrating GEMINI-3 PRO's door- mior-finding strategy and systematic sweeping pattemn in vision world: Upon detecting door, the agent navigates toward and executes strategic rotation to maximize environmental coverage. The process terminates once all target objects have been successfully identified.
scoutar
shopp-gcart
tryck
pate
agent
Stup
Stap
Stap
Stup&
Stop
Stop E
Stop 7
Stop
Step 9
Stop 10
50cp 11
Stop 12
Stop 13
Stop 14
Figure 19: Example trajectory illustrating GEMINI-3 PRO's object sweeping pattern mostly found in text world: Orbit the starting object using it the pivot point. Randomly select an observed door to jump to a new object, then resume pivoting around the new target in a continuous loop.
33

---

Published as a conference paper at ICLR 2026
Sur
Sugr
S
Supp
Skyp 10
Slup
Stap 12
Slap
Shep
Step 16
Shep
Figure 20: Example trajectory illustrating CLAUDE-4.5 SONNET'S exploration pattern: There is no clear exploration pattern.
Performance Charts
Information Gain Tur
Cognitive Map (Update)
Cogestive Map (Ful))
CogMap Accuracy
ente moness mepe rotemrree
Mop Infodain
Figure 21: Platform designed by us for analysis (chart)
34

---

Published as a conference paper at ICLR 2026
Dashboard Overview
Configuration Summaries
Semplix:
Exploration
Evaluation
Cogritive Map
Comelution
eng eevergge
ixploratton
<ogmap
ang Adge, coveraee:
0440
ta1t, nettic:
ing
Agalfarc
leg jaclenn dall Lado:
Fruil information,
www.eneeenccoo COUREE
Hearm:
2.000
(2A) coint
106
2017
Figure 22: Visualization Platform for analysis: Metrics for active exploration text world
vision sinion,activw.ee www.:rtttttttiiiee think
Cognitive
Cornelation
condutinn
lngg edge coverage
ing. LoURRgg
Iilibly
oxplonation stops
10270
ingg bettoon,, fet, rettor
0.045
ing Information, gale:
1723
Figure 23: Visualization Platform for analysis: Metries for active exploration in vision world
35

---

Published as a conference paper at ICLR 2026
Eevironenent Obscrvation diio hupiot fore-right,
MNSWER
State before Tur
Action
[KrepToptkee doory Oeserwe()
Cogalthee Map Response jas reulo
brier inleee-ctartin coarcinate leocu biraad besit pabditeema, heop agend blue dior objects mathwe bnttel origin anberpant decrwey cisarvation.
"agent" "postoon: "bongg"
Yang" "postoon"
Cognitive 350Ns
Predicted
Ground Truth ((Bserved)
Ground Truth (Ful))
State et
"postttent:
Hactigh:
Figure 24: Visualization Platform for analysis: one turn of active exploration in text- world, including agent's 's action and cognitive map.
36

---

Published as a conference paper at ICLR 2026
Envirasment Observation
curtentoctwise
exclenaion
actind)
Actions
Thisking
renatiing potestally
Stete before Turn
Actine (Robew(-40), Disenve]]
Global Cognittive Hap Response tnko loged
IiDW: heed miniral global cogethve mop indudii only entiiee observed so tor regent, 2 doorg arios lamo, trivitbo,, exp) Usa inttal agent postiio orginn apent curennl tiding east after three sp* rostions. Approdmate integer and postions consarunt with obsarvatorn as dbor
Cognitive Map 2SONs
Predicted
Sspenth:
State Turn
"Taciing
"Macan'ss
Tolu
Sdwartt:
"pprction"
Trarctson"
Spren duanis
"paiction's
"paictson"
Local Cognitivee Hap Response joia kagert
TH-NW Buts boil (agems cantri) map from the cumentt vine (ngent Bacing local frams), indudiigg only cummenty vitle dicns/ttpeds add leciig only for the talecidass
Observation L
Figure 25: Visualization Platform for analysis: one turn of active exploration in vision-
37

---

Published as a conference paper at ICLR 2026
E LLM USAGE STATEMENT
We used Large Language Model like GEMINI-3 PRO and GPT-5.2 as a writing assistant such as improving readability and phrasing. LLMs did not contribute novel content, proofs, results. All research ideas, analyses, experiments, and results were developed by the authors, who take full responsibility for the paper's accuracy and integrity.
38


Reviews:
[{"review_id": "7WsjkQWYgx", "reviewer": "Reviewer_uS5h", "summary": "The paper introduces Theory of Space (ToS), an evaluation paradigm for whether LLM/VLM agents can actively construct, update, and use an internal spatial belief (a cognitive map) through exploration. It builds a procedural multi-room world (text and vision), defines route/survey/update tasks, decouples exploration from reasoning using proxy agents, and probes the belief itself (serialized maps) rather than only task success. Empirically, top models perform well in text but struggle in vision. Active and passive performance are close, suggesting belief construction and maintenance (not coverage alone) is the bottleneck.", "questions": "- Space and ToM has been extensively studied in past psychology research. See https://arxiv.org/abs/2310.19619 for example. Your benchmark includes \"false-belief / perspective-taking\" tasks, but these seem to involve the agent's own belief about the environment rather than what another agent believes, i.e., ToS did address spatial belief of a single agent about objects and its own perspective, but did not appear to include multiple agents with independent goals or belief states. How do you justify this choice?\n- Can ToS-style belief actually improve downstream navigation/manipulation? JSON is a strong choice, and vision seems to play a minimal role here.\n- How sensitive are results to Query and termination policy? Could Query or early-stop heuristics be creating artificial gaps between active and passive? Maybe provide cost-sweeps over Query availability/penalties and termination criteria; plot accuracy vs. action cost curve?\n\n - - -\nI am happy to change my ratings if the above Weaknesses and Questions are addressed.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Concepts and definition contribution: ToS captures the ability to (1) construct a globally consistent belief from partial views, (2) update it as new evidence conflicts, (3) utilize it for downstream spatial tasks. The two-phase evaluation separates Exploration to gather observations and build a belief from Reasoning on route/survey/update tasks to test utilization. The task formulation itself is interesting, although this paradigm has been explored in early embodied AI benchmarks like EXCALIBUR (I knew EXCALIBUR evaluates where the agent goes; ToS evaluates what the agent infers and remembers, there is a difference).\n- Belief probing & metrics: Models output a cognitive map (JSON) that can be directly examined. The paper provides metrics for positional, directional, and facing accuracy, plus a composite score. It also introduces a formal Exploration Efficiency metric based on information gain with AC-3 pruning.\n- Interesting findings: Active performance roughly equals passive performance (but with premature stops). Active scores are close to passive, with gaps often stemming from premature termination and inefficient exploration, not pure lack of path coverage (see information-gain curves). Belief management matters: stepwise probing shows errors compound from perception to integration (egomotion) to stability. Text models rank higher on correctness/consistency, while in vision, perception is the bottleneck.", "weaknesses": "- The current environment relies on symbolic discretization: angles and distances are bucketed into categorical bins (e.g., {near, mid, far}) and rendered with calibration cues (reference grids, constant lighting). This yields clean supervision but strips away realistic sensory ambiguity such as partial occlusions, depth uncertainty, and texture variation. Models may appear spatially consistent only because the environment removes real-world ambiguities. This inflates performance and may not transfer to continuous or noisy perception (e.g., real egocentric videos or robot camera feeds).\n    - Possible action items: Add a continuous-valued variant where direction and distance are real-valued regressions rather than categorical bins; introduce sensor noise and texture perturbations (random lighting, viewpoint jitter, partial occlusion).\n\n- The results show a significant gap between text and vision performance, but the source of this gap (perceptual encoding vs. multimodal integration vs. belief updating) remains ambiguous. Without identifying the bottleneck, it's unclear whether improvements should target visual representation learning, grounding alignment, or belief maintenance logic.\n    - Possible action items: Implement modular ablations: Perfect-perception oracle (feed ground-truth object positions); Perfect-integration oracle (feed clean percepts, test memory/belief fusion); Perfect-stability oracle (test only inference drift).\n\n- Despite the paper's emphasis on ToS, the core scientific question (can an embodied agent build a consistent internal map from partial observations) closely parallels prior work such as EXCALIBUR (CVPR 2023) and Emergence of Maps in the Memories of Blind Navigation Agents (NeurIPS 2023). Both earlier studies explored spatial map emergence and information-gain-based exploration. The conceptual novelty is less about introducing a new phenomenon and more about repurposing it into an evaluation benchmark. Without explicit differentiation, readers may view ToS as incremental rather than foundational.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KqFGYIsN4L", "reviewer": "Reviewer_dNos", "summary": "This paper introduces the Theory of Space (ToS), a framework for evaluating the ability of foundation models to actively construct, update, and utilize an internal spatial belief of an environment from partial observations. The paper designed experiments to (1) ask an LLM/VLM agent to actively explore a partially observable environment for establishing the internal spatial model, (2) use a proxy agent script to derive a ``passive\" spatial model to disentangle active exploration from passive reasoning, and (3) ask the agent to represent its spatial belief for direct accessment. Evaluations across modern LLM/VLMs show a modality gap (stronger on text than vision) and diagnose failure modes like egomotion update errors and poor global map maintenance; humans outperform most models with modest exploration budgets.", "questions": "1. How do you ensure the cognitive map probing reflects the true internal state rather than post-hoc rationalization?\n2. Is the termination action fully decided by the LLM agent? What could be the cause that leads to agents stopping prematurely?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper designs spatial belief evaluation around task-agnostic, uncertainty-reducing exploration, rather than passive reasoning and goal-directed task completion, adding an active aspect to spatial belief construction. This makes the paper original. \n2. Spatial understanding under partial observability is central for embodied agents and planning. The ToS framework fills in the gap for LLM evaluation in enactive cognition by evaluating the goal-agnostic exploration of active LLM agents. \n3. Each experiment in the benchmark is well-designed for evaluating the representation, updating, and utilization of the internal space model. The metrics used for measuring the performance are reasonable, including the exploration efficiency against the AC-3 algorithm, belief quality assessment by correctness and consistency, and task success rate.", "weaknesses": "1. Relatively simple spatial environment and limited statistical results: most experiments use two connected $6$ by $6$ rooms with $9$ objects, and some with varying room size. While the small size of the spatial environment avoids memory capacity as a confounding factor for ToS performance, its simplicity could potentially trivialize the active exploration aspect of the agent. The generalization beyond the grid room is therefore unclear. As a consequence, there are also no statistical results on the performance. \n2. Exploration efficiency analysis: the exploration efficiency is mainly analyzed by the information-gain curve. It's unclear whether limited exploration thoroughness originates from policy or belief integration failures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AfLIBfZlea", "reviewer": "Reviewer_9VR4", "summary": "This paper introduces a new Theory of Space framework that tests the agent's ability to learn the unobserved structure of its spatial environment from partial observations.\nThey implement a benchmark including text-based and visual environments for curiosity-driven exploration, and prompt (SOTA foundation) models to present their cognitive map at each step to track the underlying spatial model.\nThey document common failure modes in how models' form spatial beliefs, including path integration errroe and incorrect map inference.\nThe primary focus of the benchmark is exploration: the agent must make its own decisions on what to observe next.\nThey find that the models perform much better in pure text-based worlds compared to visual ones.", "questions": "How would foundation models compare to human performance? \n\nIs it possible to design a human study where people navigate the same worlds? \n\nIt is worth considering that while human map drawings are notoriously poor, their navigation abilities are often excellent. Have the authors considered to what extent this pattern of knowledge accessibility may be parallelled in machines?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "Theoretical soundness: The paper is well written and well grounded in cognitive theory.\n\nTechnical soundness: A stack of tools and technologies is integrated effectively to design the ToS benchmark. A large number of models is evaluated", "weaknesses": "This work aims to produce a benchmark for human-like spatial understanding and exploration. While this motivation, and the ToS approach as a means of achieving it are very well theoretically justified, the paper offers no evidence of how a human would behave in these tasks.\n\nThe models are compared to two proxy agents, which they claim to execute a theoretically optimal path, to establish an upper bound on exploration ability.  However, no proof of proxy agent optimal is given.\n\nBoth weaknesses arise from overstated claims. While would take a lot of work to support these claims, the paper would be still valuable if the claims were more accurately reframed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xbG5B5P2OB", "reviewer": "Reviewer_TnFX", "summary": "This paper argues that today’s VLM agents excel at passive or task-driven spatial reasoning but largely fail to build a coherent understanding of space through exploration. It proposes “Theory of Space” (ToS): an agent’s ability to construct, update, and use an internal spatial belief from partial, egocentric observations, and introduces a benchmark that makes agents actively explore text and vision environments (procedural multi-room layouts in a grid world and a ThreeDWorld+Objaverse simulator).\n\n At every step the agent must externalize its belief as a cognitive map (JSON), which affords assessing not just task accuracy but belief correctness, consistency, and stability. The suite spans route and survey tasks (e.g., pairwise relations, action-to-view, allocentric mapping, mental rotation, localization) and adds belief-update probes via perspective taking and false-belief variants. To disentangle exploration from reasoning, they compare active on-policy agents with passive runs over standardized proxy trajectories (SCOUT for vision, STRATEGIST with AC-3 for text) and score exploration efficiency via information gain or coverage. Results across several state-of-the-art models show strong text-world performance but large drops in vision, common failures in ego-motion updates and global consistency, and a tendency to terminate exploration too early, highlighting the need for agents that manage spatial belief as a first-class capability.", "questions": "The statistical robustness is unclear: the paper mentions 100 seeds but not per-task item counts, variance across seeds, or multiple-comparison controls. Given many tasks and modalities, some reported gaps could reflect sampling and prompt stochasticity rather than stable capability differences.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1.  While the setup is admittedly a bit toyish, the core capability it targets—actively constructing, updating, and using a spatial belief from partial egocentric views—is exactly what real robots need to operate safely and robustly in cluttered, changing environments; from home assistants that must map and remap rooms, to warehouse and hospital robots that navigate multi-room spaces with moving obstacles, the paper’s emphasis on belief management, uncertainty reduction, and allocentric–egocentric conversion translates directly into high-value, real-world behaviors.\n\nS2. The paper cleanly isolates “active belief construction under uncertainty” as the missing capability between passive reasoning and task-driven foraging, giving the field a crisp target. Also, they require stepwise cognitive-map JSON forces models to externalize internal spatial state, letting evaluators inspect correctness, consistency, and stability—not just final-task accuracy.\n\nS3. Dual-modality design: mirrored text and vision worlds over identical layouts quantify the real modality gap and let researchers attribute errors to perception vs. spatial inference.", "weaknesses": "W1. Discrete bins (5 egocentric angle bins, 6 distance bins, 8 allocentric directions) compress geometry in ways that can hide small-but-critical errors (or inflate accuracy by near-miss snapping); no sensitivity analysis shows how bin granularity changes rankings or conclusions.\n\nW2. Layouts are grid-based with tree-structured room graphs (no loops), weakening tests of global consistency (e.g., cycle closure) and long-range metric drift—the very failure modes cited (egomotion, global map maintenance) would be more exposed in loopy topologies and irregular geometries.\n\nW3. The authors grant an action that returns ground truth, acknowledge cost-minimization under a step budget, and evaluate information gain via AC-3 in text mode—yet they don’t ablate Query usage, price, or availability. That omission makes it hard to know how much of the reported “spatial belief construction” truly reflects inference from partial observations versus selective peeks at the answer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)