Does Spatial Cognition Emerge in Frontier Models?
Authors: Santhosh Kumar Ramakrishnan
Erik Wijmans
Philipp Kraehenbuehl
Vladlen Koltun
Venue: ICLR 2025 Poster (2025)
Keywords: Frontier models
spatial cognition
Primary area: foundation or frontier models, including LLMs
TL;DR: We present SPACE, a benchmark that systematically evaluates spatial cognition in frontier models.

Abstract: Not yet. We present SPACE, a benchmark that systematically evaluates spatial cognition in frontier models. Our benchmark builds on decades of research in cognitive science. It evaluates large-scale mapping abilities that are brought to bear when an organism traverses physical environments, smaller-scale reasoning about object shapes and layouts, and cognitive infrastructure such as spatial attention and memory. For many tasks, we instantiate parallel presentations via text and images, allowing us to benchmark both large language models and large multimodal models. Results suggest that contemporary frontier models fall short of the spatial intelligence of animals, performing near chance level on a number of classic tests of animal cognition.

Published as a conference paper at ICLR 2025
DOES SPATIAL COGNITION EMERGE IN FRONTIER MODELS?
Santhosh Kumar Ramakrishnan' Erik Wijmans
Philipp Krähenbühl
Vladlen Koltun
Apple
ABSTRACT
Not yet. We present SPACE, a benchmark that systematically evaluates spatial cogmition in frontier models. Our benchmark builds decades of research in cognitive science. It evaluates large-scale mapping abilities that are brought to bear when an organism traverses physical environments, smaller-scale reasoning about object shapes and layouts, and cognitive infrastructure such as spatial atten- tion and memory. For many tasks, we instantiate parallel presentations via text and images, allowing us to benchmark both large language models and large mul- timodal models. Results suggest that contemporary frontier models fall short of the spatial intelligence of animals, performing near chance level on number of classic tests of animal cognition.
INTRODUCTION
Frontier models have achieved impressive performance in mathematics, coding, general knowledge, and commonsense reasoning (Hendrycks al., 2021a b Chen al.. 202T; Sakaguchi et al., 2021 Yue et al. 2024). This remarkable progress has inspired characterizations of frontier models as pos- sessing the intelligence of a smart high schooler and predictions of the imminent arrival of super- intelligence (Aschenbrenner 2024). These characterizations are often underpinned by the premise that competence (or even mastery) in some aspects of cognition is symptomatic of broad cognitive competence. This is not self-evident. To quote Brooks's first law of artificial intelligence, "When an Al system performs a task, human observers immediately estimate its general competence in areas that seem related. Usually that estimate is wildly overinflated." (Brooks) 2024).
Our work focuses on spatial cognition, a foundational form of intelligence that is present in a broad spectrum of animals including humans (Marshall & Fink 2001; Waller & Nadel, 2013 Mallot 2024). Spatial cognition refers to the ability of animals to perceive and interact with their surround- ings, build mental representations of objects and environments, and draw upon these representations to support navigation and manipulation. Decades of research in animal cognition have character- ized the spatial cognition of mice, rats, bats, pigeons, corvids, dogs, wolves, elephants, marmosets, tamarins, howler monkeys, baboons, chimpanzees. and humans [Tolman 1948 Menzel
1973 Pe-
ters. 1974. Gillner& Mallot, 1998| Marshall & Fink [2001] NNoerr Byrne
[007] Tommassi al.
2012 Porter & Garber, 20133 Blaser et al., 2013] Geva-Sagiv et al.] 2015
Presotto et al., 2019
de Guinea et al. 2021| Payne et al. 2021] Xu et al.
2024 Xavier et al., 2024
Welklin et al. 2024).
Human infants already possess rudimentary spatial cognition, which subsequently improves along
developmental schedules that have been characterized [Blades & Spencer 1994, Newcombe 2000
Vasilyeva & Lourenco 2012). Spatial cognition is known to underpin more advanced cognitive
abilities (Kozhevnikov et al.] 2007 Newcombe 2010 Young et al. 2018).
The emergence of spatial cognition has been linked to embodiment (Smith & Gasser 2005 Jansen & Heill 20100: Frick & Möhring, 2016).. without which the development of spatial cognition may be impaired (Foreman al. 1990] Anderson al., 2013). However, frontier models are typically trained in a disembodied manner on corpora of text, images, and video. Does spatial cognition emerge in disembodied frontier models? To study this question systematically, we develop SPACE, a benchmark that builds on decades of research in cognitive science. Our benchmark comprises two
"Corresponding author; damakrishnan@apple..c

---

Published as a conference paper at ICLR 2025
Large-scale spatial cognition
Small-scale spatial cognition
Distance estimation
Map sketching
Perspective taking
Mental rotation
Minnesota paper form board
Selective attention
Direction estimation
Route retracing
Maze completion
Judgement of line orientation
Corsi block tapping
Water level
Shortcut discovery
Spatial addition
Cambridge spatial working memory
Spatial visualization
Spatial perception
Visuospatial working memory
Spatial orientation
Selective spatial attention
Interactive task QA task
Spatial Perception And Cognition Evaluation. We design suite of spatial cognition tasks based on the cognitive science literature. These are broadly classified into large-scale and small-scale spatial cognition. Large-scale tasks require understanding space at the level of environments and evaluate spatial orientation and cognitive mapping abilitics. Small-scale tasks require understanding space the level of objects or object arrangements and evaluate skills such as spatial visualization, spatial orientation, spatial perception, selective spatial attention and visuospatial working memory. These tasks can be multiple-choice question answering, or interactive games. We develop multimodal as well as purely textual presentations, which support evaluation of both large language models (L..Ms) and vision-language models (VI.Ms).
Figure I: SPACE:
broad classes of tasks, covering large-scale and small-scale spatial cognition (Hegarty et al. 2006 Meneghetti et al. 2022 Newcombe 2024). See Figure for overview.
Large-scale spatial cognition has to do with model's ability to understand its surroundings. In large-scale tasks, the model is familiarized with environment and is then asked to estimate dis- tances and directions to landmarks, sketch a map of the environment, retrace a known route, or discover shorteut to a goal. Small-scale spatial cognition has to do with model's ability to per- ceive, imagine, and mentally transform objects in two or three dimensions. Together, large-scale and small-scale tasks evaluate core cognitive abilities such as spatial perception, visualization. orienta- tion, selective attention, and visuospatial memory (Lacroix et al. 2021 Meneghetti et al. 2022).
We design text-based and image-based presentations to evaluate both large language-only and vision-language models (LLMs and VLMs, respectively). Our results indicate that contemporary frontier models have not yet reached competency let alone mastery key large-scale spatial cognition tasks, frontier multimodal models perform near chance level, even when presented with an allocentric (map) view of the environment. The strongest models exhibit much better performance on some small-scale tasks that evaluate selective spatial attention and vi- suospatial working memory, especially with purely textual presentations via character arrays, but perform near chance on other tasks such as mental rotation (Vandenberg & Kuse, 1978), perspec- tive taking (Kozhevnikov & Hegarty 2001), maze completion (Lacroix et al. 2021), l), or the classic Minnesota Paper Form Board test (Likert & Quasha 1941 1969).
spatial cognition. On
RELATED WORK
Spatial cognition. Spatial cognition is a branch of cognitive science that seeks to understand how humans and animals perceive, interpret, represent, and interact with objects and environments (Mar-
shall & Fink 2001 Landau 2002 Waller & Nadel 2013 Mallot 2024 Newcombe (024).. This involves the perception of object sizes, shapes, and scales, as well as the relationships between ob- jects and landmarks in the environment (including location, distance, direction, and orientation). Spatial cognition is broadly divided into two categories; large-scale and small-scale (Hegarty al.) [ 2009 Meneghetti et al., [2022] Newcombe, (224)). Large-scale spatial cognition refers to the ability to build spatial representations of environments and use them effectively for naviga- tion and spatial reasoning. Large-scale spatial cognition tasks typically involve egocentric spatial transformations, where the viewer's perspective changes with respect to the environment while the spatial relationships between parts of the environment remain constant (Wang et al., 2014). Small- scale spatial cognition refers to the ability to perceive, imagine, and mentally transform objects or shapes in 2D 3D. This is typically evaluated using paper and pencil tasks that require allocentric spatial transformations of objects and shapes (Wang et al. 2014). While large-scale spatial cogni-
tion has been demonstrated in wide range of animals (Tolman] 1948 Menzel 1973 Peters 1974
2

---

Published as a conference paper at ICLR 2025
O' O'Keefe & Nadel 1978 Gillner& & Mallot, 1998 Richardson et al. 1999; Geva-Sagiv et al. 2015 Toledo et al. 2020). D. the study of small-scale spatial cognition is specific to humans.
Emergent spatial representations. Several works have shown that spatial representations. a phc- nomenon similar to spatial cogmition, can emerge in neural networks (Banino et al.) 2018 Cueva &
Wei 2018 Wijmans al., 2023 Sorscher et al. 2023 These works train a neural network from scratch for path integration or navigation tasks and analyze the model weights to identify spatial representations.
Spatial reasoning in large language models. PlanBench (Valmeekam et al.) 2024) and Co- gEval (Momennejad et al. 2023) evaluate LLMs on text-based planning tasks such navigation, delivery logistics and block stacking to evaluate cognitive mapping and planning. Yamada et al. (2024) evaluate spatial reasoning LLMs by performing map traversals on different types of graphs and evaluate the model's
ability. EWOK (Ivanova et al. 2024) studies spatial plau- sibility reasoning in LLMs. In comparison to these benchmarks, SPACE evaluates broader array of cognitive abilities and implements multimodal presentations of classic animal cognition experi- ments.
Benchmarks for large multimodal models. The recent successes of multimodal models (OpenAl Li et al.] 2024a] Reid et al. 2024) have been facilitated by large-scale training on text and
2024
multimodal corpora (Ranal 2010 Together Computer 2023 Chen et al. 2023; Laurençon et al.
2023
Gadre et al. 2023). followed by tuning on humaan preferences (Liu et al. 2023a [Aaddllaa
al.
2024| Ouyang al.; 2022 Rafailov et al., 2023). The remarkable advances in the capabiliitess
of these models inspired a variety of benchmarks that evaluate their performance. Early multimodal benchmarks consisted of single-task datasets such as visual question answering (Antol al. 2015
Goyal et al. 2019: Marino et al. 2019) and image captioning (Chen et al.)
2015). However, due to the limited scope of early datasets and concerns regarding potential test-data leakage, newer benchmarks use diverse collections of tasks (Fu et al. 2023 Yu et al., 2024 Liu et al. 2023b; Yue et 2024 Lu
al.
2024 Ying et al., 2024). While these datasets primarily focus on image understanding. newer
datasets that emphasize spatiotemporal reasoning have been proposed for video (Li et al. 2024b Fu
ct al. 2024a Majumdar et al. 2024).
Recent studies highlight number of shortcomings of frontier multimodal models (Moskvichev jet al., 2023] [Ton et al., 2024] Chen et al. 2024a] Fu et al. 2024b). One such shortcoming that models may not perceive the image in detail, often missing fine-grained details ignoring the image entirely (Chen et al.) 2024b Guan et al. 2024, Tong et al., 2024p. HallusionBench proposes a new dataset of image pairs, where tiny edits made from image another that change the answer to the question (Guan et al. 2024). MMVP identifies issues with CL.IP-based pretraining of visual encoders, which make current models blind to certain visual patterns, and proposes a benchmark of CLIP-blind image pairs where the same question has opposite answers (Tong et al.] 2024). MMStar shows that many questions in multimodal benchmarks can be answered correctly without the image and proposes a new split of existing benchmarks that addresses this issue (Chen et al. 20246).
Another shortcoming of existing models is their lack of spatial perception and reasoning Chen al., 2024a] Cheng et al. 2024). SpatialVLM proposes VQA dataset that requires answering questions about relative spatial arrangements and metric relationships (Chen et al.) 2024a). SpatialRGPT fur- ther includes region-level understanding (Cheng et al. 2024). MOCHI evaluates the ability of vision objects (Bonnen et al. (025)). 'Percep- tion test' aims to overcome shortcomings of standard video datasets by creating a diagnostic dataset where participants record videos while following complex scripts depicting interesting events (Pa- fraucean et al. 2023). It evaluates fundamental perceptual skills (memory, abstraction, intuitive physics, and semantics) and various types of reasoning.
models to identify rotated versions of eeeedurally-generraee
Another line of work considers skill acquisition (the ability to learn skill and apply it to new sce- narios). Prior work has studied this using visual analogical reasoning (Chollet 2019 Moskvichev et al.) 2023 Yiu et al. 2024). The ARC dataset contains samples consisting of few examples of abstract grids and their transformations and one or more test inputs (Chollet 2019). The objective to understand the transformation performed using the examples and apply it test inputs. The transformations have been further organized into specific concepts with varying degrees of difficulty
in the ConceptARC dataset (Moskvichev al. 2023). Inspired by ARC and developmental psy-
3

---

Published as a conference paper at ICLR 2025
chology, the KiVA dataset studies visual analogies in the context of visually realistic 3D shapes with concepts like transformations in color, size, rotations, reflections, and counting (Yiu et al. 2024).
3 SPACE: A BENCHMARK FOR SPATIAL PERCEPTION AND COGNITION EVALUATION
We develop benchmark for evaluating the spatial cognition of frontier models. The benchmark comprises large-scale and small-scale tasks and is designed for compatibility with both text-only and multimodal models.
3.1 LARGE-SCALE SPATIAL COGNITION
In large-scale spatial cognition tasks, we evaluate the ability of models to build spatial representa- tions of their surrounding environment, and whether they can use these representations to reason about and navigate in the environment. There are two stages to these tasks. First, we familiar- ize the model with an environment by showing a video walkthrough [ The model must build a mental representation of the environment that captures the locations of start, goal and landmark lo- cations, and their spatial relationships. After the model is familiarized with the environment, we evaluate the model's spatial representation using five tasks derived from the cognitive science liter- ature Meneghetti et al. 2022). See Figure 2 top) and Figure 3 for an overview.
1. Direction estimation. The goal is to determine the directions to other landmarks from given landmark. The participant is asked to pretend that they are facing landmark A, and then asked to estimate the direction (in degrees) to another landmark B. This is known pointing trial in the cognitive science literature (Allen et al.) 1996, Hegarty et al., 2006; Pazzaglia & Taylor, |007| Weisberg al.) 2014 Meneghetti et al. (0116). We formulate this multiple-choice QA lask with four options for the direction (only one correct option).
Distance estimation. The goal is to determine the straight-line distances from one landmark to all other landmarks (Allen et al.) 1996 Hegarty et al. 2006). The participant is asked to pretend that they are facing landmark A, and then asked to estimate the Euclidean distance to all the other landmarks. We pose this multiple-choice QA with four options for the list of distances to each landmark. Since current models are not good at estimating metric measurements (Chen) let al. 2024a Cheng al. 2024). generate incorrect options such that the ratios of distances between landmarks are not preserved, making it easier to identify the correct option.
Map sketching. The goal is to draw a map of the environment that contains the start, goal and landmark positions (Allen et al.) 1996; Hegarty et al. 2006 Pazzaglia & Taylor 2007 Weisberg 2016) 2021). We formulate this as multiple-choice QA with four options for the map sketches. The correct option preserves the true spatial relationships between the different map elements, while the incorrect options skew the spatial relationships randomly.
et al. 2014, Meneghetti et al.
Route retracing. The goal is to retrace the route shown in the video from the start to the goal (Allen et al., 1996. Pazzaglia & Taylor 2007. Meneghetti et al. 2016 2021). This task evaluates the model's 's ability to remember landmarks seen in the route and the actions required along the route reach the goal. We formulate this as an interactive task where the model re- ceives the current observation, decides which action to take, and receives updated observations based on the actions taken. We measure performance using the SPI. metric (success weighted by path length). which penalizes the model for taking unnecessary detours (Anderson et al. 2018 (The demonstrated route, which the model must retrace, always the shortest path to the goal.)
5. Shortcut discovery. The goal is discover shorteut (i.e., route never observed before) from the start to the goal after observing a video walkthrough that takes detours to reach the 1996 Pazzaglia & Taylor 2007; Meneghetti al. 2016 2021). The ability to discover shortcuts in familiar environments is key indicator of cognitive mapping ability (Tolman 1948). When designing environments and walkthrough paths, we ensured that novel shortcut exists that the model can exploit. Similar to route retracing, we treat this as an interactive navigation task and measure performance using the SPL metric.
goal (Tolman 1948) Allen
For text-only models, the "video walkthrough' is sequence of discrete map observations presented as arrays of characters, Figure for examples.
4

---

Published as a conference paper at ICLR 2025
Video walkthrough
Large-scale spatial coguition
Dircction estimation
Map sketching
Route retracing
au dhow aithorudg
arzahona vidos malltiradi dcoocatraing placad theant Retnoe
q: threel lus yourr my tha Arstur dardkee on digrco a ih aumgeetio adlanc you?
though entiravent Stoud the eisiis wore with locationaa fou cholee bedtaiiion
Pick
oet Dn 1I 41
Stan
X
chert
slan
Distance estimation
Shortcut discovery
Yeu mu saoom vida wnlicenggh
Qr hocked tha we ras Whet the badelou dekeneoo de metea stenge dhed Al 8. C.1..25
XCGll
X/fow XCeal
long with wiloomooowar Jover Yea muplacalal stat bomans akunta phl.
BaugX X
Sani Shongy cass
Do x
Fringy cless
Br 18
Daay
Swonge Chos
Stoee
X
Small-scale spatial cognition
Mental retation (MRT)
Perspective taking (PTT)
Maze completion (MCT)
Water level (WLT)
Viuaal
Prined thr sae daclooii seghtii
Ageit poslics
level intl Notee oceihic
Grd posiion
apple loardd iolatie your
image slowa degeemecddeee ID1
Prstend the counding Daing seglen fegres)
5.
Quel pastin
Textual
2-
5-
5-2
I- free spano
2-
Vice aooplaced Namigaic gna
nead a 3D1
Cy
15-155
g
Minnesota paper
Judgement of line orientation (IL.O)
Seleetive attention (SA1))
Corsi block tapping (CBTT)
Wieds of bden maadhikk
Jebernc
Whran (wwe ood
Jehonce
angi biteecs De tmo above?
Voual
G and
Dol and
Thetep lift god
piza ruuied cippod.
piece
dencei 18.00.
Ware isa on of boce. Ta bease tapped
cowperscinggee tryclen)
1 Zucl D.
Darte bo
Arno pka0
boatea of The ln alanset the
Hos oft boseere thight
thase They marge sith adges clat place can rounud
Which choice bas saras uaple
Spotial uddition (SAdd)
Cambridge spatial working memory (CSWM)
baagy moth @ecillass asc
Quil we 1 cals distuctioe
Fia tocoance Note The twe iea vantcedl ale and lar Acmembar too havolin puaitees
now regaoees in
0.0.0
Testal
What the sum the awosmoos
creescmen
breeatiec opon benes convere imngend fowdleddor trssess (7), Afler toworois Send. a rovved to now emppeies spe
ganns destuuatt in ticrurt.
Heagen change renbenly
Spotial visualization
Spatial orientation
Spacial perception
Selective sputial attention
Visuospatial working memory
The tasks in SPACE, For all tasks (other than the water level test), we include multimodal as well as purely textual presentations, to support evaluating both large language models (LLMs) and languae models (VLMs). For large-scale tasks, we visualize examples from the egocentric image presentation here and visualize alternate presentations in Figure 3 For small-scale tasks, visualize both visual and textual presentations here. Bolding characters in the arrays is for illustration purposes only.
Figure 2:
5

---

Published as a conference paper at ICLR 2025
Ego image
Discrete image
Discrete map lext
Obstacles
1,0,0, 2,
Obstacles Navigable spuce
Navigable space
Current position ,, 4,
Current position
2,0.C,
A-2 Landmarks
Landmark
Figure 3: Large-scale spatial cognition. We design environment layouts based experimental protocols cognitive science. top row shows bird's-eye renderings these environments. To evaluate large- scale spatial cognition in frontier models, we implement three observation spaces: egocentric image, discrete map (DM) image, and discrete map (DM) text (see bottom row). Ego image shows lirst-person view within the environment. DM image shows quantized, allocentric bird's-eye view of the 2.5m x 2. 2.5m region centered on the current position. Unlike ego image, DM image enables performing the large-scale tasks simplified setting without requiring perspective geometry. DM text depicts the DM image using text characters. We evaluate multimodal models using ego image and DM image, and large language models using DM text.
3.1.1 IMPLEMENTATION
3D environment generation. We create ten environment layouts based on prior work in cognitive science and artificial intelligence (Tolman) 1948 Gillner & Mallot 1998 Richardson et al., 1999
Banino et al.] 2018; Bouchekioua et al.) (021). Figure shows bird bird's-eye s-eye view images of each layout. See the appendix for more details about the environment generation process.
Observation spaces. We create multiple observation spaces to support evaluating both text-only and vision+text models. These are egocentric images, discrete map (DM) images, and discrete map (DM) text presentations.
Ego image. The environment is captured using forward-facine perspective camera placed at the model's location in the environment. This similar the setup of an animal navigating through an immersive environment and requires understanding perspective geometry.
DM image. This is a quantized bird's-eye view image of a 2.5m 2.5m area the environment surrounding the model's location. This is akin to human using map to navigate. The current location always at the center of the DM image. We use Pacman-like coloring scheme high- lighting the obstacles, navigable space, current postion, and landmarks. DM image simplifies the mapping process by removing the need for perspective geometry understanding.
DM text. This is translation of DM image an text array. We carefully select the text encoding to ensure compatibility with text tokenizers of popular models and ensure that each element the array is encoded by the tokenizers of all evaluated models distinct token.
See Figure (bottom) for examples of these presentations. The first two observation spaces used for models that support visual inputs, while the last observation space used for text-only models. See the appendix for additional illustrations of these tasks and dataset statistics.
3.2 SMALL-SCALE SPATIAL COGNITION
In small-scale spatial cognition tasks, we evaluate the models' ability to perceive, imagine, and mentally transform objects or shapes in two and three dimensions. We build on the body of work on visuospatial abilities, which are evaluated in humans via paper-and-pencil tasks (Allen et al. 1996 Weisberg et al. 2014 Meneghetti et al. 2022). These abilities may be used to explain individual differences between participants in large-scale scale spatial cogmition (Meneghetti et al. 2022 ) We de- fine ten small-scale tasks to evaluate abilities such spatial perception, spatial visualization. spatial orientation, selective attention, and visuospatial working memory. See Figure bbotomm for illus- trations of each task. We summarize each task below and provide additional details in the appendix.

---

Published as a conference paper at ICLR 2025
6. Mental rotation test (MRT). This is a test of spatial visualization, i.e., the ability to mentally manipulate 2D or 3D stimuli Vanddnberg & Kuse 1978). In the visual presentation, a reference 3D shape from Shepard & (ectzler[[19]]] is provided along with four choices. The correct choice is a rotated version of the reference, and the remaining choices are rotated versions of an alternate shape. The goal is to identify the correct choice from the distractors. The text-only version of this task uses 2D character arrays, akin to the card rotations test from |French et al. (1963).
7. Perspective taking test (PTT), This is a test of spatial orientation, i..., the ability to imag- ine being in a different position in space and seeing the surroundings from a new perspec- tive (Kozhevnikov & Hegarty 200T). We place N randomly-sampled objects (like apples, bats, dogs, books, grapes, etc.) at random locations in an image (with no overlap between objects). The objective is to take the perspective of standing next to an object (say, a bat) facing another object (say, a book), and determine the relative orientation of a third object (say, an apple). This is a multiple-choice QA with four options (only one correct option).
8. Water level test (WL.T). This is test of spatial perception (Piaget et al., 1957). Originally, it was designed to evaluate children's 's knowledge about the horizontal nature of the surface of water in sealed bottle regardless of its orientation. Performance on the water-level test was found to be related to performance on spatial ability tests (Foltz) 1978; Wittig & Allen 1984). We present the model with an image of a water container partially filled with water and ask to imagine the position of the water if the container were tilted. We implement this as four-way multiple- choice QA, where each choice is an image showing the tilted container with varying water levels. The objective is to select the one choice that shows the correct water level.
9. Minnesota Paper Form Board test (MPFB). This is a test of spatial visualization, where the model must perform multi-step manipulations of complex spatial information (Meneghetti al. 2022). Specifically, we provide the model with pieces of a figure and ask it to identify how the pieces fit together (Likert & Quasha 1941 1969). We programmatically segment square into five pieces, and rotate the pieces randomly to generate the final segments. We generate alternate segmentations of a square as negative choices for a multiple-choice QA presentation.
10. Judgement of Line Orientation test (JLO). This is a test of spatial perception (Benton 1994). where a model must determine the angle between two lines in image. Our visual presentation shows two lines in an image along with a set of 11 reference lines. The objective is to determine the pair of reference lines that have the same angles between them the lines in the image. This is presented as a multiple-choice QA with four choices (only one of them correct). Our text-only presentation implements the tasks via lines embedded in 2D integer arrays.
11. Selective attention task (SAtt). This is test of selective spatial attention, i.e., the ability to selectively attend to a particular region of space while ignoring others (Serences & Kastner]
2014 Pahor et al. 2022). In particular, we use the widely used cancellation task, where the goal
is to search for and mark out target stimuli embedded amidst distractors (Della Sala et al. 1992
Brickenkamp & Zillmer 1998, Dalmaijer et al.) 2015 Lacroix et al.) [021 Pahor et al.
2022
Kalina & Walgrave 2004). We design the task as multipl--hhoie QA with objects as the stimuli for visual evaluation and characters as stimuli for text-only evaluation. The target stimuli and
distractors are arranged on a grid. The answer must be selected from one out of four options. The correct option lists the (row, column) pairs that localize the target stimuli in the grid.
12. Maze completion task (MCT). This is an interactive game to evaluate spatial orientation, plan- ning, and executive functioning (Lacroix et al.) 2021). We programmatically create mazes using Mazelib (Stilley) 20014) and render them using a Pacman- color scheme for the visual pre- sentation and character array for the text-only presentation (similar to DM image and DM text in Figure 3) Using the maze rendering, a model must sequentially select an up/down/left/right action to reach the goal and execute a stop action to successfully complete the task. If the model does not reach the goal within 250 actions, it is considered to have failed. We measure the suc- cess rate, i.e., the percentage of mazes where the model reaches the goal within the allotted time.
13. Corsi block- block-tappie tapping task (CBTT), This is a test of visuospatial working memory and atten- tion (Corsi) 1972, Claessen et al. 2015). We create a digital Corsi board with N blue-colored blocks that are randomly placed on the board with no overlap (N E 8)). We randomly sample sequence of K taps, where each block is tapped most once (K E [4. The taps digi- tally rendered on the blocks by highlighting them in yellow when tapped, yielding an sequence of K images. After presenting the K images, we provide rendering of the board with integer IDs
7

---

Published as a conference paper at ICLR 2025
assigned to each block and ask the model to reproduce the sequence of taps using these IDs. We treat this as multiple-choice QA with four choices of tap sequences, only one of which is correct.
14. Spatial addition task (SAdd). This is a test of visuospatial working memory, i.e,, the ability to store and manipulate spatial information in memory (Wechsler, 2009). The model is presented with two 2D grids, where each grid location can be empty or contain a blue or red dot. The objective is to add the two grids together by following certain rules. If grid location has blue dot in exactly one of grids, the result should be a blue dot. If grid location has blue dots on both grids, the result should be a white dot. Red dots are distractors and must be ignored. We programmatically generate grid pairs with sizes sampled from {3, 5, 7,9) 9} and pseudo-randomly populate them with blue and red dots. We formulate the task as multiple-choice QA, presenting four grids as possible answers, exactly one of which is correct.
15. Cambridge spatial working memory test (CSWM). This is an interactive game that evaluates visuospatial working memory (Sahakian et al. (988). The model is presented with an image con- taining N blue colored boxes (NV E [3. 7J]. A yellow 'treasure" is initially hidden in one of the boxes. The model must sequentially select boxes one at a time to find the hidden treasure. Once the treasure is found, another treasure is placed in one of the remaining boxes. The objective is to locate all the yellow treasures via a process of climination. We programmatically generate in- stances of this task by randomly sampling blue boxes, placing them at random locations (without overlap), and placing the treasures in each box in random order. At each step. we assign random integer IDs to each box a reference for selecting box. The boxes' integer IDs randomized in each step, forcing the model to remember boxes based on their spatial positions. When the model finds treasure, the box containing the treasure becomes yellow. The model must find all the treasures before a time limit T (determined based on N) to succeed.
As with large-scale spatial cognition, we also implement purely textual presentations of these tasks to support evaluation of large language models (LLMs). Figure Jillusratts both the multimodal and the purely textual presentations. The key idea in instantiating the textual presentations is to encode all spatial information via 2D character arrays. We did not identify a natural such encoding for the water level test (WLT) and did not include text-only presentation for it for this reason. See the appendix for additional illustrations of these tasks. In some tasks, such as MRT, MPFB, and JLO, the text presentations substantially easier than the corresponding visual presentations. However, the visual and textual presentations match closely for the remaining tasks, enabling us to identify modality-specifi limitations of multimodal models by evaluating them on the two presentations.
EXPERIMENTS
Baselines. We evaluate number of LLMs and VLMs. Using text-only presentations, we evalu- ate GPT-4v and GPT-4o (OpenAI 2023 2024). Claude 3.5 Sonnet (Anthropic) 2024), the Llama3 family (Dubey et al. 2024), Mistral models such as Mixtral 8x7B, Mixtral 8x22B, and Mistral 123B (Jiang et al. 2024 Mistral Al team 2024a), and two Yi 1.5 models (Young al. 2024). Us- ing multimodal presentations, we evaluate GPT-4v and GPT-4o (OpenAl, [023] 2024). Claude 3.5 Sonnet (Anthropic 2
(Li et al. 2024a), Pixtral 12B Mistrrll Al team 2024b), and Phi-3.5-vision (Abdin et al. 2024). We use the vLLM inference engine for evaluating the open-source models (Kwon et al. 2023). For each task, we implement prompt that provides a detailed description of the task and the expected response format (see the appendix). We also list the results of chance baseline that selects an answer at random. For multiple-choice QA tasks, chance is at 25%. For interactive tasks, the chance baseline samples an action at random in each step. We further include human performance for reference for the multiple-choice QA tasks. See the appendix for additional implementation details.
Large-scale spatial cognition results. The results are shown in Table I grouped by presentation modality (ego image, DM image, DM text). For masee presentations, we evaluate Claude 3.5 Sonnet, GPT-4v and GPT-4o because they support video understanding (via a succession of im- ages). For DM text, we evaluate both open and closed LLMs. We also list the performance of the chance baseline for calibration, as well as human performance (see the appendix for details). In the text-only modality, Claude 3.5 Sonnet attains the highest average performance. Mistral 123B is the gighest-performing open model. All evaluated models struggle with large-scale spatial cog- nition, falling significantly below human performance on direction estimation, distance estimation,
8

---

Published as a conference paper at ICLR 2025
Observation space: Ego image
Method
Direction estimation
Distance estimation
Map sketching
Route retracing
Shortcut discovery
Average
Human
82.8
83.2
96.6
GPT--4o
25.0
33.3
6.6
6.4
23.0
Claude 3.5 Sonnet
29.0
34.4
#2.9
27.5 s6.3
7.4 #2.8
0.0 40.0
19.6
GPTT-4v
29.7 10.3
31.9 #2.7
20.0 2:18
1.6 11.2
3.9 40.9
17.4
Chance
25.0
25.0
25.0
15.0
Observation space: DM image
Method
Direction estimation
estimation
Map sketching
Route retracing
Shortcut discovery
Average
Human
82.9
82.5
100.0
GPT-4o
29.5 $5.5
31.9 #1.0
33.3 #3.3
23.6 +a.1
25.9 =2.0
28.8
Claude 3.5 Sonnet
32.5 $2.3
40.0 #2.6
30.0 $4.1
15.4 64.9
13.7 20.4
26.3
GPT-4v
26.3
29.3 24.1
45.0
13.7
15.3
25.9
Chance
25.0
25.0
25.0
0.0
0.0
15.0
Observation space: DM text
Method
Direction estimation
Distance estimation
Map sketching
Route retracing
Shortcut discovery
Average
Human
66.7
76.5
66.7
Claude 3.5 Sonnet
29.2 8:4.4
40.2 #3.1
51.7 26.5
26.5 $2.0
20.0 =3.0
33.5
GPT-4o Mistral 123B GPTT-4v
28.7
33.3
46.7
27.5
26.6
32.6
30.5 15.1
28.9 65.7
38.3 25.3
20.3 2388
19.9 23.0
27.6
20.6
15.4 12.0
26.8
Ilama 3 70B
27.0 1/2/2
30.4 31.9
36.0 28-3
13.2 19.2
5.3 1411
22.2
Yi 1.5 34B
26.2 14.7
35.7 214
35.0 =:0.7
3,2 40.2
1.1 11.
20.2
Mixtral 8x22B
21.3
19.4
39.2
1.5
3.9
17.0
Yi 1.5 9B
10.8 61.0
20.0 63.7
35.0 $5.0
5.0 22.2
1.3 21.5
14.4
8B
23.3
0.0 10.0
11.8
14.3
Mixtral 8x7B
15.8 $3.0
16.1 81.4
30.0
1.1 21.6
1.1 4:.6
12.8
Chance
25.0
25.0
25.0
0.0
0.0
15.0
Table 1: Large-scale spatial cognition results. The three tables show results for different observation spaces. Results below 50% of human performance are gray. Methods are sorted based on their overall performance.
and map sketching, and less than 30% SPL on route retracing and shortcut discovery, even with allocentric presentation. With egocentric multimodal presentation (the closest counterpart to classic experimental protocols in animal cognition), the models are near chance level on all tasks.
Human performance ranges from 80% to 100% accuracy on image-based presentations of the multiple-choice QA tasks. Since perceiving large sequences of text arrays is non-trivial for humans, the performance drops 65%-80% for the text presentations.
With multimodal pre- sentations, we benchmark GPT-4o, GPT-4v, Claude 3.5 Sonnet, and number of open multimodal models. With purely textual presentations, we benchmark both open and closed models. We also list the performance of the chance baseline for calibration, as well as human performance (see the appendix for details).
Small-scale spatial cognition results. The results are shown in Table 2
Performance of some model classes (c.g., GPT-4o, GPT-4v, Claude 3.5 Sonnet) on purely textual presentations is considerably higher than multimodal presentations. The best-performing models, Claude 3.5 Sonnet and GPT-40, achieve 43.8% and 40.1 % average accuracies in the multimodal regime and 64.5% and 65.2% average accuracies with purely textual presentations. (Chance is < 25%.) We attribute this in part to the simplified nature of the text-only implementations of tasks like MRT, MPFB, and JLO (e.g., the text-only presentation of mental rotation uses only 2D shapes and constrained 2D rotations) and in part to the relative developmental maturity of large language models (LLMs) versus multimodal models on the remaining tasks.
On tasks that evaluate visuospatial working memory (specifically SAtt, CBTT, SAdd, and CSWM), the strongest L.MS perform well. On selective attention (SAtt), GPT-4o, Claude 3.5 Sonnet, Mis- tral 123B, and GPT-4v all achieve over 95% accuracy, matching or outperforming the human per- formance on this task. On the other hand, all models perform poorly on maze completion (MCT), in both presentation modalities. (Note that the models operate with full visibility, as illustrated in Figure 2 With multimodal presentation, all evaluated models are near chance on perspective tak- ing (PTT)) and the Minnesota Paper Form Board test (MPFB). On mental rotation (MRT), the best models are near chance with multimodal presentation, which uses 3D shapes, and only marginally better with purely textual presentation, which uses 2D arrays and constrained rotations.

---

Published as a conference paper at ICLR 2025
Multimodal
Method
MRT
PTT
WLT
MPFB
1LO
SAB
MCT
CBTT
SAdd
CSWM
Average
Human
78.5
100.0
98.0
Clande 3.5 Sonnet
20.9 23.6
21.8 22.9
37.0 84.6
35.5 87.0
40.5 6:3.5
90.5 23.5
2,2 21.8
56.5 23.
48.0 16.3
76.7 22.3
43.8
GPT-4o
33.3 =3.9
26.5 23.6
59.0 =30,
21.0 22.
26.5 03.0
70.2 21.
10.4 =10
68.0=220
40.5 874
40.0 +0.0
40.1
GPT-4y
02.3
28.0
35.0 LTT
22.5
26.5
50.8
0.7.1
44.5
32.0
26.7 13:4
30.8
Pixtral 12B
0331
43.0 67.0
30.5 85.
653
00M
39.5 23.0
28.5 16.1
OOM
25.4"
P.iii.5-vision
24.1 23.0
27.0 m5.2
22.5 67.
26.0 50.0
21.0 54.1
11.0 84.
00M
33.0 =4.0
22.0 404
OOM
22.0"
Llava interleave 7B
25.1 23.
25.8
25.0 L8.5
25.0 133
24.0
32.0 1.0
00M
25.5 15.7
27.0 1401
0OM
20.9"
Chance
25.0
25.0
25.0
25.0
25.0
25.0
0.0
25.0
25.0
33.8 15.4
23,4
Text-only
Method
MRT
PTT
MPFB
ILO
SAn
MCT
CBTT
SAdd
CSWM
Average
Human
90.0
75.0
92.0
98.0
96.0
100.0
98.0
GPT-4o
11.9 #6.
55.5 +3.0
50.5 #0.6
66.5 #4.8
98.8 #0.4
21.5 +3.8
82.5 =1.7
93.5 =0.6
76.7 =2.5
65.2
Clande 3.5 Sonnet
37-5
50.0
45.0
70.5
97.0
21.0
10.0
97.5
91.5 14.3
82.0 23.3
64.5
Mistral 123B
39.4 1.:22
44.8 1.40
48.5 15.2
57.0 15.4
97.5
10.5
14.8 12..
88.5 10.9
92.5 10.3
62.0 12.
60.5
GPT-4y
41.2
67.5
34.0
62.0
95.8
11.3
3.7
87.5 13.5
79.0
45.3 29.5
57.3
Llama 70B
28.1 19922
29.2 12.4
38.5 101.8
42.5 10.9
71.8 L3S
1.5 13300
52.5 15.7
62.5 15.4
34.0 L5.9
40.0
Mixtral 8x22B
26.99
24.3
31.0
36.0
73.5
1.5
55.0 +3.3
68.0
17.3 =2.
37.0
Yi 34B
20.6 +6.0
28.0 +2.1
31.5 =4.0
33.5 #8.6
58.2 =4.8
0.7 #1.0
35.5 #8.4
41.5 =0.0
21.0 =0.0
30.7
Yi 1.5 9B
21.2
23.8 32.7
30.0
24.5 65.5
48.2 =4.0
0.7 8:10
36.5 2.4.6
51.5 28.9
24.7 28.4
29.0
Llama 8B
14/1+11
25.8 +5.1
26.0 +4.2
27.0 #1.7
46.0
=8.7
0.0 #000
27.5 #7.1
30.0 #7.3
26.0 =0.5
21.7
Mixtral 8x7B
19.4.44
10.5 40.0
20.5
65.7
27.5 67.5
39.0
=5.4
0.0 80000
22.5 2..8
43.5 6..3
22.7 26.1
23.8
Chance
25.0
25.0
25.0
25.0
25.0
0.0
25.0
25.0
33.0 =5.3
23.1
Table 2: Small-scale spatial cognition results. The two tables show results for multimodal and text-only presentations, respectively. Results below 50% of human performance are gray, results above 90% of human performance are bold. Mcthods are sorted based on their average performance. ("Some multimodal models out of memory on MCT and CSWM tasks; their accuracy is taken to be 0 for calculating the average.)
Humans perform well, achieving over 80% accuracy the majority of the multiple-choice QA tasks with both text-only and multimodal presentations. Humans perform better on the textual presenta- tions of tasks like MRT, MPFB and JL.O than their vision counterparts due the simplified nature of the text-only implementations.
Ecological compatibility of SPACE with frontier models. Our results indicate that current frontier models lack spatial cognition. Alternatively, these results could be the result of models not under- standing the inputs presented to them (i.e., the inputs are not ecological compatibility). We study this in Appendix A and demonstrate that this is not the case. Models can understand the inputs correctly and perform non-spatial cogmition tasks well, yet fail to demonstrate spatial cognition.
DISCUSSION
We presented SPACE, a benchmark for spatial cognition in frontier models. Our evaluation of con- temporary models brings up intriguing questions and opportunities for further investigation. First, our results underscore that frontier models exhibit fundamentally different form of intelligence from what has been observed (and studied) in humans and animals. No biological intelligence we have encountered has exhibited such advanced skill in some aspects of higher cognition (Trinh) et al., 2024) while failing so profoundly in basic spatial cognition. This is particularly intriguing be- cause in biological intelligence, spatial cognition is considered a prerequisite for higher cognition, and breakdowns in spatial cognition are diagnostic of higher-level disorders (Cappa 2008 Possin 2010 Verghese et af.) 2017) Cammiisuli et al.. 2024). From a scientific standpoint, the constellation of traits exhibited by frontier models is fascinating and may inspire a new cognitive science (Simon) 2019). As a precautionary stance, we can refrain from drawing analogies based on experience with biological cognition. (E.g., "a model won the Mathematics Olympiad therefore possesses a com- parable cognitive repertoire to human Olympiad winner and could be expected to have comparable skill in other domains".)
Could deficiencies in spatial cognition be causally linked to some of the puzzling breakdowns ex- hibited by contemporary frontier models in higher-level tasks? What is the roadmap for bringing spatial cognition in frontier models up to the level of animal cognition (and perhaps beyond)? Is this a prerequisite for attaining some of the more far-reaching aspirations of contemporary artificial intelligence research? Does embodiment play role, as it has in prior forms of intelligence (Smith) & Gasser 2005 Savva et al. 2019) Or will artificial cognition continue to develop along a fun- damentally different ontogenetic path? We expect further advances increase the robustness and generality of frontier models, and to continue to broaden our understanding of the nature of intelli- gence.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, et al. Phi-3 technical report: A highly capable language model locally your phone. arXiv:2404. 14219, 2024.
Gary L Allen, Kathleen C Kirasic, Shannon H Dobson, Richard G Long, and Sharon Beck. Predict- ing environmental learning from spatial abilities: An indirect route. Intelligence, 22(3), 1996.
David I Anderson, Joseph J Campos, David C Witherington, Audun Dahl, Monica Rivera, Minx- uan He, Ichiro Uchiyama, and Marianne Barbu-Roth. The role of locomotion in psychological development. Frontiers in Psychology, 4, 2013.
Peter Anderson, Angel X. Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir R. Zamir. On evaluation of embodied navigation agents. arXiv:180.006757, 2018.
Anthropic. Introducing Claude 3.5 Sonnet, 2024. Glaude-3-55 3-uude-355-ae sonnett
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In ICCV, 2015.
Leopold Aschenbrenner. Situational awareness: The decade ahead, 2024. https:
www.eoooonwaaaaaaeeaaeee wwwreness..e.................... ai/wp wot.tottn//ppolccca//222//////// //situational
si situationalavarenesss.
Anas Awadalla, Le Xue, Oscar Lo, Manli Shu, Hannah Lee, Etash Kumar Guha, Matt Jordan, Sheng Shen, Mohamed Awadalla, Silvio Savarese, Caiming Xiong, Ran Xu, Yejin Choi, and Ludwig Schmidt. MINT-IT: Scaling open-source multimodal data by 10x: A multimodal dataset with one trillion tokens. arXiv:2406. /271, 2024.
Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin Chadwick, Thomas Degris, Joseph Modayil, et al. Vector-based navigation using grid-like representations in artificial agents. Nature, 557(7705), 2018.
Arthur Lester Benton. Contributions to neuropsychological assessment: A clinical manual. Oxford University Press, USA, 1994.
Mark Blades and Christopher Spencer. The development of children's ability to use spatial repre- sentations. Advances in child development and behavior, 25, 1994.
Nicole Blaser, G Omo,'Omo, Dell' Ariccia, David Paul Wolfer, and H-P Lipp. Testing cognitive navigation in unknown territories: homing pigeons choose different targets. Journal of Experi- mental Biology, 216(16), 2013.
Tyler Bonnen, Stephanie Fu, Yutong Bai, Thomas O'Connell, Yoni Friedman, Nancy Kanwisher, Josh Tenenbaum, and Alexei Efros. Evaluating multiview object consistency in humans and image models. In NeurIPS, 2025.
Youcef Bouchekioua, Aaron P Blaisdell, Yutaka Kosaki, Iuu Tsutsui-Kimura Paul Craddock, Masaru Mimura, and Shigeru Watanabe. Spatial inference without cognitive map: the role of higher-order path integration. Biological Reviews, 96(1), 2021.
Brickenkamp and E Zillmer. Test d2: concentration-ennuuaae test. Gottingen Ger. CJ Hogrefe, 1998.
Rodney
Brooks. intelligence,
Rodney
brooks'
three
laws of artifi-
cial
2024.
com/
brodey-brooks-thhee leiisellwwwooo
11

---

Published as a conference paper at ICLR 2025
Davide Maria Cammisuli, Gloria Marchesi, Virginia Bellocchio, Edoardo Nicolò Aiello, Barbara Poletti, Federico Verde, Vincenzo Silani, Nicola Ticozzi, Stefano Zago, Teresa Difonzo, et al. Behavioral disorders of spatial cognition in patients with mild cognitive impairment due to alzheimer's disease (the bdsc-mei project): Ecological validity of the corsi learning suvra-span test. Journal of Personalized Medicine, 14(5), 2024.
SF Cappa. Cognitive neurology: a clinical textbook. Oxford University Press, 2008
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In CVPR, 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. 1..i:2311.117995 2023.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision- language models? arXiv:24033 arXiv:2403.20330, 2024b.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained code. arXiv:2107.03374, 2021.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv:1504.00325 2015.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language model. arXiv:2406.015844 2024.
François Chollet. On the measure of intelligence. 1xXiv:1911.01547, 2019.
Michiel HG Claessen, Ineke JM Van Der Ham, and Martine JE Van Zandvoort. Computerization of the standard corsi block-tapping task affects its underlying cognitive concepts: pilot study. Applied Neuropsychology: Adult, 22(3), 2015.
Philip Michael Corsi. Human memory and the medial temporal region of the brain. Phd thesis, McGill University, 1972. https: hEtps://ecchoaarshn www.wmcll......................m com.ee//teeeeemeeete/ee/sessss
05741s554]
Christopher J. Cueva and Xue-Xin Wei. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. In ICLR, 2018
Edwin S Dalmaijer, Stefan Van der Stigchel, Tanja CW Nijboer, Tim HW Cornelissen, and Masud Husain. Cancellationtools: All-in-one software for administration and analysis cancellation tasks. Behavior Research Methods, 47, 2015.
Hawson-Haggerry et al. trimesh, 2019.
https::/trimmet org/
Miguel de Guinea, Alejandro Estrada, K Anne-Isola Nekaris, and Sarie Van Belle. Cognitive maps in the wild: revealing the use of metric information black howler monkey route navigation. Journal of Experimental Biology, 224(15), 2021.
Sergio Della Sala, Marcella Laiacona, Hans Spinnler, and Chiara Ubezio. A cancellation test: its reliability in assessing attentional deficits Alzheimer's disease. Psychological Medicine, 22(4), 1992.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. 2:Xi::2407.21783 2024.
12

---

Published as a conference paper at ICLR 2025
Pul Ashby Foltz. Adult performance piaget's water level task and its rela- tion of spatial orientation and visualization. Master's thesis, University of Rich- egi?
mond, 1978. kttpa://scholarsbii ric.rrriihhhhho......oo.oo wtt.:///....i/ecc///////ooeeeeee weaeceee112accoossttttteeoeee
theses
Nigel Foreman, Denny Foreman, Alison Cummings, and Sandra Owens. Locomotion, active choice, and spatial memory in children. The Journal of General Psychology, 117(2), 1990.
John W French, Ruth B Ekstrom, and Leighton A Price. Manual for kit of reference tests for cognitive factors. 1963.
Andrea Frick and Wenke Möhring. A matter of balance: Motor control is related to children's 's spatial and proportional reasoning skills. Frontiers in Psychology, 6, 2016.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv:2306. 13394, 2023.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi- modal Ilms in video analysis. arXiv:2405.21075. 2024a.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng. Haoyu Wang. Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can but not perceive. arXiv:2404. 12390, 2024b.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, al. Datacomp: In search of the next generation of multimodal datasets. In NeurlPS, 2023.
Maya Geva-Sagiv, Liora Las, Yossi Yovel, and Nachum Ulanovsky. Spatial cognition in bats and rats: from sensory acquisition to multiscale maps and navigation. Nature Reviews Neuroscience, 16(2), 2015.
Sabine Gillner and Hanspeter A Mallot. Navigation and acquisition of spatial knowledge in virtual maze. Journal of Cognitive Neuroscience, 10(4), 1998.
Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual ques- tion answering. International Journal of Computer Vision, 127(4), 2019.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang. Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In CVPR, 2024.
Mary Hegarty and David Waller. A dissociation between mental rotation and perspective-taking spatial abilities. Intelligence, 32(2), 2004.
Mary Hegarty, Daniel R Montello, Anthony E Richardson, Toru Ishikawa, and Kristin Lovelace. Spatial abilities at different scales: Individual differences in aptitude-test performance and spatial- layout learning. Intelligence, 34(2), 2006.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurIPS Datasets and Benchmarks, 2021b.
Anna A Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H Clark, Carina Kauf, Jennifer Hu, RT Pramod, Gabriel Grand, et al. Elements of world knowl- edge (ewok): A cognition-inspinn framework for evaluating basic world knowledge in language models. 000::24055006005 2024.
13

---

Published as a conference paper at ICLR 2025
Petra Jansen. The dissociation of small-and large-scale spatial abilities in school-age children. Per- ceptual and Motor Skills, 109(2), 2009.
Petra Jansen and Martin Heil. The relation between motor development and mental rotation ability in 5-to 6-year-old children. International Journal Developmental Science, 4(1), 2010.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot. Diego de Las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv:2401 N0W:2400.04088, 2024.
Ashley Kalina and Suzie A Walgrave. Normative evaluation a letter cancellation instrument for the assessment of sustained attention: A construct validation study. The Journal of Undergraduate Research, 2(1), 2004.
Maria Kozhevnikov and Mary Hegarty. A dissociation between object manipulation spatial ability and spatial orientation ability. Memory & Cognition, 29, 2001.
Maria Kozhevnikov, Michael A Motes, and Mary Hegarty. Spatial visualization in physics problem solving. Cognitive Science, 31(4), 2007.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In SOSP, 2023.
Emilie Lacroix, Stéphanie Cornet, Naima Deggouj, and Martin Gareth Edwards. The visuo-spatial abilities diagnosis (vsad) test: Evaluating the potential cognitive difficulties of children with vestibular impairment through new tablet-based computerized test battery. Behavior Research Methods, 53, 2021.
Barbara Landau. Spatial cognition. In Encyclopedia of the Human Brain. Elsevier, 2002.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. In NeurIPS, 2023.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interlcave: Tackling multi-image, video, and 3d in large multimodal models. arXiv:2407.07895. 2024a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024b.
Rensis Likert and WH Quasha. Minnesota Paper Form Board Test. Psychological Corporation, 1941.
Rensis Likert and William H Quasha. Revised Minnesota paper form board test. Psychological Corporation, 1969.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He. Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: your multi-modal model an all-around player? vrXiv:2307.0628/ 2023b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024.
Arjun Majumdar, Anurag Ajay. Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Movay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the of foundation models. In CVPR, 2024.
14

---

Published as a conference paper at ICLR 2025
Hanspeter A Mallot. From Geometry to Behavior: An Introduction to Spatial Cognition. MIT Press, 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: visual question answering benchmark requiring external knowledge. In CVPR, 2019.
Jérôme Marquet-Doléac, Régis Soppelsa, and Jean-Michel Albaret. Laby 5-12: Test des labyrinthes. Hogrefe, 2010.
John C Marshall and Gereon R Fink. Spatial cognition: Where we were and where we are. Neu- roimage, 14(1), 2001.
Chiara Meneghetti, Clara Zancada-Menénder Patricia Sampedro-Piquent Laudino Lopez, Mas- similiano Martinelli, Lucia Ronconi, and Barbara Rossi. Mental representations derived from navigation: The role of visuo-spatial abilities and working memory. Learning and Individual Differences, 49, 2016.
Chiara Meneghetti, Laura Miola, Enrico Toffalini, Massimiliano Pastore, and Francesca Pazzaglia. Learning from navigation, and tasks assessing its aceuracy: The role of visuospatial abilities and wayfinding inclinations. Journal of Environmental Psychology, 75, 2021.
Chiara Meneghetti, Laura Miola, Tommaso Feraco, and Veronica Muffato. Individual differences in navigation: An introductory overview. In Prime Archives in Psychology. Vide Leaf, 2nd edition, 2022.
Emil W Menzel. Chimpanzee spatial memory organization. Science, 182(4115), 1973.
Mistral AI team.
Large enough, 2024a.
satps://mistie ai n
mistral maastral-1arge-400
Mistral AI team.
Announcing pixtral 12b, 2024b.
atips://iistral.al//nes
pixtral-12b/
Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Robert Osazuwa Ness, Nehojsa Jojic, Hamid Palangi, and Jonathan Larson. Evaluating cognitive maps in large language models with cogeval: No emergent planning. In NeurlPS, 2023.
Arsenii Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The conceptare benchmark: Evaluating understanding and generalization in the ARC domain. Transactions on Machine Learning Research, 2023.
Nora S. Newcombe. Making space: The development of spatial representation and reasoning. MIT Press. 2000.
Nora S. Newcombe, Picture this: Increasing math and science learning by improving spatial think- ing. American Educator, 34(2), 2010.
Nora S. Newcombe, Spatial Cognition. In Open Encyclopedia of Cognitive Science. MIT Press, 2024.
Rahel Noser and Richard W Byme. Mental maps in chacma baboons (papio ursinus): using inter- group encounters as a natural experiment. Animal Cognition, 10, 2007.
John O' Keefe and Lynn Nadel. The hippocampus as a cognitive map. Oxford University Press, 1978.
OpenAl. GPT-4 technical report. arXiv:23033.8877..2222
OpenAl. Hello gpt-4o, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NewrlPS, 2022.
15

---

Published as a conference paper at ICLR 2025
Anja Pahor, Randy E Mester, Audrey A Carrillo, Eunice Ghil, Jason F Reimer, Susanne M Jaeggi, and Aaron R Seitz. Ucancellation: A new mobile measure of selective attention and concentration. Behavior Research Methods, 54(5), 2022.
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adrià Recasens, Larisa Markeeva, Dylan Ba- narse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, et al. Perception test: A diagnostic benchmark for multimodal video models. In NewrIPS, 2023.
HL Payne, GF Lynch, and Dmitriy Aronov. Neural representations of space in the hippocampus of a food-caching bird. Science, 373(6552), 2021.
Francesca Pazzaglia and Holly A Taylor. Perspective, instruction, and cognitive style spatial representation of virtual environment. Spatial Cognition and Computation, 7(4). 2007.
Michael Peters, Bruno Laeng, Kerry Latham, Marla Jackson, Raghad Zaiyouna, and Chris Richard- son. redrawn vandenberg and kuse mental rotations test-different versions and factors that affect performance. Brain and Cognition, 28(1). 1995.
Roger Paul Peters. Wolf-sign: Scents And Space In A Wide-ranging Predator. University of Michi- gan, 1974.
Jean Piaget, Baerbel Inhelder, F. J. Langdon, and J. L. Lunzer. The child's conception of space. British Journal of Educational Studies, 5(2), 1957.
Leila M Porter and Paul A Garber. Foraging and spatial memory in wild weddell's saddleback tamarins (saguinus fuscicollis weddelli) when moving between distant and oft-of-sight goals. International Journal of Primatology, 34, 2013.
Katherine L Possin. Visual spatial cognition in neurodegenerative disease. Neurocase, 16(6). 2010.
Andrea Presotto, Richard Hosken, Caitlin Curry, and Marguerite Madden. Spatial mapping shows that some african elephants use cognitive maps to navigate the core but not the periphery of their home ranges. Animal Cognition, 22, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In NeurlPS, 2023.
Ahad Rana. Common crawl building open web-scale crawl using hadoop, 2010. https:
7/ 7/www.a wwwdeshareeeeeeeeee net http:////ee.aaaaaaaee/////oooooa
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean- Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. 20Xi::2403.05500, 2024.
Anthony E Richardson, Daniel R Montello, and Mary Hegarty. Spatial knowledge acquisition from maps and from navigation in real and virtual environments. Memory & Cognition, 27(4), 1999.
Barbara Sahakian, Robin G Morris, John Evenden, Andrew Heald, Raymond Levy, Michael Philpot, and Trevor W Robbins. A comparative study of visuospatial memory and learning in alzheimer-type dementia and parkinson's 's disease. Brain, 111(3), 1988.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adver- sarial winograd schema challenge at scale. Commun. ACM, 64(9), 2021.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruy Batra. Habitat: A platform for embodied AI research. In ICCV, 2019.
John T. Serences and Sabine Kastner. A multi-level account of selective attention. The Oxford Handbook of Attention, 2014.
Roger Shepard and Jacqueline Metzler. Mental rotation dinenniimelii objects. Science, 171(3972), 1971.
16

---

Published as a conference paper at ICLR 2025
Herbert A. Simon. The Sciences of the Artificial. MIT Press, 3rd edition, 2019.
Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial Life, (1 2005.
Ben Sorscher, Gabriel Mel, Samuel A Ocko, Lisa M Giocomo, and Surya Ganguli. A unified theory for the computational and mechanistic origins of grid cells. Neuron, 111(1), 2023.
Robert J Spencer, Carrington R Wendell, Paul Giggey, Stephen L Seliger, Leslie Katzel, and Shari R Waldstein. Judgment of line orientation: an examination of eight short forms. Journal of Clinical and Experimental Neuropsychology. 35(2), 2013.
John Stilley, mazelib: A python api for creating and solving mazes, 2014. https://githui:
com/johnn wcweencemmazeeaai...
Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset, 2023.
https://github: http:///hhb....ooooooeeeeeorrrrr /ttpr/copppeeeeeeaaaaaaaaaaaaaae Dama-Daaa
Sivan Toledo, David Shohami, Ingo Schiffner, Emmanuel Lourie, Yotam Orchan, Yoav Bartan, and Ran Nathan. Cognitive map-based navigation in wild bats revealed by new high-throughput tracking system. Science, 369(6500), 2020.
Edward C Tolman. Cognitive maps in rats and men. Psychological Review, 55(4), 1948.
Luca Tommasi, Cinzia Chiandetti, Tommaso Pecchia, Valeria Anna Sovrano, and Giorgio Vallorti- gara. From natural geometry to spatial cognition. Neuroscience & Biobehhvioral Reviews, 36(2), 2012.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? Exploring the visual shortcomings of multimodal LLMs. In CVPR, 2024.
Trieu Trinh, Yuhuai Wu, Quoc Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 2024.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. PlanBench: An extensible benchmark for evaluating large language models planning and reasoning about change. In NewrIPS, 2024.
Steven G Vandenberg and Allan Kuse. Mental rotations, group test of three-dimensional spatial visualization. Perceptual and Motor Skills, 47(2), 1978.
Marina Vasilyeva and Stella F Lourenco. Development of spatial cognition. Wiley Interdisciplinary Reviews: Cognitive Science, 3(3), 2012.
Joe Verghese, Richard Lipton, and Emmeline Ayers. Spatial navigation and risk of cognitive impair- ment: A prospective cohort study. Alzheimer's 's & Dementia, 13(9), 2017.
David Ed Waller and Lynn Ed Nadel. Handbook of Spatial Cognition. American Psychological Association, 2013.
Lu Wang, Allan S Cohen, and Martha Carr. Spatial ability at two scales of representation: A meta- analysis. Learning and Individual Differences, 36, 2014.
David Wechsler. WMS-IV: Wechsler Memory Scale. Pearson, 2009.
Steven M Weisberg, Victor R Schinazi, Nora S Newcombe, Thomas F Shipley, and Russell A Ep- stein. Variations in cognitive maps: understanding individual differences navigation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(3), 2014.
Joseph F Welklin, Benjamin R Sonnenberg, Carrie L Branch, Virginia K Heinen, Angela M Pitera, Lauren M Benedict, Lauren E Whitenack, Eli S Bridge. and Vladimir V Pravosudov. Spatial cognitive ability is associated with longevity in food-caching chickadees. Science, 385(6713), 2024.
17

---

Published as a conference paper at ICLR 2025
Erik Wijmans, Manolis Savva, Irfan Essa, Stefan Lee, Ari S. Morcos, and Dhruv Batra. Emergence of maps in the memories of blind navigation agents. In ICLR, 2023.
Michele Andrisin Wittig and Mary J Allen. Measurement of adult performance on piaget's water horizontality task. Intelligence, 8(4), 1984.
Déverton Plácido Xavier, Filipa Abreu, Antonio Souto, and Nicola Schiel. Choosing the best way: how wild common marmosets travel to efficiently exploit resources. Animal Cognition, 27(1), 2024.
Jiayun Xu. Mauricio Girardi-Schappo. Jean-Claude Beique, André Longtin, and Leonard Maler. Shortcutting from self-motion signals reveals cognitive map in mice. Elife, 13, 2024.
Yutaro Yamada, Yihan Bao, Andrew Kyle Lampinen, Jungo Kasai, and lIker Yildirim. Evaluating spatial understanding of large language models. Transactions on Machine Learning Research, 2024.
Kaining Ying. Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang. Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask AGI. In ICMI., 2024.
Eunice Yiu, Maan Qraitem, Charlie Wong, Anisa Noor Majhi, Yutong Bai, Shiry Ginosar, Alison Gopnik, and Kate Saenko. Kiva: Kid-inspired visual analogies for testing large multimodal mod- els. arXiv:2407. 17773, 2024.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, al. Yi: Open foundation models by 01.ai. arXiv:2403.0m 04652, 2024.
Christopher J Young, Susan C Levine, and Kelly S Mix. The connection between spatial and math- ematical ability across development. Frontiers in Psychology, 9, 2018.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICML, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In CVPR, 2024.
18

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ECOLOGICAL COMPATIBILITY OF MULTIMODAL INPUTS WITH FRONTIER MODELS
Our results in Section suggest that ttate-of-the of-the-art frontier models fail in the spatial cognition tasks presented in SPACE. These failures can be attributed to the lack of spatial cognition these models. Alternatively, these failures could be due to models not comprehending the inputs presented to them (i.e., the inputs are not ecologically computible with the models). To rule out this alternative possibility, we design additional tests unrelated to spatial cognition used in our benchmark. If models succeed on these tests, we can infer that the inputs are ecologically compatible since the models understand and perform tasks using these inputs. In each test, pose a series of multiple-choice questions evaluating a model's fine-grained understanding of the inputs. We now describe these additional tests.
the same vision text inputs
Test 1: Given discrete map image text inputs (see Figure 3), answer the following questions:
Q1. What is the size of the grid (H W)?
Q2. What is your current (x, y) location?
Q3. What are the (x. y) locations of all navigable cells? Include cells containing landmarks and your current position.
Q4. What are the (x, y) locations of all obstacle cells?
Q5. What are the landmarks visible in the image / array?
Q6. What are the locations of the landmarks visible in the image / array?
Test 2: Given ego image (see Figure 3) answer the following questions:
QI. What is the name of the landmark visible in the image?
Q2. Is the landmark <name>> in the left half of the image?
Q3. Is the landmark <name>> in the right half of the image?
Q4. Is the landmark <name>> in the central section of the image?
Test 3: Given two consecutive ego images from a walkthrough (see Figure 4 answer the fol-
lowing question:
QI. What is the action taken to go from image I to image 2 (move forward, turn left, turn right, wait/do nothing)?
Test 4: Given a perspective taking image / text array (see Figures and 10), answer the follow- ing questions:
Q1. How many objects/ non-zero locations are present the image array?
Q2. What objects / non-zero locations are present in the image array?
Q3. <object / location>> to the left of <object/ / location>> in the image array?
Q4. Is <object / location>> to the above <object / location in the image / array?
Test 5: Given water level test images (see Figure 11 answer the following questions:
Q1. Is there water in the water container?
Q2. From image to image 2, is the water container rotated to the left, right or not rotated at all?
Q3. From image to image what the absolute rotation angle of the water container (in degrees)?
Test 6: Given a grid of icons / characters from selective attention (see Figures 16 and 17 answer the following questions:
Q1. How many total objects / characters are present in the image / grid (including repetitions)?
Q2. What is the size of the grid of objects/ characters (width height)?
Q3. How many unique objects / characters are present in the grid (ignore repetitions)?
Results discussion: We evaluate GPT-4o and GPT-4v these tests. The results are shown in Tables 3 and 4 Both models largely understand DM image and text inputs (test I). However, they fall short in calculating the grid size for DM images (Q1). GPT-4o understands egocentric images,
19

---

Published as a conference paper at ICLR 2025
Multimodal evaluation
Test 1
Test 2
Test 3
Model
Q1
Q2
Q3
Q4
Q5
Q6
Avg.
Q1
Q2
Q3
Q4
Avg.
QI
GPT-4o
30.4
78.2
89.4
87.8
100.0
93.6
79.9
100.0
84.0
95.5
83.5
92.6
59.3
GPT-4v
55.8
86.2
89.8
91.2
99.8
79.2
83.6
98.0
45.0
36.5
56.5
66.8
48.0
Text-only evaluation
Test Q4
Test
Test
Model
Q5
Avg.
Q2
Q4
Avg.
Q1
GPT-4o
100.0
100.0
77.5
85.6
100.0
82.8
90.9
GPT-4v
100.0
100.0
96.8
91.0
100.0
77.6
94.2
Table 3: Mcasuring ecological compatibility of multimodal inputs with frontier models (part 1)
Multimodal evaluation
Test
Test
Test
Model
Q1
Q2
Q4
Avg.
Q1
Q2
Q3
Avg.
Q1
Q2
Q3
Avg.
GPT-40
99.6
99.6
89.8
87.7
92.4
100.0
73.9
38.7
64.0
83.0
90.5
58.2
77.2
GPT-4v
78.3
87.4
78.4
76.0
79.0
100.0
56.3
32.4
55.4
74.5
88.0
35.8
66.0
Text-only evaluation
Test
Test
Test
Model
Q1
Q2
Q3
Q4
Avg.
Q1
Q2
Q3
Avg.
Q2
Q3
Avg.
GPT-4o
97.6
99.6
99.5
94.2
97.4
100.0
100.0
99.5
99.8
GPT-4v
96.8
98.1
92.2
76.8
90.8
100.0
94.8
98.1
Table 4: Measuring ecological compatibility of multimodal inputs with frontier models (part 2)
ie., recognizes and localizes landmarks in egocentric images (test 2). GPT-4v recognizes landmarks well (Q1), but performs poorly in localization (Q2, Q3 and Q4). Both GPT-4o and GPT-4v perform poorly on action estimation (test 3) and estimation water container rotations (Q2 and Q3 in test 5). GPT-4o excels in understanding the perspective taking inputs with multimodal and text-only presentations (test 4). GPT-4v also performs well on test 4, but is worse with multimodal inputs when compared to text-only inputs. Finally, both GPT-4o and GPT-4v perform adequately with counting objects (QI in test 6) and grid sizes (Q2 in test 6) on selective attention task inputs with multimodal inputs. However, they struggle calculate the number of unique objects / characters (Q3 in test 6). Both GPT-4o and GPT-4v excel at the text-only presentation of test 6.
Our results indicate that srate-of-the-at models can understand multimodal and text-only inputs provided in our benchmark. They perform well in most of the tests, but have specific shortcomings (e.g., localizing landmarks in ego images for GPT-4v, understanding rotations of water containers and counting unique characters / objects in grid). Importantly, the average results on each test much higher than the SPACE task counterparts. For example, even though GPT-4o and GPT-4v understand DM text inputs nearly perfectly (test 1), they perform poorly in the DM text versions of the large-scale spatial cognition tasks (see TT) Similarly, even though GPT-4o understands the perspective taking inputs nearly perfectly for both text-only and multimodal presentations, it performs poorly on the perspective taking task in SPACE (see Table 2). Therefore, the failure of frontier models SPACE most likely due to their lack of spatial cognition, and not because they cannot understand the inputs presented to them.
A.2 SMALL-SCALE SPATIAL COGNITION: ADDITIONAL. DETAILS
We described the small-scale spatial cognition tasks from our benchmark in Section 3.2 Here, provide additional details about the historical context and motivations behind these tasks
Mental rotation test (MRT). This was introduced by Vandenberg & Kuse (1978) as test of spatial visualization. The original MRT contained 20 items, where each item consisted of a criterion figure, two correct alternatives, and two distractors (Vandenberg & Kuse 1978). The criterion figure is perspective rendering of a 3D criterion shape from Shepard & Metzler (197I). The correct alterna- tives are rotated versions of the criterion shape, where the rotation applied in the 2D image space
20

---

Published as a conference paper at ICLR 2025
on the criterion figure, along the vertical axis in 3D for the criterion shape. The distractors rotated mirror-images of the criterion shape or renderings of other criterion shapes. The goal was to identify the two correct alternatives from the four choices. We implement a version of MRT with one correct choice and three distractors, and incorporate rotations along multiple axes (Peters et al.) 1995).
Perspective taking test (PTT). This was introduced by Kozhevnikov & Hegarty (2001) as a test of spatial orientation. An arrangement of objects is shown on a piece of paper. A test participant is asked to take the perspective of standing next to an object (say, object A) facing another (say. object B), and is required to point to a third object (say, object C). This task has been used extensively in subsequent literature (Hegarty & Waller [004] Weisberg et al. [2014] Meneghetti et al. (2022. We implement this task by randomly sampling N icons of objects like cars, carrots, chairs, and grapes and place them at random locations in an image (with no overlap between objects). We then randomly sample three of the N objects as A, B, and C.
Water level test (WLT). This was introduced by Piaget et al. (1957) as a test of visuospatial per- ception. Originally, the test was designed to evaluate children's 's knowledge about the horizontal nature of the surface of water in sealed bottle regardless of its orientation. Children were presented with bottles partially filled with colored water and asked to imagine the position of the water if it were tilted. Children had to gesture, draw, or use cardboard cutouts to answer the question (Piaget jet al. 1957 Foltz 1978) Wittig & Allen 1984). Performance on the water-level test was found to be related to performance on spatial abillity tests (Foltz 1978 Wittig & Allen 1984).
Judgement of Line Orientation test (JLO). This was introduced by (Benton 1994) as a measure of visuospatial perception. The original implementation contained 30 samples presented in a flip-book style, where two lines are shown at the top of each page. The goal is to determine the angles between the two lines by comparing them to an array of reference lines (i.e., pick two reference lines that have same angle between them as the lines at the top). There have been multiple variations of JLO with subsets of the 30 questions for faster evaluation (Spencer et al.) 2013). We recreate the JLO test suite by randomly sampling pairs of lines on a 2D plane with an angle between O to 180 degrees (in multiples of 18 degrees) and formulate it as multiple-choic QA.
Selective attention task (SAtt). This is designed to evaluate selective spatial attention (Serences & Kastner 2014 Pahor et al. 2022). In particular, we use the widely used cancellation task, where the goal is to search for and mark out target stimuli embedded amidst distractors (Della Sala et al. 1992.
Brickenkamp & Zillmer 1998 Dalmaijer et al., 2 et al.) 2021 Pahor et 20.2 Kalina
& Walgrave 2004)). The stimuli may be characters (Brickenkamp & Zillmer 1998 Dalmaijer al.
2015
Pahor et al. 2022 Della Sala et al. 1992 Kalina & Walgrave 2004). pictures (Lacroix et al.,
2021
Pahor et al. 2022). or icons (Lacroix et al. (021). We implement this task with objects the
stimuli for visual evaluation and characters as stimuli for textual evaluation.
Maze completion task (MCT), This task designed to evaluate spatial orientation, planning, and executive functioning (Lacroix et al., (002). It was used as a neuropsychological test to assess executive function disorders in children (Marquet-Doléac al. 2010).
Corsi block-tapping task (CBTT). This is designed to assess visuospatial working memory and attention in healthy participants and patients with known or suspected brain damage (Corsil 1972] Claessen et al., 2013). An examiner demonstrates a sequence of block-tapping movements on a board containing fixed blocks placed in pseudo-random positions. Participants are required to re- produce the same sequence (forward condition) or the inverted sequence (backward condition) of block-tapping movements to succeed. We evaluate frontier models on the forward condition since prior work has not found significant differences between task performance in the forward and back- ward conditions (Claessen et al. 2015).
Spatial addition task (SAdd). This was introduced in the fourth edition of the Wechsler Memory Scale, a suite of neuropsychological tests to evaluate memory function in individuals aged 16 to 90 (Wechsler 2009). SAdd evaluates visuospatial storage and manipulation in working memory. A test participant is shown a grid with blue and red dots for five seconds. The participant is asked to remember the location of the blue dots and ignore the red dots. The participant is then shown another such grid. The objective is to add the two grids together by following certain rules. If a grid location has blue dot in exactly one of grids, the result should be blue. If grid location has blue dots on both grids, the result should be white.
21

---

Published as a conference paper at ICLR 2025
Cambridge spatial working memory test (CSWM), This was designed evaluate spatial working memory in human subjects (Sahakian et al. 1988) Multiple colored boxes are shown on a screen. A yellow 'treasure" is initially hidden in of the boxes. The participant must select boxes one at time to open them and search for the treasure. Once the treasure is found, another treasure is placed in one of the remaining boxes. The intention is for the participant to locate all the yellow treasures via a process of climination.
A.3 SPACE EXAMPLES
We illustrate examples for each task from our proposed SPACE benchmark. Large-scale spatial cognition
Egocentric image observations:
Figures
DM image observation
P
Figures
Small-scale spatial cognition
MRT: Figures and
PTT: Figures and
10
WL.T: Figure
MPFB: Figures
and
13
JLO: Figures
14
and
15
MCT: Figures
22
and
CBTT: Figures
18
and
19
SAdd: Figures
20
and
21
CSWM: Figures
24 and
25
A.4 IMPLEMENTATION DETAILS
We provide additional implementation details about our experimental setup in this section.
3D environment generation: We create ten environment layouts based on prior work in cognitive
science and artificial intelligence (Tolman 1948 Gillner Mallot 1998: Richardson et al., 1999]
Banino et al., 2018 [Bouchekioaa et al. 2021). Figure 3 shows bird's s-eye view images of each layout. We populate each environment with visual landmarks in the form of paintings hanging on walls, where the painting frames are 3D meshes and the paintings are images from ImageNet (Deng cct al. (209)). To create 3D environment for given layout, we first randomly sample textures for walls, floors, and ceilings from a database of textures create the base 3D mesh. Next, randomly assign ImageNet images and 3D frame meshes to predefined landmark locations in the environment. We create the 3D environment using the Trimesh library and export it in gITF format (Dawson- Haggerty et al.] 2019). We simulate the environment using the Habitat simulator (Savva et al. 2019). We create 3 environments per layout, for total of 30 environments in our benchmark.
For multiple-choice QA, we randomize the placement of the correct answer among the four choices such that it appears each of the four positions once, yield- ing four trials per question. For each trial, we evaluate the performance over all questions to obtain the average accuracy. For interactive tasks like route retracing, shortcut discovery, MCT and CSWM, we evaluate each model in three independent trials and obtain the corresponding metrics. By per- forming multiple trials, we can compute means and standard deviations for each model on each task across the trials.
Randomized trails for evaluation:
Human performance: We obtain human performance on SPACE tasks by evaluating 29 participants (aged 20 - 50). We evenly divide the questions from our benchmark across all participants. For each participant, we provide HTML files containing a subset of questions from each SPACE task and the corresponding choices. The HTML files contain formatted versions of the prompts used to evaluate
DD text ohservations obtained by simply converting the DM image text illustrated Figure
22

---

Published as a conference paper at ICLR 2025
frontier models. We do not provide any additional instructions background information about how to solve the tasks. For efficiency, we group all questions corresponding to a single environment in the large-scale spatial cognition tasks. Each participant is assigned to view video walkthrough from one environment and asked to answer a series of questions about that same environment. This is in line with classical protocols in human cognition (Allen al., 1996 Heearryy et al.) 2006) Pazzaglia & Taylor 2007; Weisberg et al., 20144] Meneghetti et al. | 2021). We further provide CSV file where the participant instructed enter the answers. We instruct the participants to perform all tasks mentally without any aids like pen and paper. Each participant is estimated to have taken 60 to 90 minutes to answer all the questions. The participants send us their responses and we evaluate them collectively. We denote the collective performance of all participants as the human performance Tables andd
Note that we establish the human baseline only for the multiple-choice QA tasks since it was straight- forward to share the test materials with the participants online and obtain their answers. The inter- active tasks would require us to meet participants in person to perform evaluations and were not equipped to do this.
For most of our experiments, we use square images. We provide the images to models as is without preprocessing. For most models (especially closed-souree ones), the pro- cessing of the image beyond the imput stage is outside control. We rely on the model creators to correctly process the images. The exact image resolution and aspect ratios are task-dependent and listed in Table For egocentric video inputs in the large-scale spatial cognition tasks, the number of frames varies from 61 to 240. Since GPT-4o, GPT-4v and Claude 3.5 Sonnet APIs did not permit 240+ frames as inputs, we subsample the video frames by a factor of 2 before providing them the model. For DM video inputs, the number of frames varies from 13 to 72. We provide them as is to the model.
Image preprocessing:
Task
Image resolutions (W x H)
Large-scale spatial cognition (Ego images)
512 x 512
Large-scule spatial cognition (DM images)
512 x 512
Mental rotation (MRT)
Varies from 595 x 541 1133 1432 since we crop the
redundant white space around images.
Perspective taking (PTT)
6410 x 480
Water level (WLT)
Varies from 239 488 to 787 631 since crop the redundant white space around images.
Minnesota paper form board (MPFB)
480 x 480 for choice images (i.e., puzzle pieces put together).
Varies from 831 x 578 1211 x 740 for the image containing the puzzle pieces.
Judgement of line orientation (JLO)
512 512 for the input image, 1656 x 910 for the legend
Selective attention (SAtt)
512 512 for 3 x 3 grids, 768 x 768 for 4 x grids, and 1024 x 1021 for 5 x 5 and x 6 grids
Maze completion (MCT)
1100 x 1100 for 11 x 11 mazes, 2300 x 2300 for 23 x 23 mazes, and 3100 x 3100 for 31 x 31 mazes
Corsi block-tapping (CBTT)
1024 x 1024
Spatial addition (SAdd)
300 x 300 for 3 x 3 grids, 500 x 500 for 5 x 5 grids, 700 x 700 for 7 x grids, and 900 x 900 for x 9 grids
Cambridge spatial working memory (CSWM)
1024 x 1024
Table 5: Image resolutions and aspect ratios for images and videos in SPACE.
We evaluate frontier models on each of the SPACE tasks using zero-shot prompting. For each task, we design a prompt that provides detailed de- scription of the task and the expected response format. Below, we provide the prompt templates for each of the SPACE tasks. While the prompts have been formatted for visual display in ETEX, the content remains the same. We have replaced images and arrays (in some cases) with placcholders for brevity.
Prompting frontier models for SPACE tasks:
23

---

Published as a conference paper at ICLR 2025
Video walkthrough
Direction estimation
Route retracing
Q: hictoal tla yourre danding feing the Gamaa sa chown in this srage At wlat dreccion is degrces) stonge
You shoom vidio walletoaggh dossrnitaiigg the skistol pulk to goal trom mont lecaiie. You are pliced at sran locanon. Merace pash goal
Distance estimation
Pretaal oboues What Eafidaan disaness (is
H
Gular, Cuski, Bell and Acripland
and cheene
40,43
Mi K1.43
Map sketching
You are sheom video wallhhough denitteratigg explomaory pars through
discovery
fnlliinugghe The with
cholore doacher
deoors You ploco dam
stan
locatie. Find karricu scul.
Fist
Exgarr
Glatt
Salt
Faleo walld enghl
No
Curtur
Stant
Cuete
Coale
Guitz
Figure 4: Large-scale spatial cognition with ego image observations
Large-scale spatial cognition
Direction estimation: Prompts
Distance estimation: Prompts and
Map sketching: Prompts
5
and 6
Route retracing: Prompts
10
and
13
Shortcut discovery: Prompts
15
16
and
Small-scale spatial cognition
Mental rotation test: Prompts 18 and 19
Perspective taking test: Prompts 20 and 21
Water level test: Prompt 22
Minnesota Paper Form Board test: Prompts 23 and 24
Judgement Line Orientation test: Prompts 25 and 26
Selective atfention task: Prompts
28
Maze completion task: Prompts
29
and
30
Corsi block-tapping task: Prompts
32
and
31
Spatial addition task: Prompts
33
and
34
Cambridge spatial working memory test: Prompts
35 and 36
24

---

Published as a conference paper at ICLR 2025
walkthrowch
M
estimation
Youre vilto willhwegg
skzea the kill Wint theungh sanklin slggeo) beveau de line oxmentigg your besiiin and
de neequting the derrn part no
can becaoo. You plood
Chekee Al T2 To 1 Bj
Distance estimation
Precsd stnning kndrand
Whut nne he Caddiciaanee
ditnnco to
Al 7. Dal
122,12,31
112, 12
Map sketching
an dooon vidao wallhhnugh doncauezting rejacrery pat oftta
discovery
lasiraaaa You dheloes ofong skeadee Pick bochol optios
from autart
lecaee The mes may long with
le alaty
eenocwary You
Vilee waceooecceeggee pete
Figure 5: Large-scale spatial cognition with DM image observations. Please note that the top-down visu- alization the left needs to be rotated 902 clockwise get the DM images.
Freert casting fnderot alaemus iac What snale dapaseo satooaill lis ccnsectig your
Dircction esti catimation
You video sulchhrugh denrurtting the duner poal fios van buuion You ax and
Route retracing
Claiass Al 191 Bi-14) C:d D1-171
you slaniing Zeciddor N. E.V.O and L.
11..
CI
11435,45 100
ES 24,54 143335
Vale WORL
N
Youare alamau valoo vallebzruat conotatraing oxplendace (urs thneggh at aevivonsin Shevc nop of onchomeee wit bicatiee of run goi
Map sketching
You no shent villo wellibeggh
Shortcut discovery
You
wenr vore god Rees stat
dalors
beatos Fiel cherrce the
i
8
Statte
Vidio vallis Movntrat
Figure 6: Large-seale spatial cognition with DM image observations. Please note that the top-down visu- alization on the left needs to be rotated by 902 clockwise to get the DM images.
25

---

Published as a conference paper at ICLR 2025
Reference shape
Choice
Choice 2
Choice 3
Choice 4
Figure 7: Mental rotation (MRT) with visual inputs: Which choice image shows the reference shape rotated 3D?
26

---

Published as a conference paper at ICLR 2025
Reference array
Choice I
Choice 2
Choice
Choice 4
0,0, 8,1,8,0,0
0,0,8 8,1,8,8,, 0,0
7.0, 7.0.1,0 8, 4.0,0
0,0,0,3 9, 9.0.0.0
0,0.0, 9,00 0,0,0
0,0,2, 1,0,
0,0, 0,,4,,022 2,0,0
8 82.0,0,0,0,3
2.0,0.0.0.0 0.0.0.0
0,0,0,0 0,0,0.0,0,0, 0,0,,,0,,70
0,8,7, 0,0,4
6.0.7.0.7.0
8.0.2 00..2..7,17
3,5 8 2.5.0.97,2.5 9,7,2,8
8,2,7,9 0,
08.9,5.000.0
0.0.0.5,9.0.9
0.1,7,5.0.01.
0, 0.1,7,5.00 0,000,1
1,0,0.5,7,1.0
0,8.0.7,2.00.
4,0,2,7,800,8
3,5.0.9.7,22
0.0 2 0,,,,,07,,,8
8,1,7 0,0
0,0.5,1, 2,0
0.2.0 1,5.0.0
2.0.0.00 00.000.0...0
0.2 0...0.0..00 0.0.0.0...0
0,0.0.0,0 0,0
0,2,3,0 0,0,7
7,0,0.0 3,2.00
0,0.0 9.0.0.0
0,0, 1,0,7
view. none. back used, year,eaa,nnnn view,been. been, none
view, used, view been, year, mone none, back
back, none, none, used none, been, view
view, been none used, mone view. none back
view, used, view year, been back, none, none
0.0 8.8. 9,1,9.0
0.5.0. 0.0.0,0.0
0,0,3,9,2,,774
7,,,2,,,33.
9,1,9 0,0,0,000
0,0 0.00. 0,2,0,22033 0,3,0,3...00 0,000.0
8 4,0,0,000 9,4.0,0
00,00000,0.00 8,5,
5,4.0 0..000,000.00 0,0,0.0,0,0
0,0
0,0,0,0
3 8.0 8. 0.0.0.0..00,00..
7,0,0,0 2.2.0
0.0.0 9,2,1.0,0.
0.0.00 9.2.9......0. 0.0.0.000
0.0,0,0.3
9.0.9.10.0.0.00
4,0,7 0.0.9.0
0.8.0.1.000 0.0,0,00003
3 3,0,00 0.0.1.0000000 8,8
0,0,0 0,0,0,00 1,9
2,0 2. 8,8,8,8
2,0 3.0.0.0.0
9,2,1.0 3.0.0.0.0
0,0
0.0.1.2,.2
0,0,0
0.8.1.8..0000. 8.0.0........
9,0,9, 0.0,0.0,0
1.0.0.0.0.0.00
8 8.9.0 0.8.8.8808...0
0. 0.00000 0.0.000,0.0 0,0,
7 7.0.0 0..0000000 0,2,,,..22222 2.2.0
3,0,0,0 0.0.0
9,3.0.0.0.0 2,4,
4. 0.0.0.0..0000..
2,2 0,0 0,0,0,00000000000000 0,0
8.88
0,0.0.8 0,0,0
0.0.0,0,0. 0,0.0.0.0
0.0 0.0.4, 9.9.0.00.0.4...... 0.0.4.8
0.0,0.00,000 0.000.0... 0.
0,0,0,0. 9,9,1,9,00 0,8
1.0.3.0.000,000
6.0, 0.000.0000000
6,0,0,0,0 5,0,5.0
0,9,0,3,9.0
0.0.0..000.0
0,0,9, 3,3,00 0.0,0
8 0,5, 0,5,0,0 0.000.20 2,5,8
0,00 0.000.0 0,0,0,00000 0,0
0,0.3.0 0.0.0.
0.0 0.000.0.00.0
5 5.0.8.0 3.0,0
6,1,1,4,5 0.0.0
4.0,0
0,4,1,0,000 0,
0,3,1,0,0 0,0,0,1000
2,0,0,0,0 1,4,0
0,0,0,3 3.0 8,9
1,3
0,0,0,0.3,4..
3.0.0.0.9,4,
8. 8,4,3.000,0,
8,4,9.0,,00. 0,0,0,003
8 9,0,000
0,0,0,9,000 0,0
0.0.0.00.0.0
8.1.0. 0.0,0.0
1 1,0,0.0 0, 0,1,
0 0 8,
4.0.0..0.1,5
0.0.0 4,1,,,,,,1,
5,1.0 5.1.0.4.00 0.000.0
0.0 0.0
1 1, 1..00..0000
8,
8,5,2.0,0,53 5,0
0,6,0 8,1,0,0
0, 0.0,0
0.0 2,5,8
Figure 8: Mental rotation (MRT) with text inputs: Which choice arry shows the reference array rotated in 2D?
you are standing at bat and facing the book, At what clackwise angle (in degrees) the apple located relative you?
Preteud that you standing grapes and facing the donut, At what clockwise angle (in degrees) the book relative to you?
Pretend that you standing cake and facing the desk, At what clockwise angle (in degrees) the relative to you?
Cholces A)
Cholees A)
Choices B) -19
-37
.115
-77
C) 21
D) -55
D) -57
Figure 9; Perspective taking (PTT) with visual inputs.
27

---

Published as a conference paper at ICLR 2025
Pretend that you are standing at 6 and facing 9. At what clockwise angle (in degrees) is relative to you?
Pretend that you are stunding at and facing 2. At what clockwise angle (in degroes) is relative to you?
Pretend that you are standing at and facing 9. At what clockwise angle (in degrees) is relative to you?
Pretend that you are standing at 9 and facing 4. At what clockwise angle (in degrees) relative to you?
8.0.0,0,000
Choices
0,3,0.0
Choices
0,8,0, 08,0,00 0,0,0
Choices
0,9,0,5
Choices
8,8,8,8
A) -119
1.6,0,8
A)
0.3,0,0
0.0.0.0.
-59
0.7.2.0
165
0.8,0,1,000
B)
8.0.8.0
-96
3,0.8.8
0,4.8,8
7.0.0.0.0.0,0
4,7,2.6
0.00000000 0.0.000.0000
-99
-135
2.0.5.9.0.4,8
C) 76
0.0.1.8.0
-159
-75
0.3.0.9.0.0.0
D) 156
D)
0,0,0,5
0,0,0,8,0
Figure 10: Perspective taking (PTT) with text inputs. array colors are only for illustration purposes.
Original container
Rotated container
Choice
Choice
Choice
Choice
Figure 11: Water level (WLT) with vision inputs: Given a water container filled with water, predict the water level in the rotated container.
28

---

Published as a conference paper at ICLR 2025
Puzzle pieces
Choice
Choice 2
Choice 3
Choice 4
Figure 12:
Minnesota Paper Form Board (MPFB) with visual inputs: Which one of the four choices shows what would be like when the puzzle pieces put together? The puzzle picces can be rotated but not flipped.
Array pieces
Choice 1
Choice 2
Choice 3
Choice
1,11111 111.111111
1,0,0.1
1,0,8,000,0:1
1.t.1.1...11111.11111111.11..111
111.1.11..11..11111111
1.J.A.1.......I111111.1 I.A.A.1.......1......11
1,8,0.1
1.8.0.0000..1
0,0,0,0,8 1.0.0.111
0,,000000::1:::00
1:::00000:000110001
1:0:0:0.0:0:::1:00
1,8,0,1
1.9.0,0.0...1
1,0.0.0,8,,,11 8.
1,0,0,0,8,, 1.8.0.11
1...0...P........1111
1.0.0.0......11...0
1,8.0.1
1.8.0.0.800.1
111.1.1,11111111111
0,, 1.8.
111,1,1,11,11111111111
1,,,,,,,,1,,,,,,,,,,
1,8,0.1 1.0,0.0.0,0.1
1.0.0.1.0,0.00
1.8,0,0,8,81100 1.0.0.1
1.8.011..0...00000
1.0.0.0.1.8,,0.000
1,1,1,1 1,1,1,,,,,1,,1
1,0,2,0,0,0.1.1
1.0.0.1.8,0,,,00.
1.0.0.811.,0....
1.0,8.0,0,0.3
1.0.0.1..00...0.0
1,0.0,8.1.8.00 0.8.0.00.
1,1,1,1,11111
1,1,1,1
1.0.0,0..00.0....
1.0.0,0.0.0.1
1.8.0.1
1.0.0.1 1,1,1,
1.1.1.1.112.11111..111
I.1.1.1.1.1.1.1.1..
I.1.1.1.11.......1.1
I.1.I.A.............
11,1,1
11111,
1,8,8,1
1.0.8,0.0.1
1,8,0.1
1.0.0,0.011
1,8,0.1
1,0,0,0.0.1
1,0,1,000,,
1,8,0.1
1.0.0,80.11
1,0.1,000,0000000
1,1.111
1.0.0.0.0.1
1
181.0.0.0.00000
1.0.0 0.0.0.0.....
1.0 1.0.1.00.0000000
11111,1
1,1.1.1.111
18..8....00.000
1.0.1.8..0.00
1,0,0.8,1
1.1.1.1.1
1,0,0.8,1
1.0.0.0.1
1,0,0,0,1
1,0,0,0,1
1,1,1,1,1
1,1,1,1,1
111,1.1
1.0.0.1
1.0.0.0.0.0.1
1.1.1.111..111
1.1.NA111111111
111.111111111.
0.0.0.00001.000001
1.0.1.0..0.000000.
10000:00001110000111
100.0.00..1.000001
18..0.0......00
1..0...0.0.1.80000
8.8.0.8.0.8. 8.8.8
1,8.0.1
111.1.1,,,,,1111111
111.1.111111111111
6...8.0..6.188888
1,1,1,111111
1,1,1,1,11111 1,1,11
1.8.8.
1.0.0.0.0.1.00000
1.0.0.0.....11.000
1.0.8.00.0.11.000
0.0.0.0.0.0.1
111111
1,0.8.1 1.0.0.0.0.0.
1,8,8.1 1.0
1.0.0.0.0.0.1
1,1.1,1,,,,,,,1111111
1,1,1,1,1,11,3,,,11111
1,1.1.1.111,,,,,1111
1.0.0.1.00.0.0000
1,1,1,1 1,1,1,
Figure 13: Minnesota Paper Form Board (MPFB) with text imputs: Which one of the four choices shows what it would like when the array pieces put together? The pieces merge the edges (Is). They be rotated in multiples of 90 degrees but not flipped. The array colors are purely for illustration purposes.
29

---

Published as a conference paper at ICLR 2025
Legend
Input image
Choices
Input image
Choices
Lines 1 and
A) Lines 1 and
Lines and
B) Lines I and
Lines I and
C) Lines and
D) Lines and
D) Lines I and
10
Input image
Choices
Input image
Choices
Choices
A) Lines and
A) Lines
A) Lines 1 and
B) Lines and 9
Lines and
B) Lines and
C) Lines and
C) Lines
C) Lines and
D) Lines and
Lines and
D) Lines
Figure 14: Judgement of line orientation (JLO) with visual inputs: Which pair of lines from the legend
have matching angle to the lines input image?
Input array
Choice
Choice
Choice
Choice
0.b.8.D.0
U.D.B.D.
B.C.D.C.8
0.8.0
8.D.B.
8
8:0
O.D.8.D.E
0.0.0.8
B.C.8.0.8.0
0.0.6.2 8 8-6
0.0.0
8.0
1,1.1.1,1 0.0
D.C.9.0.8 1.t.B..0.0....0
s.n.a.n
D-6 0.5.0.2000
w.w.S.0......
2--0
Figure 15: Judgement of line orientation (JLO) with text inputs: Which choice has an angle between lines
and 2 that matches the angle from the input array? The array colors are purely for illustration purposes.
30

---

Published as a conference paper at ICLR 2025
Target object
Target object
Choices (1, 9.
Choices
A) (2.23)
B) (0.01 (1, (,
9. 0.
(0.0% 3..33,
Dj (1.5.. 21
(3. 3) (5.5)
(0.20 3.C.
Target object
Target object
Choices
A] (0. O, S. D. (3.
D) B. D. 0.
Choices
A) (0, 6, (1, 2), 23, 2),
(0.3)
Rj 00, (1, 7). 23, (4,40)
00, 9, 21..
Figure 16: Selective attention (SAtt) with visual inputs: What the (row, column) grid locations of the reference object? top-left element of grid (0, 0).
Target character:
Target charaeter: 5
X.l.s.2.p.D.k.nnh
n.h.I.22KKKxp.s.bb
n.p.k.s.khh.l.2..
K.b.s.K.P.LL2.h..
b.p.x.nsi.nnhh...
L.K.h.s.p..x.
L.b.
A) (0. S
D) (0.7). 82 R
(6.
(0.3). D
Choices
(.
D.
(6.66) 86. (8.6
(7. ()
3L (8.0)
(7. (8,
0.7.6.5.4
0.6.7.4.5
7.0.6.5.4
7,4.5.6.0
C (0,3). 3) (4,44
B) (0,2).) 26, (4.
A) 5) (1.2),
D) (0.3). (2.2)
Choices
X.b.n.L...h.K...s
b,x.n.ke,n,L,b,2..
Target character:
Choices
9.3.5,2.4.7,6
Target character: k
5.4.2,9,7.6.3
A) (0.3) 32 3L 61 6.666,666.166
c,2,k,d.1
1,d,7,0kk
1,0,k,d,2
d,c.L,2,k
(0.2). 4, (4.
(0, 4y (3, (4.44)
Choices
(4
5.7.6.9,2.4.3
3.6.2.4.6.9.7
2,6.9.4.7.5..
3.4.7.2.5.6.5
2.6.7.42..9.5
B) (0, (1, 31 (2. 3L
(6.3) D) (0.0) (1. 2, (3, (4,
(0.0),
(4.5% (8.28 (6.5) 31,
(6.
d.l.k.c.2
Figure 17: Selective attention (SAtt) with text inputs: What the (row, column) grid locations of the target character? The top-left element the grid (0, 0).
31

---

Published as a conference paper at ICLR 2025
Soquence of taps yellow)
Cholces 3,4,5,0,12
5,0,2,4,3 5,1.0,2,,44 4,233.5L00
Choices 1,4, 0,0 2,0,6,4,11, C 0,4. 1,4.2,65500
Choices 1,2,4,3,0 2,3,2,4,0 2,4,6,0,3
1,2,4,0,3
Figure 18: Corsi block tapping (CBTT) with visual inputs: Each row shows a set of blue boxes. The boxes are lapped in sequence (highlighted in yellow). What is the sequence of taps? Use the box ids in the reference.
Sequence of taps ("T")
Reference numbers
0,0,0.0.0.0,3
0,0.0.0.0.0.0
0.8.0.0.0.0,0
0.8,8.0.0,0.8
0.0.8.0.8.2.8
Choices
0,0,0,0.,0,3
0,0.0.0.0.0.0
0.3.0.0.0.0.0
0.0,9.0.0.0.0
0.0.0,
A) 2, 5, B) 4,2, 3,
0.0.B.0.0.8.0
0.9.1.0.0.6.8
0.0.8.0.0.8.
9.0.6.9.9.1.9
8.8.0.0.0,8.
0,8,0,0,0.81
8.0.
0.0.0.0.0.0.7
8,8.0.0.8.0.8
0.0.0.0000088
0,0,0,0,0,8
0,0
C) 1,3, 5,2
0,0.0.0.0.3.0
0,3,0,0,0,0,0
0.0.0.0.0.0.0
0,0,0,0,0,0,8
0,0,0,0.3,2,0
D) 4,2, 6,3,
0,9,0.0,
B,B.0.0.9...8
8,8,0,0,8,0.0
B,8 8.8.0.0.....
8,
3,4.9,9,0,8,0
8.8.0.0....88
U.O.O......o
8.8.0.0.0.0.0
8.0.0.0.0.0.0
0.8.8.8.0.0. 8.8
0.0.0.0.8.8.
Cholees
0,8,0.0.0.9.3
0.0.0.0.0.8.3
0.B.O.0.0.9.8
0.0.0.0.008.8
0.3.0.0.0.1.0
0.8.0.0.0.0.0
0.0.0.....00
8,
8.8.8,8,0.8.
0.3.0.0.9.0.0
0,0,9,0,8,4,0
A) 2, 5,4, B) 5, 6,1,1, C) 6,
0.0.0.0...00
0.0.0.0.000.
8 0.0.0.0.
0.2.0.0.8.8.1
8.3
0.0.0.0.0.0.0
0 0.0.0.0...
0,0,9,9.8,8.8
B.O.O.T.O.O.G
8,0.0.8.0.0.0
8.8.0.8...0
8.0.8.8 0.0.
5.0.0.6.0.8.8
0,0,0,0,0,0,3
0,0.0.0.0.3.3
0.3.0.0.0.0.0
2.9.0.0.9....
0.0.3.0.0..3
0,0,0,0,0,8,0
D) 2,4,
Choices
B. 8.8.0.0.0
8.I.0.0.00
0.8.0.0.0
0.0.0.0.8
8.1.9.8.8
A) 2,
B.B.B.
8,
B.R.B.
8.8.B.8.B
8.0.0.0.0
0,0.0.7,6
0.8.8.8,T
8.0.0.0.8
0.9.9.7.6
5.9.9.8.8
B) 2, 7,
0 0.0.0.0
0.0.0.0.0
0.8,8.0.0
0.0,0.0.0
9.2.4.0.8
C) 2,
0.0.0.00
0.0.0.0.8
0,0,0,0.8
0,0.0.0.0
8.0.0.0.3
D) 1,3,7,6
Figure 19: Corsi block tapping (CBTT) with text inputs: Each row shows a set of boxes (B) laid out in space. This is shown 2D array where 0 is empty space. The boxes are tapped in sequence (highlighted as T). What the sequence of taps? Use the box ids in the reference. The array colors purely for illustration purposes.
32

---

Published as a conference paper at ICLR 2025
Array
Array 2
Choice 1
Choice
Choice
Choice 4
Figure 20: Spatial addition (SAdd) with visual inputs: What is the sum of the two arrays? Ignore red circles. Blue circles represent white circles represent 2 and empty spaces represent 0.
Arrny
Array 2
Choice
Choice 2
Choice 3
Choice
R.B.E.R.E
E.B.E.EEE
C.W.E.EE.
E.E.E.E.E
E.E.E.E
C.E.C.E.E
E.E.C.E F.F.F. B
E F
F.F.F F E.E.B.B B.E C.C.E.C.C
F.F.F.F.D
C.E.B.E.E E,E.E,E,E
E.R.C.EEE E,E.E.E.E
E.E. E.E. E.C.E.E,B
E.E.E.E.E
E.E E
E.C.C.E.E
E.C.C.E.E
E.E.R E.F.F
E.C.E E.F.R
E.E.E
E.E.B
E,E.E
E.B.E
F.E.F
F.E.F
B.F.F
E.F.F
R.B.BEE.EEEEE
SEE.BEEB
F.F.F.F...FFFFF
E.E.E.ME E.E.E.E.EEE B.U
B.F.F.M.FBBB.
F.B.F.B.F
E.E.F.E
E.F.E.E.S..
C.E.E.EEE
E.E.F.E.EEEE B.E
B.B.B.E..EEEEE.
E.E.E.E. B.F.F.F. B.M.B.B.FFE
C. E.E.
B L F C.E.C.
E.E.E
E.E. E.E.E E.E.
E.E.E
E.E.
E E.
E
E
B
F
E.E B
E.E.EEEEEEE
CEEE E.E.E
F.F.F F.F.E E.E.F.E...EEEEEE
B
E.E.C.E
F.
F.B.F.F. F.F.F.FF
E.E.E.E.E.E
B.
Figure 21: Spatial addition (SAdd) with text inputs: What is sum two arrays? Ignore is O, and W 2. The array colors are purely for illustration purposes.
33

---

Published as a conference paper at ICLR 2025
Figure 22: Maze completion (MCT) with visual inputs: We illustrate examples of mazes used for the MCT task. We programmatically generate mazes of different sizes using Mazelib (Stilley) 2014). Blue cells are obstacles. Black cells navigable space. The yellow square represents the current location. The red circle represents the goal location.
0,0, 0,0.0.000000.000000000000000000.
0.0.0.0000.000000000000.00
8.0.D.0...D...0O0OO. 0,000
0.A.0
811.1.1111111111 1.1.1.1....11......1111111111111
1.
0.0.0.001..1...0000000 01110000000000000101110001101111
1.0.0.00.00000000
0,1,1,1,88
1,1,1,8
111.1111111111111111111111111111
1,1,1,1,1111,,,,11111111
8.1.8.00
0.0.0.0...0...0..0000.
8
0,1,0.
1,1.1.1111111111111111111 1.1.1.1111111111111111 1,1,1,1,
1.0.1.1111111..
0,0,00 0,0,0,0,11110
1,0,,, 1.0. 0.0.8.1..... 1.0, 0.0.1.111100 1.0,0,0...0...
1,A.1,11111
1, 0,0,0,0000000000..00.0000.00..00 0.0.0.0.,0000000 0.0.0.
1.1.0.0011.00000011.111
1,1.1,,,,,,,,,,,,,,,,,,,,,,1,,,,
0.1.0.1111...10.100.0000
1,1,1,8
1,,,1 0,0 1.1.1.111111111.1..11111
8.0,0.0.0.80 8.8.8..8.8.0 0,0.0,00000000...000000000000000 8 A 1,1,1,1,,,,,,,,,,
8.8,8.8.8.
8,0,0..00 8.9 0,0,0.0,800000
1.0.1.11111.000001 1.1.1.118..0000 0.0 0.0.0.0.00.00000000
1,,,,, 1111011111111111111
8.1.1.1 8
8
0.0.0.0.0
1,110
1,111111111111111111111111111111
1.8 1.0.0.0000.00
8,
0.0.0.0000001111100.000000000000
8.8.0.1 8,8,8 8,0
B.B.B.
1,1,1,11 1,,,1,11111 1,1,11111 1,1
0,,.1,111111 11111111111
1,0,
8,8.0
1.0.0.0......1......11
0,0,
11 1,1,1,1,111 1.1.1. 1.1,1,1111111111
1111
10111111
11.00 1 0.0.0.000.800
8, 8,0 8, 8,8 1.0.0.0.......111
0,0,0,00
1111111111111111111111111
1.1,1,1 1,1,1,1,
1, 0.0.0.1..000
0.0.0.0.0
1.0.0.00000000110..000000
8.0.8,8,00
1,,,
8.8.8.0.0.1.0
8,1.9 1,1,1,18
1,1,1,111
0,1,0
0.0.0.0.. 0.
1.0.0.0.0000..000
8 1.1.1111111
8
1, 1,0,1.1, 1,1,1,1111111 8.0
111.1111111111111111111111111111
8 8.8.8888888
8.8.8.888811.
0,1,8 1.8.
8.8.
1.1..........
8 1.1.8 8.1.8.1..1111 1.1.1.11111
0.1.1.1111.
8 111.8.11111 1.0.8....0000
0.0.0.0.000000 0,1.8 0.0.8.333.
011.01.000000000000.00000000
111.1.1111
0,1,
1, 1,,,0,1111111
1,1,1,1111111111
0.1.1.1.
0.8
6.8.0.1.......1
1.1.1.1.1....
1,1,1,1,
0.0.0.0.00
0,0,0,
Figure 23: Maze completion (MCT) with text inputs: We illustrate examples of mazes used for the MCT task. We programmatically generate mazes different sizes using Mazelib (Stilley 2014). are obstacles. 1s are navigable space. A represents the current location. G represents the goal location. The array colors are purely for illustration purposes.
34

---

Published as a conference paper at ICLR 2025
collected
Action: Open box
Action: Open box
Action: Open box
Action: Open box
Treasures collected
Treasures colleated
Treasures collected
collected
Treasurer collected
Action: Open box
Action: Open box
Action: Open box
Action: Open
We illustrate game plays the CSWM task two rows. each row, we show initial observation followed by actions taken and the resulting observations. Note how the box identities change after each step. This is intended to force models to remember boxes by their spatial locations instead of their integer identities. As treasures get collected, they populated the "Treasures collected" section of the game screen. When treasure is collected. new treasure placed one the boxes where treasure never appeared before.
Figure 24: Cambridge spatial working memory (CSWM) with visual inputs:
0.0.0.0.0:1.00 0.00
2.8.0.0....00
B.0
00-888-9-4.8.3
0. D.D.D.D.. D.
B.
0...3.00...0.
5.
2.0.0.0.0.000.0
0.
11
0.0.0.0...0
8.
2.
11
11
02.0
10.00
Manbos of LEcaSHDeS collocted::
Numbez of teossures collocted:
Narbes LECaSILEGS
Kurbee of tzcasures collected: 1//.
Manbez o5 LecasHipes collezted:
Nuahen
Treasures
collacted:
Action: Open box
Action: Open box
Action: Open box
Action: Open box
Action: Open has
B.B.
D. 2. 2.0
BL
0.0,0,0:1.0
0.0.0
0,0,0
Nurbez of Lecasures Coltoct::: 2
collecsod:
COl Locted:
colloctod: 1 1/7
Locted:::: 2
collectod: 2
Action:
Action:
Action:
Action:
Figure 25: Cambridge spatial working memory (CSWM) with text inputs: We illustrate game plays CSWM task in the two rows. each row, we show initial observation (provided as text arrays) followed by actions taken and the resulting observations. The boxes in each array are the non-zero elements. Note how the box identities change after each step. This is intended to force models to remember boxes by their spatial locations instead of their integer identities. As treasures get collected, the "Number of treasures collected" gets incremented. When a treasure collected, a new treasure is placed in one the boxes where the treasure never appeared before. The array colors are purely for illustration purposes.
35

---

Published as a conference paper at ICLR 2025
Prompt Direction estimation (Ego image)
USER:
You are senticut Al system capable of eaaaa landmarks in the world and navigating in Here is video taken in physical cnviormment as you were navigating in it. Understand enviromment you navigating build a map cuvironment. Landmarks are paintings of objects hung on walls.
visually understanding the physical world, performing spatial reasoning, remembering
environment keep your position locations landmarks
<IMAGE >>
<IMAGE
Pretend that you are standing facing the painting of Shopping Cart. See image below,
Shoppiing. Carto Cants
At what direction (in angles from -180 to 180 dogrees) is the painting of Nower relative to you? See image below. <IMAGE OF Sawn_Howez>
Here are your choices: 1)-39 2) III 31 4) -69
Think step by step. Then answer your question following json format.
"answor": <fill in ona 1/2/3/6 intagez choice value>
Prompt 2: Direction estimation (DM image)
You playing game 2D world. Each the image (in yellow). The black cells are obstacles, i... you cannot move over them. The blue colls are navigable spaces that you can move over. Some cells have landmarks them (red circles with a label). These are important remember. taken in the 2D world track of your position as well as locations of landmarks in the world.
USER:
shows immediate surroundings around you, with you the center of
were navigating in it. Understand the 2D world are uavigating build a map of the world to keep
<TMAGE 1>
<IMAGE 2>
You must now unswer a question hased on your understanding of the 2D world. Pretend that you are standing next to the landmark A. See image below.
<IMAGE OF
What is angle between line connecting your location the landmark A and line connecting your location the landmark Angles range from -180 to degrees. Here are your choices: 156 2) 66 4) -24 Think step by step. Then answer your question in the following json Eormat.
aanswer": <fill in one of 1/2/3/4 lncegez choice value>
Large-scale spatial cognition
Direction est.
Distance est.
Map sketching
Route retracing
Shortcut discovery
Ego image
questions
150
135
30
30
30
# videos
30
30
30
30
30
DM image
# questions
150
135
30
30
30
videns
30
30
30
30
30
DM text
questions
150
135
30
30
30
Small-scale spatial coguition
MRT
PTT
WLT
MPFB
JLO
SAUl
MCT
CBTT
SAdd
CSWM
Visual
questions
172
100
50
100
# imuges
139
20
300
250
51
200
297
300
Textual
# questions
40
100
50
50
100
45
50
50
50
Table 6: SPACE benchmark statistics: We show the number of questions, images, and videos for each SPACE task. For large-scale spatial cognition tasks, have video per environment. We generate questions navigation lasks based on these videos. Some small-scale spatial cognition tasks have multiple images for the same question (e.g.. MPFB, WLT, SAut and CBTT), while other tasks have multiple questions for the same image (c.g., PTT, MRT). For interactive tasks like MCT, M,WM, route retracing and shortcut discovery, images rendered conditioned the actions taken by the agent.
36

---

Published as a conference paper at ICLR 2025
Prompt 3: Direction estimation (DM text)
USER:
You playing game 2D world. The console of the game represented comma-separated array. Obstacles are represented using 0, i... you cannot move over them. Navigable spaces that you can move over are represented using 1. Some navigable spaces
character (A acci churacter. These hadmarks are important to remember. You will always be located at the center of the array with your position highlighted using Understard the world you navigating build map of the world keep track of your position well locations of Lamdmarks in the world.
landmarks represented
also navigable spaces
just
character. Here is sequence console screen recordings taken as you were navigating in the text world.
Start of console sereen recerdiags
Screen at time
0, D, 0, 0,0
0, 1.1
D, 1,
0, 0,0,0, 0,0
0, C,
0, 0, 1,
End of console acreen zwcordinga
You must now answer question based understanding of the 2D text world. Pretend that you standing to the landmark shown below.
1, 1,1, 1.1
1,
1, 1,
1, D2 0,
1,1,-,1,2 1.1
What is the angle between the line connecting current location landmark and the line connecting your current location to Landnark C? Angles range from 180 180 degrees. Note that you may not see landmark your inmeddaate vicinity. You question.
spatial knowledge from the sequence screen recordings locate your position and hoth landmarks answer this
Here are your choices: 1) 127 2) 3) 4) 97
Think step by step. Then answer your question following json format.
*answor"s <eill in one 1/2/3/4 intager
37

---

Published as a conference paper at ICLR 2025
Prompt 4: Distance estimation (Ego image)
You are sentient AT system capable of visually understanding the physical world, performing spatial reasoning, remembering landmarks the world and navigating in lere is a video taken in a physical enviormment as you were navigating in it. Understand the envirnment in the cuvironment. Landmarks are paintings of objects hung on the walls.
USER:
navigating build map the enviromennt to keep track of your position well as locations of landmarks
<IMAGE >>
<IMAGE 2>
Pretend that you are standing facing the painting of Shopping.Cart, See image below.
<IMEGEE Shopping__azts
the euclidean distances (in meters) to ench the following landmarks from your current position?
Landmark: painting of Guitar
<IMAGE OF Guicas?
Landmark: painting of a Horse.Cart
<IMAGE OF Berse_Cort>
Landmark: painting of Lawn, Mower
<IMAGE OF Savnl, Movers
Landmark: painting Hammer <IMAGE OF Hammer>
Landmark: painting of Soccer_Rall Balls
Here are your
2,2, 4.0, 5.7, 5.0
217.7 4.0, 22, 5.0
1.7, 20, 3.3,
4.0, 5.0, 22, 5.7,
step by step. Then answer your question Following json
Tammeess still une 1/2/3/4 intagez choicw value>
Prompt 5: Distance estimation (DM image)
USER:
You are playing game in 2D world. Each image shows the immediate surroundings around you, with you at the center of image yellow). black cells are obstacles, i.c. you cannot move over. Some blue cells have landmarks in them (red circles with a text label). These are important to remember: Here is a video taken in the 2D world as you were navigating in it. Understand 2D world you are navigating in, build a map of world to keep track of your position well locations of landnarks in the world.
blue navigable spaces you
CIMAGE >>
20
You must are the cuclidean distances (in meters) from landmark C to each of the following landmarks: B, A, N, O. Y? Assume that each grid square borders)
question based your understanding of the 2D world. Pretend that you stamling the landmurkk What
size. are your choices: 4.5, 11.4, 11.2 4) 12.5, 6.0, 3.4, -355, 7.2
3) 10.0.
step by step. Then answer your question following json
*answer": <fill in ore of 1/2/3/4 integer choice value>
38

---

Published as a conference paper at ICLR 2025
Prompt Distance estimation (DM text)
USER:
You are playing game in 2D text world. The console of the game is represented as comma-separatee separated text array. Obstacles represented using i... you navigable spaces have landmarks represented as an asci character (A 2). These are abs navigable spaces and are just labeled with always
move them. Navigable spaces that you
over represented using
the arra with your position highlighted using the ne" character. Here is sequence of console screen recordings taken as were navigating in the 2D text world. Understand the world you are navigating in, baild a map of the world to keep track of your position as well as locations of landmarks the world.
ascii
landmarks importunt remember.
the
console recerdings
0, 0.0 0,
0, 1,
Cr 1.
0, 0, 1,0
Screen at Line
0, 0,
0,0, 0.0
D, m 1.1
0. 0, 1,1
comsole accwen ewcordinga
You must now answer a question based on your understanding of the 2D text world. Pretend that you are standing on the landmark C. What the euclidean distances meters) from lamimark C to each of the following landmarks: A, N. O, Y? Assume that each array location Im x size.
Here are your choices: 1) 4.5. 8.5, 11.4, 11.2, 10.0 2) 10.0, 19.2, 16.5, 12.5 3) 4.5, 11.2, 10.0 11.2 114, 8.5, 4.5. 10.0
Think stop by step. Then answer your question the following json format.
*anssor": <rill in one or 1/2/3/4 intoger choice value>
39

---

Published as a conference paper at ICLR 2025
Prompt 7: Map sketching (Ego image)
You landmarks in the world and navigating in Here is a video taken in physical cuviormment as were navigating it. Understand the enviromment you are navigating in, build a mup of the environment to keep track of your position as well as locations of landmarks enviromment. Landmarks are paintings of objects hung the walls.
system capable visually undersianding physical world, performing sputial reasoning. remembering
CIMAGE >>
must sketch map environment are the landmarks present in the environment. Landmark: SoccerBall
of start, goal locations. refresh your memory,
<IMAGE OF Beccor_acll>
Landmark: Shopping,Cho
<IMAGE Shopsing_Cazt>
Landmark: Lower
<IMAGE Lavn_Mowez>
Landmark: Horse.Cart
<IMAGE OF Horse_Cart>
Landmark: Guitar
<IMAGE OF Guitar>
Landmark: Hammer
<IMAGE Eanner>
Follow these map conventions. Your initial heading direction the video must be along the axis (upward). Which of these map sketches best capture the structure of the enviromment?
Choice
<BKETCH INAGE cP Choiue 1>
Choice
<SKEРCH INAGR CP Choice 2>
Choice
<SKETCH INAGB Choice
Choice 4
<OкEРCH INAGR Choice
Think by Then answer your question the following json format.
*answor": <fill
1/2/3/4 intager choice
40

---

Published as a conference paper at ICLR 2025
Prompt 8: Map sketching (DM image)
You are playing game in 2D world. Each image shows the immediate sumoundings around you, with you at the center of the image (in yellow). The black cells are obstacles, ie.. you camnot move over them. blue cells are navigable spaces that you can Some blue cells have landmarks them (red circles with label). These important remember. Here video taken in the 2D world as you were navigating in it. Understand the 2D world you are uavigating in. build a map of the world to keep your position well locations of landmarks world.
USER:
<IMAGE >>
must now map environment locations landmark locations, your understanding 2D world. Which of these map sketches best capture the true structure of 2D world?
Choice
<SKETCH INNGE OF Choice
Choice 2
Choice 3
<SKE7CH INNE OF Choice 2>
Choice
<SKECCH INNE OF Choico d>
step by step. answer your question following json
*answor": <fill in one of 1/2/3/4 iateger choice value>
41

---

Published as a conference paper at ICLR 2025
Prompt Map sketching (DM text)
You are playing a game 2D text world. The console screen of the game represented as comma-scparrted text arry.. Obstacles represented using i..., you camnot move over Navigable spaces you muve over are represented using Some navigahle spaces have landmarks represented as ascii character (A 7J. These are also navigahle spaces and are just labeled with an ascii character. 'Teee landmarks are important remember. You will always be located at the center of the arryy with your position highlighted using the "e" character. Here text world. Understand the world you are navigating in, build map of the world keep track of your position as well as locations of landmarks world.
USER:
of console
recordings taken you navigating the 2D
Screen at time conacle
zecordings
D, 0,0,0, 0,0 0,
D, G.C, 1.
D, 1,0
Screen at iime
0, 0,000
0, 1,1
0. 0,
coneo'l e screen recordings
You must now skotch map of the environment with the locations of the start and landmark locations. Use your understanding of the world.
Which of these map sketches best capture the structure of the 2D world? The map sketches 2D with markers highlighting the landmarks (ascii characters from to Z) and the start location (marked as *). Locations in the mup sketch with are empty and ignored.
Choice
Chaile
Choice
<7FX* ARBAY OF Choice
Choice
<"EX"" ARRAY oF Choice
Choice
<"EX" ARRAY or Chai oe
Think step by step. Then answer your question the following json format.
*answer"; <till in ore ot 1/2/3/4 integer choice value>
42

---

Published as a conference paper at ICLR 2025
Prompt 10: Route retracing (Ego image)
SYSTEM:
You are seatient living creature capable of navigating in cavironments, building internal spatial representations of cavi- ronmentt, and linding gouls in them. You will shown a video of the shortest rouie from the initial position to the goul. You must look taken. Then, will placed the enviromment the
video and understand the envirenment structure and the initial position. You must navigate from the initial position to the goal using the same route shown in the video, as quickly as possible. Below, you will find sections highlighting details about task. You
these for information.
OBSERVATIONS:
Thee images are recorded from perspective viewpoint (i... egoceatric or first-person). This means that you are likely see objects from different angles, resulting skewed appearance underlying 3D objects. It is important for look past this skew in the appcarance and percive the true shape of the object in 3D.
GOAL:
You will be provided an object goal using description and image the object. You must find the goal object in the environment by repeating the path shown the video walkthrough. Once you find il, move close the location of the goal and re orient yoursell to face the object.
ACTIONS:
You have four actions available.
forward by 0.25m alony the current heading direction. does not change the heading angle. turn left: decrease your heading angle by 30 degrees. It does not change the (x. y) position.
forward:
turnurghh::
your heading angle
degrees. does change the position.
stop: ends the current task. Issue this action only if you think you have reached the goal. If you baven't reached the goel. this action mavigation failure cunnot
STUCK IN PLACE BEHAYIOR:
Avoid getting stuck in one place, i.c. do not alternate between left and right tarns without going anywhere. You must try and move consistently
being
one place.
STOP CRITERIA:
Before executing stop, you must ensure that you've "reached" the goal correctly. To reach goal, you have to muve close enough to the wall where you the goal,
the object clearly your observation in front you.
RESPONSE FORMAT:
Respond in the following format:
Reasoning: text explanation string
short sentences provide all your explanations and inner thoughts here avoid
verbosity
Intent: state your intent in one short sentence, i.e. what you are trying achieve
Then provide the final action to take in jscn formatted string.
*astion"s <acti
nove Forwerd, turr_lefs, rightright, atogo
USER: sequence of walkthrough video demonstrating the route you Analyze walkihrough understund the movements and the maze structure, Take note of all details needed help you repeat this route when navigating next. Think step by step.
<IMAGE 2>
<TMAGE: 2>
USER:
Now, you must navigate to the goal. Here is the goal description and the image: Painting of Seccer Ball
<TMAGF: Socper. r.
USER: Here is the observation.
<CURREN" TMAGP: ORSERVATTONN>
ASSISTANT:
USER: Here is the current observation.
<CURREN" TMAGP: ORSERVATTON
43

---

Published as a conference paper at ICLR 2025
Prompt 11: Route retracing (DM image)
roste from an position goal. You must look the video and understand the 2D world structure and the route taken. Then, you will be placed in the 2D world the same initial position. You must navigate from the initial position to the goal using the same route shown the video, as quickly refer these more
are playing game world.
shown video
possible. Below; you will find sections highlighting details about the 2D world and the information.
2D WORLD:
The world consists of the following.
black cells: these are obsincles, ie., canmot move over them
navigable spaces, ie.. you move them Some blue cells contain landmarks, which are red circles Gilled with text character. These are important as they will allow to world
yourself. position
using yellow square.
OBSERVATIONS:
The images are recorded from birds-eye view of the 2D world. The images capture a local neighborhood surrounding your current image while world changes around you.
position world, Lo.., you will always
GOAL:
You will asked to navigate goal landmark. You must the goal in the 2D world by repeating the path shown the video. Once you find it, to the location the goal till you standing the landmark and then execute stop action.
ACTIONS:
have four actions available.
up: up one unit cell
down: move down by one unit cell
unit right: move right by one unit cell
left: left
stop: ends
only you you goal. If you reached goal, will result in navigation failure that cannot be recovered from.
STOP CRITERIA:
that you' 've "reached" goal correctly. To reach goal. you the containing the goal landmark. Then execute the stop action.
Before exccuting stop, you
RESPONSE FORMAT:
Respond in the following format:
Reasoning: text explanation string in one or two short sentences - provide all your explanations and inner thoughts here avoid verbosity and be concise
Intent: your
you trying achieve action in json formatted string.
short sentence,
Then provide
Maction"s
Saptios name must be one of up, down, left, sight>
USER: Here is sequence of vidco frames recorded in the 2D world. This demonstrates the route you need repeat. Analyze the vidco uoderstand the movements and the world structure. Take note all the details needed help you repeat this route when navigating next. step by step.
<IMAGE :>
<IMAGE 2>
ASSISTANT:
USER: Now, you must navigute to the goul baved on your knowledge of the 2D world you obtained from the videeo. Here the goul description: landmark Y
USER: Here is local view of your surroundings 2D world. You are the center this view.
SCURREN" TMAGE: ORSERNATION
ASSISTANT:
USER: Here local view of your surroundings world. You are the center this view.
SCURRENT IMAGE: OBSERNATION
44

---

Published as a conference paper at ICLR 2025
Prompt 12: Route retracing (DM text) part
You playing game in 2D text world. The console screen the game represented comman-sepun cemma--eparated amay. You will be shown a sequence of console sereen recordings that demonstrate the shortest route from an initial position to a goal. You must
SYSTEM:
sequence and understand the at the same initial position. You musi navigate from the initial position to the goal using the same route shown in the screen recording highlighring details
weeld structure
Then, you
placed in the 2D
sequence, as quickly as passihlle. Relow, you refer these for more information.
2D
task. You
2D TEXT WORLD:
console the game represented comma-scperaced array. Obstacles represented using Le., you cannot move them. Navigable spaces that can move over represented using 1. Some navigahle spaces have landmarks represented as ascii character Z). These
navigable spaces and are just labeled with ascii character. These landmarks are important to remember. You will always be located the center the with your position highlighted using the "*" character.
OBSERVATIONS:
images recorded from birds-eye view 2D world. images capture a neighborhoud surrounding your current position in the world, ie., you will always remain at the center of the image while the world changes around you.
GOAL:
world repeating the path shown console screen recording sequence. Once you lind it. move to the location of the goal till you are standing on the landmark and then execule stop action.
You be asked to navigate goal landmark. You find goal
ACTIONS:
You have four actions available. up: move up by unit cell
down: move down by one unit cell move left by one
right: move right by one unit cell
stop: ends the current lask. Issue this action only il you think you have reached the goal. If you haven't reached the goal, this action recovered from.
will resalt navigation failure that
STOP CRITERIA:
Refore executing stop, you must ensure you' "reached" goal correctly. To reach goal, you have move the containing the goal landmark. Then execute the stop action.
RESPONSE FORMAT:
Respond in the following format:
Reasoning: text explanation string one short sentenees provide all your explanations and inner thoughts here avoid short sentence, what you trying achieve Then provide the final action
verbosity
Intent: your json formatted string.
...
*action":
<action name most one of up, down, lett, right>
Here is the sequence of console screen recordings taken in the 2D text world. This demonstrates the route you need to repeat. Analyze the sequence to understand the movements and the world structure. Take a note of all the details needed to help repent this route when navigating Think step by step.
USER:
**d Conscle screzn recorded at time
0, 0,
0, D, 0.
0, 0,
...
#11 Conscle scresn recorded at time
0,0,0 0,1yiy 1.1
0, 1,1
0, 0,1,1,1,1
45

---

Published as a conference paper at ICLR 2025
Prompt 13: Route retracing (DM text) - part 2
ASSISTANT:
USER:
Now, you must navigate to the goal based on your knowledge of the text world you obtained from the sequence of console recordings. Here the goal description: landmark Y
USER: Here is a birds-eys view of the 5x5 area sarrounding current position. You are located at the center of this view. Your position denoted by
D. 0, 0,
0,
B, C.
0, 0, 7, 1,
landmarks
your
landmark navigable spaces, i.., you move over
them. Your objective is to reach landmark: Y
ASSISTANT:
Here is a birds-eye view of the 5x5 area sarrounding current position. You are located at the center of this view. Your position denoted me"
USER:
0, 0.
D. C, 1.
D, 1.0
landmarks visible your local
that landmark navigable spaces, ie., you over
them. Your objective is to reach landmark:: Y
ASSISTANT:
46

---

Published as a conference paper at ICLR 2025
Prompt 14: Shortcut discovery (Ego image)
SYSTEM:
You are sentient living creature capable of navigating in environments, building internal spatial representations of envi- ronments, and finding goals in them. will shown video some route from the initial position to the goal. must look the vidco and understand the envirenment structure, and remember the locations the and the goal. The video may show long-winded route from start to the goal with unnecessary detours. Based on the enviromment structure, you must identily a laster route to the goal. Then, you will placed in the enviromment the initial position. You must navigate the goal using your identified shortest route as quickly as possible. Below, you will find sections highlighting more details about the task. You can refer to
OBSERVATIONS:
The images are reconled from perspective viewpoint (i.e, egocentric or first person). This means that you are likely to see objects from different angles, resulting skowed appearance the underlying 3D objects. It is important for you look past this skew in the appcarance and perceive the true shape of the object in 3D.
GOAL:
You will he provided object using description and image the object. You must find the goul object in the enviranment by identifying the shortest route based on your experience from the video. Once you find the goal, move close to its location and re- orient yourself
object.
ACTIONS:
You have four actions available.
forward by 0.25m along the current heading direction. It does not change the heading angle. lum_eeft: decrease your heading angle by 30 degrees. It does not change the (x, y) position.
forward:
degrees. stop: ends the current task. Issue this action only if you think you have reached the goal. If you haven't reached the goal, this action result navigation failure cannot recovered from.
lum.right: increase your heading angle
change (x. position.
STUCK IN PLACE REHAVTOR:
Avoid getting stuck in one place, i.c. do not alternate between left and right tarns without going anywhere. You must try and move around consistently
being stuck one place.
STOP CRITERIA:
Before executing stop, you must ensure that you' ve "reached" the goal correctly, To reach goal, you have to move the robot close objoct clearly in your observation in front you.
enough to
where you the goal,
RESPONSE FORMAT:
Respond in the following format:
Reasoning:
verbosity explanation string
short sentences provide all your explanations and inner thoughts here avnid
Intent: state your intent in one short sentence, i.., what you are trying to achieve
Then provide the final action to take in json formatted string.
*astion":
<act
Forward, curn_left, turr_riggt
USER: Here the sequence frames from walkthruugh video demonstrating suhoptimal route from to goal location. Analyze the walkthrough to understand the movements and the environment strücture. Keep track of start goal locations, and current location in the environment as you watch the walkthruuh. Then plan a shorteut route that takes you to the goal while avoiding unnecessary detours. Think step by siep.
<IAGGE >>
CIMAGE 2>
ASSISTANT:
USER: Now, you must navigate to the goal. Here is the goal description and the image: Painting of Soccer Ball
<TMAGF:
Socper r. Rall3
USER: Here is the chservation.
<CURREN" TMAGE: OBSERVATTONS>
ASSISTANT:
USER: Here is the chservation.
<CURREN" TMAGP: OBS
ASSISTANT:
47

---

Published as a conference paper at ICLR 2025
Prompt 15: Shortcut díscovery (DM image)
You are playing game 2D world You will look at the video and understand the 2D world structure and remember the locations of the start and the goal. The video may show long
SYSTEM:
vidco
initial position goal. You
the structure, you must identify faster the goal. Then, you will be placed in the 2D world at the same initial position. You must navigate from the initial position to the gool using your identified shortest route as quickly as possible. Below, you will find sections highlighting more detuils about 2D world and the task. You refer these for more information.
route
goal
unnecessary detours.
2D WORLD:
consists the following.
black cells: these are obstacles, i.c. you cannet move over them
blue cells: these are navigable spaces, Le., you can move over them
Some blue cells contain landmarks, which are red circles filled with character. These important they will allow you better understand the world and locate yourself. Your position will be marked using yellow square.
OBSERVATIONS:
The images recorded from birdseye view of the 2D world. The images cuplure a local neighborhoud surroumding your current position the world, Le., you will always remain at the center the image while the world changes around you.
GOAL:
You will he asked to navigate tn goal landmark. You must find the goal in the 2D world by identifying the shortest path based your your experience from the video. Once you find it, move location of the goal till you are standing on the landmark and then execute stop action.
ACTIONS:
have actions available. up: move up by one unit cell
down: move down by one unit cell
left: left
unit cell
right: move right hy one unit cell
stop: ends
only you think you reached gual. If you reached goal, will resalt in navigation failure that cannot he recovered from.
STOP CRITERIA:
that you'se 've "reached" goal correctly. To reach goal, you the cell containing the goal landmark. Then execute the stop action.
Refore executing stop, you
RESPONSE FORMAT:
Respond the following format:
Reasoning: text explanation string in one or two short sentences provide all your explonations and inner thoughts here avoid
verbosity and be concise
Intent: your
short sentence, what you trying achieve
Then provide
action in jann formatted string.
*accion":
<action name mast be cne of up, down, left, right>
Here is the sequence of video frames recorded in the 2D world. This demonstrates a suboptimal route from the start to some gnal location. Analyze the video to understand the movements and the world structure. Keep trck of the start and goal locations, and the current Incation the warld you watch the video. Then plan shortcut that takes you the goal while avoiding any unnccessary detours. Think step by step.
USER:
CIAGGE <IMAGE
ASSISTANT:
Now, you must navigate to the goal based on your knowledge of the 2D world you obtained from the vidco. Hore is the goal description: landmark Y
USER:
USER: Here is the local view of your surroundings in the 2D world. You are the center of this view.
0BSERVAZION>
ASSISTANT:
USER: Here is the local view of your surroundings in the 2D world. You are the center of this view.
INAGE OBSERVAAIIO>>
ASSISTANT:
48

---

Published as a conference paper at ICLR 2025
Prompt 16: Shortcut discovery (DM text) - part
SYSTEM:
You are playing game in a text 2D world. The console screen the game represented as comma-separated text aray.. You will be shown sequence console screen recordings demonstrates route at the sequence and undersiand the 2D text world structure and remember the locations of the slart and the goal. The recordingss may long- winded mone from route to the poal. Then, you will be placed the 2D text world at the same initial position. You must navigate from the initial position to the goal using your identified shoress route as quickly as possible. Below, you will find sections highlighting more details about the 2D world and the task. You refer these for more information.
initial position goal. You look
goal unnecesary detours.
on the world structure, you must identify
2D TEXT WORLD::
The console the game represented comma-separated text array. Obstaeles are represented using O, i.., you cannot move over them. Navigable spaces that
represented using Some navigable spaces have landmarks represented as ascii churacter (A Z). These are alsu navigable spaces and are just labeled with un uscii churacter. These landmurks are imporiant to remember. You will always be located the center the army with your position highlighted using the "*" character.
OBSERVATIONS:
'The images are recorded from birds-eye view of the 2D world. The images capture a local neighbothood surrounding your current position the world, Le., you will always remain at the center of the image while the world changes around you.
GOAL:
navigate goal landmark. You your your experience from the screen recording sequence. Once you find it, move the location of the gool till you standing the landmark then exccute a stop action.
the goal
world by idlentifying shortest path
ACTIONS:
You have four actions available.
up: move up by one unit cell
down: move down by unit cell
left: move lelt by one unit cell
right: right by unit cell
stop: ends the current task. Issue this action only if you think you have reached the goal. If you haven't reached the goal, this action recovered from.
will
navigation failure
STOP CRITERIA:
Before executing stop, you must ensure that you' 've "reached" the goal correctly. To reach gaal, you have to move to the cell contnining the goal landmark. Then
the stop action.
RESPONSE FORMAT:
Respond in the following format:
Reasoning: text explanation string
short sentences provide all your explanations and inner thoughts here avoid
verbosity
Intent: state your intent in one short sentence, i.e, what you are trying to achieve
Then provide the final action to take in jsun formatted string.
"accion": <action nome most
up, SoNn, lest, right>
USER:
sequence console screen recordings to undersiand the movementt and the world structure. Keep track of the start and
world. demonstrates suboptimal route
start to some goal location. Analyze the goal locations, and the current location in the world as you study the sequence. Then plan shortcut route that takes you to the goal while avoiding any unnecessary detours. Think step by step.
244 Consele le screen recorded at time
0, 0, 0.
8, 0, 0, 0,
0, C.
8, 0. 1.
screen recorced time
0,000,0,0
0,0,
0, 0,1,,
0, 0,0,C, 1.1
0,0,0,
49

---

Published as a conference paper at ICLR 2025
Prompt 17: Shorteut discovery (DM text) part 2
ASSISTANT:
you navigate goal based your knowledge world you obtained sequence screen recordings. Here the goal description: landmark Y
USER:
eye 5x5 surruunding your current position. You are center
position denoted by
0,000 0, 0, 0,
landmarks visible in your context arc: that landmark locations also navigable spaces, i.c., you can move over them. Your objective is reach landmark:
ASSISTANT:
USER: birds-eye
5x5 surrounding your current position. You located the view. Your
position donoted thah
Cy Dy Oy D,
1, 1,
landmarks visible your
navigahle spaces, i.e,, you mowe over
them. Your objective is to reach landmark: Y
ASSISTANT:
Prompt Mental rotation (vision)
USER:
Here is an image of a three dimensional shape.
these images show same object rotated 3D?
Choice
<CROCCE
Choice
<CHOTCE INAGE>
Choice
<CBOTCE IKAGE>>
Choice
<CROCCE IMAGE>
Think step by step. Then answer your question in the Following json format.
*answer"s xfill in of 1/2/3/4 integer values
50

---

Published as a conference paper at ICLR 2025
Prompt 19: Mental rotation (text)
USER: two-dineensional aray..
none,
these options same aray
only rotated,
Choice 1:
page, Iee0,
nume, soree, ridn oves, such, site
Choice
nono, none,
ai-Le suuchy piwa
Choice
page, none,
Choice
oveE, such, site
Think step by step. Then answer your question in the Following json format.
"anowess sfill in one of 1/2/3/4 integer value>
Prompt 20: Perspective taking (vision)
USER: Here is image of various objects (animate and inanimate) on ewo-dimensional planc.
<IMAGE OF OBJECTS>
Proteod that you standing the centroid of guitar and facing the centroid of Visualize the world around you. (from
located relative you? Clockwise rotations are positive and anti-clockwise rotations are negative. Here are your options: 1) 45 85 3) 4).5 Think by step. Then answer your question the following json format.
degrees)
"answor": <fill
1/2/3/6 intoger value>
Prompt 21: Perspective taking (text)
USER: Here arryy of numbers representing the hirds view dimennsiona plane.
0,7,0,9
0,000
0,0,5
0,3
Empty locations indicated
Important locations indicated with number 1- 9, Pretend that you standing the degroes) the
you. angle to you? Clockwise rotations are positive and anti clockwise mintions are negative. Here are your options: 1) 52 2) 32 3) 72 4) 112 Think by step. Then
lacing location Visualize the
your question the following format.
"answos"s <fill
1/2/3/4 Integez
51

---

Published as a conference paper at ICLR 2025
Prompt 22: Water level (vision)
USER: Here is container dilled with water.
PILLED WATRR CONTAINERS
What will be the water level when it is rotated as shown bere?
CONTAINER>>
are your
expected rotated container?
Choice
CHOICE
Choice
<IMBGE OF CHOICE 2>
Choice
<IMAGE OF CHOICE J>
Choice
<IMAGE CHOICE
Think stop by step. Then answer your question in the following jsoa format.
*answor"s <fil1 in one of 1/2/3/4 incegez value>
Prompt 23: Minnesota paper form board (vision)
image different pieces puzzle.
<IMAGE OF
PUZZLE DIECES>
These pieces put together by oracle. Which one of these four options shows what it would look like when the pieces are together? Pny close attention not just shape, individual pieces within shape.
Choice
<TMAGF:
CHOTOR
Choice
<TMAGE: OF CHOTOR 2>
Choice
<TMAGE: OF DHOTCR 3>
Choice
Think step by step. Then answer your question in the Following json format.
*answer": <till 1/2/3/s integer value>
52

---

Published as a conference paper at ICLR 2025
Prompt 24: Minnesota paper form board (text)
USER: You playing patting together puzzle. Here the pieces, where represents the interiors of the puzzle piece and represents the boundary.
1, 1,
1,0,0, 0,00
02 0,
0,0,0,
1,0,0,0,0,1 1,1,1
1,0,0,0,0,0,1
1,0,0,000,0,1
1,0,0,00,,0,1
1,,41,11
1,0,0,0,0,1
1,0,0,000,1
1,,,4,
1,4,,,111
1,0,0,0,1
1,0,0,0,1
1,1,9,1,1
These pieces are now put together solve the puzzle by an oraele. Which these four options shows what it would like when the pieces pat together?
Choice
<CHOΣCE ARBAY>
Choice
<CHOICE
Choice 3:
<CHOICE ARBAY>
Choice
<CHOICE ABBAY>
Pay close attention not pieces may need to be rotated before fitting them together Think step by step. Then answer your question in the following json format.
firted shape, the individual pieces contained within shape. puzzle
*answor"
<fill in ona oc 1/2/3/4 integer value>
Prompt 25: Judgement of line orientation (vision)
USER:
Here an image showing two lines. Your goal to measure angle between two
<IMAGE
SINEa>
Here legend showing set of reference lines numbered from to 1I.
LEGEND>
Which of the following reference line paiss match the angle between the original lines shown in the image?
Lines and
Lines and
3) Lincs and 10
Lines and 3
stop by step. Then answer your question following json format.
"answor": one 1/2/3/4 lntagez
53

---

Published as a conference paper at ICLR 2025
Prompt 26: Judgement of line orientation (text)
USER: Here is reference arry..
Oy 0, 0y O, B, 0, C2 3, 0, Cy 0,
B, O, C, D, C2 3, O2
3,
0, D, G D, C 0,
0, 0, Oy
0, 0,
0,0,
0,,00,,,,,,
0,
0,
0,0,0, 0.000.00..
0, 6, 0,000,00 8, 8, 9, 9. 0, 0,0,0
0,0,0
mean empty space, ignore the two lines. Specificalll, here are four choices of amays with two lines per Which one of the choices has an angle between the two lines that matches the angle between lines in the referenee aray?
two
2s, respectively. goal measure angle
Choice I:
<ARRAY OF CHOICE 19
Choice
<A8RAY OF CHOICE 2>
Choice
<188AY CHOICE
Choice
OF
Think step by step. Then answer your question in the following json format.
*answer": <till in 1/2/3/4 integer value>
Prompt 27: Selective attention (vision)
USER: Here an image of a apple. call the target.
<IMAGE OF
TARGET>
Here grid of apple images. This contains multiplc instances of apple, but not all of them are the target object. The grid indexed from top-left to bottom-right, starting from row, column (0, 0).
<IMAGE OF GRID>
Which these options represent true locations of the target object the grid! Locations are represented as (row, column).
Choice 3),
Choice 2. (0, 1) (1. 0), (2, 0) (3,
Choice
Choice (0, 3), (1. 2), (2. 2), (3,
step by step. Then answer your question following json
"answer": <till in ore ot 1/2/3/4 iateger volue>
54

---

Published as a conference paper at ICLR 2025
Prompt 28: Selective attention (text)
USER: Here is a grid of mumbers / letters. The grid indexed from top-left bottom-right, starting from row, column =
E, Ty
Ty exeyott t
Your goal is to find all occurrences of "c" in the grid. Which of these options represent the true locations of the where "e" occurs in the grid?
are represented (row, columm).
Choice
(2, (3, (4,3)
Choice (0, 3y, (1, 1), (2, 4), (3, D,, (4,
Choice
(4.
Choice 4. (0, 2), (1. 1), (2, 4), (3, 1), (4.3)
Think step by step. Then answer your question following format.
"anoxor": orc
intager
Prompt Maze completion (vision)
USER:
You are a sentient living creature capable navigating in mazes, planning, and spotial reasoning. You are playing a Pacman-style mazee game,
some position the must escupe maze as quickly as possible
goal. You
are given the game screen that shows the following:
maze structure-
obstacle space, black navigable space. You can only move on black spaces. You cannot move through blue
spaces.
your current position yellow square
goal position cirele
Below the screen, status
might appear indicating that collided into wall after previous action.
Actions available: You can take five possible actions.
position by one step right move right from your current position by one step position one step down move down from your current position by one step
Jett
your
up move up from your
stop issue this action only after you have reached the goal position. If you execute it prematurely, you will fail, If you do not execute after reaching the goal, you will again fail.
Response format: Respond
following formal.
explende repllentiinn string explain your <next, provide jeon fosmatied outipa with next actloon
sel
Paction"; "eect ians*
ASSISTANT:
...
USER: Here is the current of the
<TMAGE. OF MARr>
Think stop-by-stop about how to reach goal. What action do you take next?
ASSISTANT:
USER:
Here is the current state of the maze.
Think stop-by-step about bow to reach the goal. What action do take next?
55

---

Published as a conference paper at ICLR 2025
Prompt 30: Maze completion (text)
USER:
You sentient living creature copable navigating maze game. You start at some random position in the maze. You must escape the maze as quickly as possible to reach the goal. You are given array representing muze, which maze structure 0 obstacle space, navigable space. You can only move on Is (i.e, navigable spaces). You cannor move through (i..,, obstacles).
planning, and spatial reasoning. You playing text-based
following:
position- marked goal position - marked as
Goal current positions always navigable spaces.
Actions available: You take five possible actions. left move left from your current position by one stop right- move right from your current position
stop up move up from your current position by one step your current position by one step stup- issue this action only after you have reached the goal position. If il prematurely, you will fail. If you do after reaching the goal, you will again fail.
move down
Response format: Respond in the following format.
sthinkk atup-by huppbyy-ttep step about what action Lo Lake next. Bu cundise.? <next, provide 18e0 formttted output with the naxt action> ...
*action": Peactions*
ASSISTANT:
Here current the maze.
0,
0, 0, 1,1,1, 1, 0,
0, 0, 1, D, 1,0, C, 3,
0,0
0, 0, 0, 1, 0,
0,1,,,1,,,,,,,,,, 1.t.s.00..................... 0, 0, 1,11:/1,112,,,1,,,,,,, 0, 0,1, 0, 0,0, 0,0, C, 0.0...0.............0.00000000.. 1,,1,,,
2.1.
0,1,
6.
0.
1,0,,,,, 1.1.1,,,11....,,,,,11.1, 0,1,
0, 1,0,1,
0,0, 1.t.1...................
0.
1,,
6.1
1, 1, 1, 0,0,0,0,0,0 0,0,0,0,0000000 0, 0, 0, 0, 0,0,0,
represents obstacles. represents spaces. G is the goal. is your current position maze. Your current location maze row, column (1. 1). The goal location column (21, 19). Think step-by-step about how reach the goal. What action do you take next?
ASSISTANT:
56

---

Published as a conference paper at ICLR 2025
Prompt 31: Corsi block tapping (text)
dimensional array. The array contains emply shown ordered sequence arrys, representing sequence taps on the bourd. each stop of the sequence. one boxes is tappd, and it the tap is highlighted by marking box as I instead of B. You must remember the sequence of taps by remembering which exact boxes were tapped. lere is the sequence of urays.
USER:
You playing the Corsi hoard tapping (marked
The board represented
locations (marked B),
B2 O, O D Dy By D. Oy O
D O
By T B
D, D, O, O. D,
0, D, 0,
6, 0,
0, D. O
0,0,0 0, 0
B, 0, B, 7
0, 0,000, 0,
0, 0, 8, 0.0.0. 0,0,0
8, 0,0,
0,0,8
0.00
0,
8, 8,
0,
0,
0. 0, 8
0,
0. 0,
0, 0,0, 0,0,
By
B, D,
0, 0, 0,
0. 0, 8,
0, D, 0. 0,
T. 0, 0,
0, 0,0,0, 0, 0,0,0,
8,
You must now identify the sequence of taps. Here is the corsi board with numbers 7 assigned on each box. Use this numbering reference answer the question.
0, 0, 3,
0, D, D,00
7, 0
G, D, 0,
0,0, 02
What is the sequence boxes that tapped! Here cheices:
(1) 1, 2, 4,
2, 6.
(3 3,
(4) 3,3,
step by step. Then answer your question following json
Pamsoeess sEill use 1/2/3/4 integez
57

---

Published as a conference paper at ICLR 2025
Prompt 32: Corsi block tapping (vision)
You playing the Corsi board lapping game, You will shown video with boxes (bluc squares). These boxes will be tapped one specific soquence.
USER:
will shown by highlighting box yellow. You remember sequence of lups by rememberinng which exact boxes were tapped. Here video.
<IMAGE 2>
<IMAGE 3>
identify the sequence taps. Here imago of the board numbers relerence answer the question.
image
<IMAGE NUKGERS oK BoxBs>
Here your choices:
(1) 1,
(2)
(3)0, 1,4
(4) 0,
step by step. Then answer your question following joon
"answor": <rill in ore or 1/2/3/6 intogee value>
Prompt 33: Spatial addition (vision)
You are playing the arry addition You have add two arrays following certain rules. Each arry location be emptyy (ie.. fully while) or filled with colored circles. Empty locations represent zeros. The colors circles meun specific things.
USER:
Blue circle
Red cirole is a distretion and must be ignored (i.c., it does nor contribute to the aray addition)
White circke
Aray addition works follows::
sum of zeros must zero (i.e., an empty ammy cell) (i.c., blue circle)
of and
and onc)
sum of one and one must be two (l.c., white circle)
Here the first aray.
<IMBGE OF ARBAY 1>
second army.
<IMAGE OF ARBAY
What is the of the arrays? Pick from one of these four choices.
Choice
<IMAGE CHOICE
Choice
CHOTCE
Choice 3
<TMAGE CHOTCR
Choice
<TMAGE. OF CHOTOR
step by step. Then answer your question following json
*answer": <fill in ore of 1/2/3/4 integer value>
58

---

Published as a conference paper at ICLR 2025
Prompt 34: Spatial addition (text)
USER:
You are playing the arry addition game. You have to add two arrays by following certain rules. Each array location can be represents represents two. is distraction and ignored does
filled with B, W. E represent
not contribute to the array addition)..
Army
follows:
sum of zeros must be zero (i.c., E)
sum of one zero (or zoro onc) must one (i.c,,
of and
two (.e.,
Here the first aray..
E.R.H,E,E, B,B. B.B E, E, R, E, F. Ft
B
E, F,R, F.,.R
Here second array.
F. E,
F, R. F2 two arrays? Pick one these
E, E,
Choice 1:
F,R.R
E, B, R, B,
F, R. B, R. E, F. R, 8,
Choice
E, H,B
E, R, F,F, F. E,
R, E,E.R
Choice
E, F,
R, P.R E, F,E.R B, H, E
Choice
F, R, F, E, R,E,R,
E, F, R
B, E
E. M.
Think step by step. Then answer your question in the following json format.
*ansxer": <fill in of 1/2/3/4 integer uel
59

---

Published as a conference paper at ICLR 2025
Prompt 35: Cambridge spatial working memory (vision)
You are playing Cambridge Spatial Working Memory game. You will hidden in one of the blue boxes. You must ideniily the box containing the treasure, which is shown as an yellow square. Once you lind treasure,
USER:
sereen blue
trassre
hidden other boxes where the trensure did not appear before. You must again find the treasure. This process repeated till you find all treasures placed each the blue boxes once. Note: The treasure will never appear in a box where had already been placed to aid you with communication, ic.. specify what box you want opcn in that turn. However, these numbers will change after every turn. So do NOT associate boxes number identity of a on their spatial pusitions and not the numbers.
collected placed
"Treasures collected" section
the image. new treasure
Each tum, there randomly selected numbers associated with each box. These numbers
numbers over long
change any time. Therefore, you must
the
RESPONSE FORMAT:
"Think step-by-ste about where the treasure might be based on your past actions. Afler that, indicate box, you want to open in the following ison format:
*ast fon"s
<hox integer indexo
ASSISTANT: ...
USER:
Here is the current state of the gume. You must lind the next treasure. Note that the numbers of the boxes have changed, but the box locations fixed. Decide which box you open and then the number associated with the box the action.
<TMAGE OF SCREEN
ASSISTANT:
Prompt 36: Cambridge spatial working memory (text)
You playing Cambridge Spatial Working Memory game. will be shown array with integers. represents empty locations, Locations numbered -9 represent boxes. A treasure is hidden in one of the boxes. You must identify the box containing the treasure. you find a treasure, location will momentarily shown as then collected and lind the new treasure. This process repeated till you lind all treasures placed in each of the boxes once. Note: The treasure will never appear hox where it had already been placed.
USER:
indicating treasure will be hidden one of the other boxes where the treasure did not appear before. You must then
trensure was found.
treasure
While the boxes are represented using integers from 1- 9, the true identity of the box is its location (row, column) the amny. The and the number boxes will change). However, each associated with the box will change randomly. These integer ids are meant to aid you with communication. iec. specify what box you want open
location always fixed (i.c., boxes will
integer id
turn. However. these numberrs change every do asociate boxes numbers term. The number identity of a box can change any time. Therefore, you must remember the boxes hased on their spatial positions and not the numbers.
long
RESPONSE FORMAT:
Think step-hy-see step about where the might hased your past actions. After that, indicate the hox you open the following json format:
*accion": <box inboger Index>
ASSISTANT:
is current view You must next treasure,
of the have changed,
the box locations are fixed. Decide which box location you want to open next. Then provide the number associated with the box as the action.
D, 0, 0,
D D, D,
1
0, 0, O, C,
0, 0, 0,
0, 0
Number of treasures collected: 0/
ASSISTANT:
60


Reviews:
[{"review_id": "EHkrIjmT1I", "reviewer": "Reviewer_pZuW", "summary": "This paper proposes the SPACE Benchmark, designed to evaluate spatial cognition capabilities in large models by focusing on both large-scale and small-scale spatial cognition. Large-scale spatial cognition pertains to environmental reasoning, where the environment stays constant while the viewer’s perspective shifts. Conversely, small-scale spatial cognition involves the perception and imagery of changing spatial states of objects, with a fixed viewer’s perspective. The benchmark comprises 15 tasks distributed across image-based and text-based experiments, designed to test VLMs and LLMs. Each task contains questions between 30 and 100+. The findings indicate that current models underperform relative to human-level reasoning across various spatial tasks, suggesting that these models have yet to achieve robust spatial reasoning capabilities. Notably, some models, such as GPT-4 and GPT-4v, demonstrate superior performance in text-only tasks compared to multimodal tasks.", "questions": "A. In the direction estimation task, take Figure 6 for example, pretend that you are standing below landmark F and facing F, while N is right above F. Why does the angle become 78 degrees? The two landmarks seem to be almost on the same line.\n\nB. The article omits some details of data processing and requires greater clarification. For instance:\n\t1. In the section on Large-Scale Spatial Cognition, while the Bird’s-Eye View (BEV) is mentioned as covering 2.5m x 2.5m surrounding area, the dimensions of both the 2D and 3D views,  the duration of videos as well as number of sampled images of each video for the Ego image are not provided. It might affect the models' performance if the sampled images are over dense and include too much redundancy or images are too sparse therefore missing key frames containing landmarks.\n        2. For BEV text, the authors state that they “carefully select the encoding to ensure compatibility with text tokenizers” but do not detail this process. How is this encoding achieved, and how is the dimension of the text array determined? It would be helpful to know how many different array dimensions were tested to understand the effect of text encoding choices. Additionally, since LLMs reportedly perform better with BEV text, could this be due to the much smaller dimensionality of text arrays compared to BEV images? \n\t3. In Table 3, only the number of questions is presented per task, without image counts. Although most tasks include 30-50 questions, there appear to be fewer than 10 images for some tasks, which limits the robustness of any conclusions drawn. To strengthen the study’s validity, I recommend providing the exact image counts per task and ensuring a sufficient number of images for each task. This would enhance the reliability of conclusions and support a more balanced dataset for each task.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. It is interesting to test both VLM and LLM models’  large scale 2D and 3D spatial reasoning abilities using three scenarios: Ego Image/ Video, Bird’s Eye View (BEV) images, and BEV text. It also allows for the assessment of models across different modalities, including textual descriptions, single images, and video content. This design reflects the complexity of real-world spatial understanding and demonstrates the comprehensive nature of the evaluation.\n2. This article uses interactive tasks for route retracing and shortcut searching where the model receives the current observation, decides which action to take, and receives corresponding updated observations", "weaknesses": "A. Providing specific details such as the resolution of the videos and images used in the experiments would be better because models might take in different sizes of input images, therefore model performance can vary significantly based on input resolution. if an image is downsampled incorrectly, it may disrupt the original aspect ratio, potentially impairing the performance in tasks that require precise spatial reasoning, such as distance and direction estimation. This issue might partially explain why LLMs outperform VLMs in some scenarios.\n\nTo better explain this, could try to\n\t1. Specify the exact resolutions used for images and videos in their experiments\n\t2. Discuss how they handled any resizing or aspect ratio changes when preparing inputs for different models\n\t3. Consider running an ablation study with different image resolutions to quantify the impact on performance, especially for tasks like distance estimation\n\nB. Only two VLMs are tested for image relevant tasks. And those two models even belong to the same family (GPT). I would recommend to test more SOTA VLMs to validate your conclusions.\nC. As I stated in the following questions, the images used for experiments might be too small to draw a reliable conclusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IvxPlZW7E1", "reviewer": "Reviewer_Mbyv", "summary": "A benchmark is introduced for evaluating spatial cognition in LLMs and VLMs, comprising both large-scale spatial reasoning tasks (e.g., navigation) and small-scale spatial reasoning tasks (e.g., mental rotation). The results indicate that both VLMs and LLMs perform well below human participants on these tasks, indicating weak spatial reasoning abilities in these models.", "questions": "Previous work on spatial reasoning in LLMs should be cited, and the paper should address how the proposed benchmark goes beyond the already existing benchmarks, and what new information we have learned about spatial reasoning in LLMs.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed benchmarks are very extensive, covering different types of spatial cognition with many tasks and input modalities.\n- The paper is well written and many details are provided on the evaluation methodology, including detailed prompts and model settings.\n- The topic investigated in this paper is an important one, with many implications for real-world use of foundation models.", "weaknesses": "Although the benchmark appears to be well designed, there are a number of previous works that address very similar questions, and the results are not particularly surprising. Both Valmeekam et al. (2023) and Momennejad et al. (2023) have shown that LLMs are very poor at navigation and planning tasks (i.e., large-scale spatial cognition), and studies from both Yamada et al. (2024) and Ivanova et al. (2024) and have already shown that LLMs have a limited ability to reason about spatial relations. These works are not cited, and there is no discussion of how the proposed benchmark goes beyond previous studies of spatial reasoning in these systems.\n\nValmeekam, K., Marquez, M., Olmo, A., Sreedharan, S., & Kambhampati, S. (2024). Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems, 36.\n\nMomennejad, I., Hasanbeig, H., Vieira Frujeri, F., Sharma, H., Jojic, N., Palangi, H., ... & Larson, J. (2024). Evaluating cognitive maps and planning in large language models with CogEval. Advances in Neural Information Processing Systems, 36.\n\nYamada, Y., Bao, Y., Lampinen, A. K., Kasai, J., & Yildirim, I. (2023). Evaluating spatial understanding of large language models. arXiv preprint arXiv:2310.14540.\n\nIvanova, A. A., Sathe, A., Lipkin, B., Kumar, U., Radkani, S., Clark, T. H., ... & Andreas, J. (2024). Elements of World Knowledge (EWOK): A cognition-inspired framework for evaluating basic world knowledge in language models. arXiv preprint arXiv:2405.09605.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yL2c0Cc0CD", "reviewer": "Reviewer_61bg", "summary": "This work develops SPACE, a benchmark for systematic evaluation of spatial cognition in foundation models, vision-language or language only. These are a large variety of tasks designed to test spatial cognition taken from various places within the cognitive science literature. They separated the tasks into two categories - \"large-scale\" which are testing the development of a robust cognitive map of an entire environment and \"small-scale\" which are testing targeted spatial relationships between specific objects in various contexts (mental rotation, putting pieces of a board together, spatial addition, etc). The large-scale tasks have an egocentric visual representation of the task, an allocentric \"birds eye view\" visual representation, and an allocentric text-only representation. The small-scale tasks have a vision vs. text version of the tasks. They evaluate various popular closed models (e.g. GPT4v) as well as various open-source models (Llama3), both multimodal and language-only models. Performance was generally above chance but far below human performance.", "questions": "* Related to weakness 1, have you tried egocentric text representations? For a given location and direction of the environment, you could have a way to parse the egocentric image observation into a textual description (e.g. \"there is a landmark with a picture of a dog on the left wall and a deadend wall directly in front of you. On both your left and right sides, there are potential paths leading elsewhere\"). This kind of representation may be more ecological to some of these models. This could apply to the small-scale tasks (corsi block tapping,  spatial addition, cambridge spatial working memory) with grid inputs as well (e.g. \"Row 1 column 2 has a blue, row 2 column 5 has a red,...\") Even if it's a small change in how the input is represented, a change in performance would be very informative.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Very timely topic. \n\n* It is a quite impressive set of tasks that really engaged with the literature on spatial reasoning. The translation of spatial reasoning tasks to text only is an especially nice contribution, because it allows for the evaluation on text-only models. I think the benchmark will be immediately valuable to the community. \n\n* Comprehensive evaluation. Lots of models were used, which really gives us an idea as to how the current landscape of models performs at tasks like these.", "weaknesses": "1. The authors took the best first step in evaluating a capability that humans have (spatial cognition), which is to directly take tasks that have been used in cognitive science to test this capability in humans. However, these are *tests that were designed for humans specifically*, not Large (vision)-Language Models. And I think this would definitely have implications on the results. For example, vision encoders like CLIP are trained on a lot of ImageNet-style object image data and so a model using CLIP as a visual encoder may be less used to, for example, the egocentric observations or BEV images used in the large-scale studies. I don't know if the specific composition of GPT4v's or GPT4o's image training data is public knowledge, but I think the larger point still stands. This doesn't mean what the authors did is incorrect - it is a good first step and valuable contribution towards understanding spatial capabilities in these models. However, I think more work will need to be done in the future to *ensure these kinds of tasks are presented to the model in a way that is ecological.* This is really common in animal behavioral paradigms. Often when animal researchers are testing a capability known to exist in humans, the original paradigms are modified so that the task makes enough sense to the animal and is actually testing the desired capability. For me to raise the score above threshold, *I would like the authors to engage with this point a little more than what's currently written in the paper.* \n\n2. I think this is a more minor weakness. There are a very large number of tasks and each of them are described independently. However, there isn't much discussion on what features are shared across the different tasks. I think the paper would be a lot more stronger if the authors were able to pick out specific features of spatial cognition that are salient across many of these tasks and identify which tasks test which features. If the paper was structured more like: \"we identified 4 features of spatial cognition common to many tasks in the literature: A, B, C, and D. Here are some tasks from the literature. The mental rotation task tests A and B, perspective taking tests B and C, etc.\" Currently the shear number of tasks (while impressive) is overwhelming, and looking at performance across all of them is harder to interpret. I think this would make evaluation on SPACE more digestible, because you'd be able to diagnose which aspects of spatial cognition your model succeeds at and which aspects your model fails at.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oJxa0WwoCP", "reviewer": "Reviewer_HHQw", "summary": "The paper a comprehensive benchmark SPACE that systematically evaluates spatial cognition in large language models (LLMs) and vision language models (VLMs). SPACE divides into large-scale spatial cognition tasks and small-scale spatial cognition tasks. Large-scale spatial cognition is at the environment level and the goal is to test whether LLMs and VLMs can build spatial representations to reason about and navigate in the environment. Small-scale spatial cognition tasks test models' ability to perceive, imagine, and mentally transform objects or shapes in 2D and 3D scenarios. The experimental results on frontier models (e.g. LLMs, VLMs) show that these frontier models has much lower capability in spatial cognition when compared to humans, and suggest frontier models that have acquired advanced skills may have a fundamentally different form of intelligence from humans and animals because of the failure in basic spatial cognition.", "questions": "**Questions**\n- The Discussion Section mentions that spatial cognition is considered to be a prerequisite for higher cognition and frontier models can acquire tasks requiring higher cognition like olympiad geometry problems, but lack basic spatial cognition as indicated in the results. Is there any study in psychology or cognitive science show that basic spatial cognition is related to solving geometry problems? \n\n- Trinh et al [1] trained the neural language model for geometry theorem proving from scratch using their large-scale synthetic data to guide the symbolic deduction engine. Therefore it seems not very clear that whether their model can perform better on spatial cognition tasks with language-heavy prompting because their model may be specialized to theroem proving while lacking language understanding and generation skills when comparing to state-of-the-art language models.\n\n*References*\n\n[1] Trinh, Trieu H., et al. \"Solving olympiad geometry without human demonstrations.\" Nature 625.7995 (2024): 476-482.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**Originality**\n- While there's an increasing interest in spatial cognition for frontier models, there are not enough datasets to evaluate them systematically. The proposed benchmark SPACE is novel in terms of its comprehensiveness and diversity in tasks.\n\n**Quality**\n- SPACE is a very comprehensive and diverse benchmark. The large-scale spatial cognition tasks cover 5 tasks. The small-scale spatial cognition tasks cover 10 tasks. All 15 tasks are based on prior studies in cognitive science, which ensures plausibility in psychology and cognitive science domains. The input of the tests can be pure language or multimodal, which makes the benchmark suitable for evaluation on LLMs and VLMs. All experiments have human performance as the reference, which makes it easier to compare frontier models with humans.\n\n**Clarity**\n- The paper is well-written and easy to follow. Figures are helpful for understanding each task. The gray coloring of the results indicating they are below 50% of human performance makes the results very interpretable.\n\n**Significance**\n- This paper is valuable to the research community of large language models since it reveals the shortcoming of basic spatial cognition for these frontier models. Additionally, it could make the research community in both artificial intelligence and cognitive science rethink about the difference in intelligence between humans and these models. More specifically, it may indicate only casual language modeling will not lead to the human intelligence without sufficient embodiment in the training process. On the other hand, the results show that while these frontier models are surpassing humans on many tasks, but still lack basic cognition on simple tasks. These findings are insightful as researchers can study (1) How do models skip basic cognition to reach higher cognition? (2) Does skipping basic cognition make the frontier model unreliable and untrustworthy?", "weaknesses": "**Lack of experiments to demonstrate prompts can be understood by frontier models**\n- Most results are around chance level accuracy. There are two possible conclusions: \n1. Models have weak spatial cognition. \n2. Frontier models have better spatial cognition internally, but it doesn't show externally because of the unfamiliarization of the multimodal and text-only prompt formats.\n\nIt would be more rigorous if there are some experiments show that these formats of prompting can be understood by these frontier models (e.g. achieving high accuracy in some easier tasks with the same format of prompting).\n\n**Unclear use of Chain-of-Thought (CoT) techniques in prompting**\n- Chain-of-Thought (CoT) prompting techniques [1] are known for improving language model in solving tasks as it allows LLMs to have some reasoning steps before arriving at the final answer. CoT techniques are not mentioned in the main paper, except \"Think step by step\" prompts in the appendix, but it is unclear which tasks use the CoT, which doesn't. If some tasks use CoT, it should be listed separately to the results table as a comparison to investigate whether reasoning steps strengthen the frontier models.\n\n*References*\n\n[1] Wei, Jason, et al. \"Chain-of-thought prompting elicits reasoning in large language models.\" Advances in neural information processing systems 35 (2022): 24824-24837.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)