summarize the journal below:


Frontier models have achieved impressive performance in mathematics, coding, general knowledge,
and commonsense reasoning (Hendrycks et al., 2021a;b; Chen et al., 2021; Sakaguchi et al., 2021;
Yue et al., 2024). This remarkable progress has inspired characterizations of frontier models as possessing the intelligence of a smart high schooler and predictions of the imminent arrival of superintelligence (Aschenbrenner, 2024). These characterizations are often underpinned by the premise
that competence (or even mastery) in some aspects of cognition is symptomatic of broad cognitive
competence. This is not self-evident. To quote Brooks’s first law of artificial intelligence, “When an
AI system performs a task, human observers immediately estimate its general competence in areas
that seem related. Usually that estimate is wildly overinflated.” (Brooks, 2024).
Our work focuses on spatial cognition, a foundational form of intelligence that is present in a broad
spectrum of animals including humans (Marshall & Fink, 2001; Waller & Nadel, 2013; Mallot,
2024). Spatial cognition refers to the ability of animals to perceive and interact with their surroundings, build mental representations of objects and environments, and draw upon these representations
to support navigation and manipulation. Decades of research in animal cognition have characterized the spatial cognition of rats, bats, dogs, chimpanzees, wolves, and humans (Tolman, 1948;
Menzel, 1973; Peters, 1974; Gillner & Mallot, 1998; Marshall & Fink, 2001; Tommasi et al., 2012;
Geva-Sagiv et al., 2015). Human infants already possess rudimentary spatial cognition, which subsequently improves along developmental schedules that have been characterized (Blades & Spencer,
1994; Newcombe, 2000; Vasilyeva & Lourenco, 2012). Spatial cognition is known to underpin more
advanced cognitive abilities (Kozhevnikov et al., 2007; Newcombe, 2010; Young et al., 2018).
The emergence of spatial cognition has been linked to embodiment (Smith & Gasser, 2005; Jansen
& Heil, 2010; Frick & Mohring, 2016), without which the development of spatial cognition may ¨
be impaired (Foreman et al., 1990; Anderson et al., 2013). However, frontier models are typically
trained in a disembodied manner on corpora of text, images, and video. Does spatial cognition
emerge in disembodied frontier models? To study this question systematically, we develop SPACE,
a benchmark that builds on decades of research in cognitive science. Our benchmark comprises two
broad classes of tasks, covering large-scale and small-scale spatial cognition (Hegarty et al., 2006;
Meneghetti et al., 2022; Newcombe, 2024). The tasks are schematically illustrated in Figure 1.
Large-scale spatial cognition has to do with a model’s ability to understand its surroundings. In
large-scale spatial cognition tasks, the model is familiarized with an environment and is then asked
to estimate distances and directions to landmarks, sketch a map of the environment, retrace a known
route, or identify a shortcut to the goal. Small-scale spatial cognition has to do with a model’s ability
to perceive, imagine, and mentally transform objects in two or three dimensions. Together, largescale and small-scale tasks evaluate core cognitive abilities such as spatial perception, visualization,
selective attention, and visuospatial memory.
We design text-based and image-based presentations to evaluate both language-only and visionlanguage models (LLMs and VLMs, respectively). Our results indicate that contemporary frontier
models have not yet reached competency – let alone mastery – in spatial cognition. On key largescale spatial cognition tasks, frontier multimodal models perform near chance level, even when
presented with an allocentric (map) view of the environment. The strongest models exhibit much
better performance on some small-scale tasks, especially with purely textual presentations via character arrays, but perform near chance on image-based tasks such as mental rotation (Vandenberg &
Kuse, 1978), perspective taking (Kozhevnikov & Hegarty, 2001), maze completion (Lacroix et al.,
2021), or the classic Minnesota Paper Form Board Test (Likert & Quasha, 1941; 1969).
2 RELATED WORK
Spatial cognition. Spatial cognition is a branch of cognitive science that seeks to understand how
humans and animals perceive, interpret, represent, and interact with objects and environments (Marshall & Fink, 2001; Landau, 2002; Waller & Nadel, 2013; Mallot, 2024; Newcombe, 2024). This
involves the perception of object sizes, shapes, and scales, as well as the relationships between objects and landmarks in the environment (including location, distance, direction, and orientation).
Spatial cognition is broadly divided into two categories: large-scale and small-scale (Hegarty et al.,
2006; Jansen, 2009; Meneghetti et al., 2022; Newcombe, 2024). Large-scale spatial cognition refers
to the ability to build spatial representations of environments and use them effectively for navigation and spatial reasoning. Large-scale spatial cognition tasks typically involve egocentric spatial
transformations, where the viewer’s perspective changes with respect to the environment while the
spatial relationships between parts of the environment remain constant (Wang et al., 2014). Small
scale spatial cognition refers to the ability to perceive, imagine, and mentally transform objects or
shapes in 2D or 3D. This is typically evaluated using paper and pencil tasks that require allocentric
spatial transformations of objects and shapes (Wang et al., 2014). While large-scale spatial cognition has been demonstrated in a wide range of animals (Tolman, 1948; Menzel, 1973; Peters, 1974;
O’Keefe & Nadel, 1978; Gillner & Mallot, 1998; Richardson et al., 1999; Geva-Sagiv et al., 2015;
Toledo et al., 2020), the study of small-scale spatial cognition is specific to humans.
Benchmarks for large multimodal models. The recent successes of multimodal models (OpenAI,
2024; Li et al., 2024a; Reid et al., 2024) have been facilitated by large-scale training on text and
multimodal corpora (Rana, 2010; Together Computer, 2023; Chen et al., 2023; Laurenc¸on et al.,
2023; Gadre et al., 2023), followed by tuning on human preferences (Liu et al., 2023a; Awadalla
et al., 2024; Ouyang et al., 2022; Rafailov et al., 2023). The remarkable advances in the capabilities
of these models inspired a variety of benchmarks that evaluate their performance. Early multimodal
benchmarks consisted of single-task datasets such as visual question answering (Antol et al., 2015;
Goyal et al., 2019; Marino et al., 2019) and image captioning (Chen et al., 2015). However, due to the
limited scope of early datasets and concerns regarding potential test-data leakage, newer benchmarks
use diverse collections of tasks (Fu et al., 2023; Yu et al., 2024; Liu et al., 2023b; Yue et al., 2024; Lu
et al., 2024; Ying et al., 2024). While these datasets primarily focus on image understanding, newer
datasets that emphasize spatiotemporal reasoning have been proposed for video (Li et al., 2024b; Fu
et al., 2024a; Majumdar et al., 2024).
Recent studies highlight a number of shortcomings of frontier multimodal models (Moskvichev
et al., 2023; Tong et al., 2024; Chen et al., 2024a; Fu et al., 2024b). One such shortcoming is that
models may not perceive the image in detail, often missing fine-grained details or ignoring the image
entirely (Chen et al., 2024b; Guan et al., 2024; Tong et al., 2024). HallusionBench proposes a new
dataset of image pairs, where tiny edits are made from one image to another that change the answer
to the question (Guan et al., 2024). MMVP identifies issues with CLIP-based pretraining of visual
encoders, which make current models blind to certain visual patterns, and proposes a benchmark of
CLIP-blind image pairs where the same question has opposite answers (Tong et al., 2024). MMStar
shows that many questions in multimodal benchmarks can be answered correctly without the image
and proposes a new split of existing benchmarks that addresses this issue (Chen et al., 2024b).
Another shortcoming of existing models is their lack of spatial perception and reasoning (Chen et al.,
2024a; Cheng et al., 2024). SpatialVLM proposes a VQA dataset that requires answering questions
about relative spatial arrangements and metric relationships (Chen et al., 2024a). SpatialRGPT
further includes region-level understanding (Cheng et al., 2024). ‘Perception test’ aims to overcome
shortcomings of standard video datasets by creating a diagnostic dataset where participants record
videos while following complex scripts depicting interesting events (Patraucean et al., 2023). It
evaluates fundamental perceptual skills (memory, abstraction, intuitive physics, and semantics) and
various types of reasoning.
Another line of work considers skill acquisition (the ability to learn a skill and apply it to new scenarios). Prior work has studied this using visual analogical reasoning (Chollet, 2019; Moskvichev
et al., 2023; Yiu et al., 2024). The ARC dataset contains samples consisting of a few examples of
abstract grids and their transformations and one or more test inputs (Chollet, 2019). The objective
is to understand the transformation performed using the examples and apply it to test inputs. The
transformations have been further organized into specific concepts with varying degrees of difficulty
in the ConceptARC dataset (Moskvichev et al., 2023). Inspired by ARC and developmental psychology, the KiVA dataset studies visual analogies in the context of visually realistic 3D shapes with
concepts like transformations in color, size, rotations, reflections, and counting (Yiu et al., 2024).