REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories
Authors: Jacob Thompson
Emiliano Garcia-Lopez
Yonatan Bisk
Venue: COLM 2025 (2025)
Keywords: Multimodal Reasoning
Embodied AI
VLMs
Spatial Reasoning
Object Permanence
Visuospatial Representation
Large Language Models (LLMs)
Egocentric Vision
Video Understanding
Evaluation Benchmarks
Synthetic Environments
Long-horizon Reasoning
Numerical Tracking
Temporal Ordering
Scene Understanding
LLM Limitations
TL;DR: We introduce REM, a benchmark revealing that current multimodal language models lack fundamental abilities in spatial reasoning, object permanence, and tracking objects over changing viewpoints.

Abstract: Humans build viewpoint-independent cognitive maps through navigation, enabling intuitive reasoning about object permanence and spatial relations. We argue that multimodal large language models (MLLMs), despite extensive video training, lack this fundamental spatial reasoning capability, a critical limitation for embodied applications. To demonstrate these limitations and drive research, we introduce REM: Reasoning over Embodied Multi-Frame Trajectories, a benchmark using controllable 3D environments for long-horizon embodied spatial reasoning. REM systematically evaluates key aspects like object permanence/distinction, spatial relationships, and numerical tracking across dynamic embodied viewpoints. Our evaluation shows that the best-performing current models exhibit promising overall performance, but become increasingly unreliable at even moderate complexity levels easily handled by humans. These findings highlight challenges MLLMs face in developing robust spatial representations from sequential visual input. Consequently, REM provides targeted metrics and diagnostics to foster improved spatial understanding in future models.

Published as a conference paper at COLM 2025
REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories"
Jacob Thompson, Emiliano Garcia-Lopez & Yonatan Bisk Department of Computer Science
Carnegie Mellon University
Pittsburgh, PA 15213, USA (jbthomps, egarcial, ybi sk (oriisk@@@nd @andrew. www. edu
Abstract
Humans build viewpoin--ndepeedeet cognitive maps through navigation, enabling intuitive reasoning about object permanence and spatial relations. We argue that multimodal large language models (MLLMs), despite exten- sive video training, lack this fundamental spatial reasoning capability, a critical limitation for embodied applications. To demonstrate these limi- tations and drive research, we introduce REM (Reasoning over Embodied Multi-Frame Trajectories), benchmark using controllable 3D environments for long-horizon embodied spatial reasoning. REM systematically evaluates key aspects like object permanence/ /diisiicciinnnn spatial relationships, and numerical tracking across dynamic embodied viewpoints. Our evaluation shows that the best-performing current models exhibit promising overall performance, but become increasingly unreliable at even moderate complex- ity levels easily handled by humans. These findings highlight challenges MLLMs face in developing robust spatial representations from sequential visual input. Consequently, REM provides targeted metrics and diagnostics to foster improved spatial understanding in future models.
1 Introduction
When navigating familiar campus, park, or neighborhood, we rarely consult external maps or need explicit directions. Instead, we rely on sophisticated mental representations built through experience. These three-dimensional cognitive maps (Tolman, 1948; O'Keefe & Nadel, 1978) function as accessible spatial databases that persist independently of our current viewpoint. Our mental maps encode precise spatial relationships ("standing near the main gate, the statue is the left of the library")) and allow us to plan optimal routes through areas currently out of sight ("cutting through the science building will save time"). This remarkable ability to maintain and interrogate comprehensive spatial representations across changing viewpoints even enables us to answer difficult questions like "How many blue recycling bins are on campus?" through explicit mental enumeration, drawing on our persistent cognitive model rather than immediate visual perception. This fundamental spatial reasoning capability, which humans develop naturally, represents a critical challenge for current AT systems attempting to understand embodied environments.
Multimodal Large Language Models (MLLMs) have emerged as powerful tools, demonstrat- ing impressive capabilities on static image and video understanding tasks. Their ability to process and reason about visual and textual information makes them promising candidates for robotics and embodied AI applications. However, success in complex, dynamic envi- ronments requires more than recognizing objects or actions in isolated frames; it demands robust embodied spatial reasoning. This entails building and maintaining persistent inter- nal models of 3D relationships and object identities even as they move in and out of view during navigation.
to human cognitive maps -that accurately encode spatial
* Dataset and Code:
RRM REM
1

---

Published as a conference paper at COLM 2025
there?
O
O
A
Object Comparisons
A
A
O
Touns
black
A
O
A
equal
Temporal Ordering: Did see the purple cone before, after or at the same times wen scuare?
Fame 6O
O
Frane
Answer: before
Relative Posttioning:
Ramn 30
black sphere the laft right the green sphane?
Trama 00
Scene Plot
Egocentric Images
Q&A Pairs
Figure 1: REM at glance. Left: top-down plot showing object distribution and camera trajectory. Center:
egocentric views from selected frames, simulating an agent's perception during navigation. Right: example question-answer pairs that test different aspects of spatial reasoning: counting, comparison, temporal ordering, and left/ right relative positioning.
To investigate the extent to which current MLLMs possess these capabilities, we introduce REM (Reasoning over Embodied Multi-Frame Trajectories). REM a novel benchmark specifically designed to evaluate embodied spatial reasoning using multi-frame visual sequences simulating egocentric navigation within synthetic 3D environments of controlled complexity. Unlike benchmarks focusing solely static images (e.g,, Goyal et al., 2017; Hudson & Manning, 2019; Liu et al., 2024a) or complex, uncontrolled video scenarios (e.g., Mangalam et al., 2023; Li et al., 2024a), REM allows for systematic probing of fundamental spatial understanding across changing viewpoints. It assesses critical capacities including object permanence (tested via numerical counting across frames), spatial relationships (tracking left/ /right positioning), temporal ordering (understanding appearance sequence), and the ability to integrate visual context with movement history for object identification.
Our evaluation using REM reveals significant shortcomings in state-of-the-art MLLMs. While newer models show impressive capability in reasoning over simpler scenes, shown in our 'baseline' dataset, performance degrades rapidly with increased complexity. Furthermore, models fundamentally struggle with object permanence and distinction, particularly under challenging viewpoint shifts, as demonstrated in our 'Full Rotation' experiments where performance typically collapses. These findings underscore the specific, deep challenges MLLMs face in developing robust spatial representations from sequential visual input, highlighting the benchmark's utility in driving targeted improvements for embodied Al.
2
REM: Reasoning over Embodied Multi-Frame Trajectories
Summarized in Table 1, we introduce REM, set of three Blender-generated egocentric datasets designed to quantitatively evaluate multimodal large language models' abilities in embodied visuospatial reasoning, specifically focusing object permanence and distinc- tion, spatial relationships, and accuracy counting and numerical object comparisons. Each dataset consists of trajectories composed of sequential egocentric image frames. Crucially, the discrete camera action taken between consecutive frames (e-g., 'move forward 1m', 'rotate right 15")) is explicitly provided alongside the image sequence input to the model,
2

---

Published as a conference paper at COLM 2025
simulating an agent aware of its own movements. These trajectories occur within simple syn- thetic environments containing objects of visually distinct shapes (cuboids, spheres, cones) and colors (brown, yellow, red, green, blue, purple, black, orange).
These environments are intentionally made to be simple, with easily distinguishable object shapes and colors, REM aims to diagnostically pinpoint failures and performance scaling behavior in visuospatial reasoning and memory, not object detection or visual distinction.
Single Frame, and Full Rotation. The Baseline dataset benchmarks general capabilities across carefully varied task complexities; Single Frame isolates single-image counting ability control; and Full Rotation specifically probes object permanence/ distinction and contextual reasoning under challenging view- point changes.
REM consists of three Mitasets-Baaseliie
2.1 Baseline dataset
We introduce our baseline dataset the foundation of REM, designed to comprehensively evaluate MLLM performance across four visuospatial reasoning tasks. contains nearly 50,000 question-answer pairs distributed across over 3,000 embodied image-action trajecto- ries. T'hsse trajectories systematically vary three key dimensions:
Trajectory Length:
Sequences contain either 2, 4, 8, 16, 32, or 64 image frames with associated actions, to evaluate how performance scales with increasing context length and scene views.
Scene Congestion:
Environments contain varying object densities, with either 8, 16, 24, 36, or 48 total objects test performance under different visual complexity conditions.
Object Duplication: Scenes range from all-unique objects to mostly duplicated objects (as few as two distinct types), testing identical object individuation and tracking.
Trajectory lengths are distributed uniformly, while trajectory object densities and total duplicate counts can be found in Figure 10 of the Appendix. The baseline dataset includes four primary question categories, meant to probe different aspects of visuospatial reasoning:
Object Counting: "How many blue objects are there?"
Comparison: "Are there more red cones, spheres, or equal?"
Relative Positioning: "Is the green sphere to the left right of the blue cuboid?"1
Temporal Ordering: "Did we see the purple cone before, after, same time as the orange cuboid?"
Each trajectory is automatically generated to include a variety of movement patterns, with randomized sequences of both forward movements and rotations. We vary both the number of consecutive forward movements before rotating (randomly left or right) and how many consecutive 15 rotations occur in a single direction, ranging from single 15s turns to complete 180 (or greater) viewpoint changes, as permitted by the remaining trajectory length. This design ensures that for longer trajectories, models cannot rely solely on tracking objects through small, incremental camera changes, but must maintain object representations across significant viewpoint shifts where objects completely leave and re-enter the field of view. An example length-4 trajectory is depicted in Figure 2.
2.1.1 Mini-Baseline dataset for Human Comparison
We also sample representative Mini-Baseline subset (18 trajectories; 154 QA pairs) from the larger dataset for a human performance comparison. Participants were given unlimited time, provided physical scratchpad, and used an interactive interface that allowed them to use arrow keys to freely navigate images in the trajectory, similar to a reasoning model
Importantly, this question is only asked in trajectories where objects maintain their left/ left/right positioning throughout all appearances in the trajectory.
3

---

Published as a conference paper at COLM 2025
Rgh
Forward
15" Right
Figure 2: Example length-4 trajectory from the baseline dataset. Models receive the sequence of egocentric visual frames and the corresponding discrete actions ('159 Right', 'lm Forward', '159 Right') taken between frames. Evaluating performance on such sequences tests the model's ability to integrate visual perception with known movement for spatial reasoning across changing viewpoints.
repeatedly accessing its context while producing reasoning tokens. Mini-Baseling trajectory statistics can be found in Figure 11 of the Appendix.
2.2 Single Frame dataset
The Single Frame dataset, consisting of 350 trajectories and approximately 1,300 QA pairs, isolates visual counting ability by eliminating frame-to-frame tracking requirements. This control condition helps disentangle whether counting errors in sequential settings originate from fundamental perceptual limitations, from failures in maintaining object identity across frames, mixture of both. Unlike the baseline dataset, we include only object counting questions, providing comparative baselines for our multi-frame evaluations. Dataset object and total duplicate statistics can be found in Figure 9 of the Appendix.
2.3
Full Rotation dataset
The Full Rotation dataset, containing 100 trajectories and approximately 2,400 QA pairs, challenges models with a trajectory consisting of 360 rotation in a cluttered scene. At the 180 mark, the scene deliberately mirrors the initial 0 view: while the visual arrangement appears identical, most objects are different, with 1-2 target objects intentionally duplicated. Importantly, the rotating camera maintains continuous stream of object views, testing whether MLLMs distinguish identities through integrated contextual and movement cues or use simpler heuristics like spatially invariant attention aggregation. An example is illustrated in Figure 3.
2.4 Automated Question-Answer (QA) generation and verification
In REM, each trajectory is automatically generated with glender-provineg per-frame ground truth object annotations, specifying color-shape identities and left-to-right spatial position- ing. We then construct detailed data dictionaries recording object counts by color, shape, and color-shape combinations, along with the specific frames where each object appears.
We automatically generate question-answer pairs using a template system for each question type. For left/ left/right relative positioning and temporal ordering questions, we randomly select two color-shape entities (e.g., "red cube," "blue sphere") and consult our ground truth annotations to determine correct answers. For counting and numerical comparison questions, we randomly select entities (which may be specific color-shape combinations, colors, or shapes) and reference our ground truth counts.
For evaluation, we employ keyword-basee based verification approach that identifies terms uniquely associated with correct answers. For instance, in counting questions, we extract the first numerical value. For positioning questions, we search for single directional indicator ("left" /"right") if both are provided, the answer considered wrong per our system prompt. For temporal questions, look for single sequence markers ("before"/" /"after").
4

---

Published as a conference paper at COLM 2025
A
A
180" Rotetion
O
O
O
160 Rotation
A
A
Scene Plot
Egocentric Images
Figure Full Rotation dataset example scene: (left) top-down scene layout showing object positions, with the camera at the origin, (top-right) view at 09, and (bottom-right) view after 180 rotation. Note the red sphere is intentionally duplicated between views, while other objects occupy identical spatial positions but are visually distinct entities. Peripheral objects at the top and bottom of the scene layout maintain visual continuity during camera transitions, preventing empty frames with movement ambiguity.
To ensure maximum fairness to models, we exclude any question where our verification system cannot identify unambiguous correctness markers (note this is exceedingly rare). A detailed diagram depicting the dataset generation and verification pipeline, as well as the prompting format, are provided in Figures 12 and 13 of the Appendix.
Property
Baseline
Single Frame
Full Rotation
Num. Trajectories
3,119 (18)
350
100
Total QA Pairs
47,019 (154)
1,289
2,424
Trajectory Length(s)
2, 8, 16, 32, 64 (6,
1
24
Object Count
8-48
24 55
24
Duplicate Count
0 46
0 20
1-2 2
Purpose
General Capabilities
Single-frame Counting
Object Distinction
Table 1: Comparative summary of the REM component datasets (Baseline, Single Frame, Full Rotation), detailing differences in dataset size, trajectory lengths, scene complexity, and evaluation purposes. Mini-Baselimg dataset for human evaluation shown in parentheses.
Dataset evaluation and results
3.1 Baseline results overview
We evaluate seven multimodal models: OpenAl o3 and GPTT-o, three from the Gemini family (2.5-Pro, 1.5-Pro, 1.5-Flash), Nova-Lite-vl, and Llama-3.2-11B. The new SOTA rea- soning models (o3, 2.5-Pro) significantly outperform the older non-reasoning models, with the lightweight proprietary Nova-Lite-vl and open-source Llama-3.2-11B performing only slightly above chance. Note Gemini-2.5-Pro is only tested the Mini-Baseling

---

Published as a conference paper at COLM 2025
o3 performs the best on the full Baseline with impressive overall performance of 80.0% but with the exception left/ rright relation questions, performs noticeably worse than the near-perfect human baseline. Notably, o3 comparatively struggles counting and numeri- cal comparison tasks, which require aggregating information from multiple objects across multiple images while respecting object permanence and distinction. Á more thorough analysis of individual question performance and scaling laws follows in Section 3.2.
Question Metrics
Overall
Num. Comparison
Left/Right Rel.
Temp. Ord.
Counting
Full Count
47,019
15,580
1,576
14,304
15,559
Mini Count
154
39
38
37
40
Random Chance
33.3
50.0
33.3
Models
Full Baseline (Accuracy %)
o3
80.0
78.3
92.3
88.4
60.9 (35.7)
GPT-40
61.7
57.9
61.4
73.0
(24.4)
Gemini-1.5-Pro
59.6
54.3
67.7
69.0
47.4 (14.1)
Gemini-1.5-Flash
58.2
59.5
59.5
65,8
47.8 (13.8)
Nova-Lite-vl
38.7
38.1
51.3
43.4
21.9 (7.3)
Llama-3.2-11B
37.3
31.6
51.5
45.7
20.2 (7.0)
Models
Mini Baseline (Accuracy %)
Human Average
97.8
96.3
99.3
98.2
97.5 (97.7)
o3 Cemini-2..-Pro
80.1
64.1
100.0
83.8
72.5 (59.1)
79.4
74.0
78.9
70.0 (54.5)
Cemini-l15-Pro
58.4
51.3
71.1
51.4
60.0 (36.4)
CPT-40
57.1
56.0
57.9
55.0 (31.8)
Table 2: Top: dataset metrics. Middle: evaluation the full Baseline. Bottom: evaluation the mini baseline sample. Reasoning models separated from non-reasoning models. Overall performance shown in descending order. Counting question performance with ground truth > 2 shown in parentheses.
Baseline performance breakdown
1.0
Fraction Cornect
0.7
04
10
Observed Object
Observed Duplicate Count
Trajectory Length
Numerical Object Comparison
Temporal Ordering
LeftRRight Posttioning
Figure 4: o3 QA performance across three scaling factors: (a) observed object count, (b) observed duplicate count, and (c) trajectory length. Curves show average correctness for Numerical Object Comparison, Temporal Ordering, and Left/ Positioning tasks.
Figure demonstrates basic scaling laws of non-numerical questions with overall object count, total number of observed duplicates, and trajectory length. For simplicity, we limit our analysis to o3, although similar scaling laws are observed with other models.
Object Comparison. We observe greatly decreasing performance with overall scene conges- tion (both observed object count and duplicate count), with the most complex scenes (36

---

Published as a conference paper at COLM 2025
viewed objects) dropping below 60% overall accuracy. Similarly, moderate decrease in performance is also observed with increasing trajectory length beyond 4 images. In both cases, this is likely due to decreased signal-to-noise ratio (occlusion, scene diversity) of the two objects being compared, and greater need to sift through the scene for the correct objects.
Additionally, as shown Figure 5, we notice a significant decrease in model performance as targeted object counts become more similar, with 0 difference in target object quantities resulting in only 66% accuracy. Qualitatively comparable performance drop-off with GIT--4o and Gemini suggests models are using kind of "fuzzy counting" heuristic that begins to fail with decreasing difference in target object quantities.
62
620
62
Otfonece Target Oged Quartrv
Figure 5: 03 numerical object comparison count accuracy vs. the difference in target object counts. Includes 95% confidence interval. Random 33% baseline provided in red.
Figure 6: temporal ordering question per- formance as a function of observed duplicate and total object counts. Different colored lines bin respective object counts, with ran- dom 33% baseline provided in red.
Temporal Ordering, We initially observe strong reduction in performance with increasing scene congestion (object count), but unlike the other question types, not with trajectory length. Interestingly, we also see a strong positive correlation with maximum duplicate counts, with the highest counts reaching saturation. The explicit relationship between duplicate count and overall object count is more directly observed in Figure 6, which shows that even the most congested scenes, increasing duplicate count can lead to performance saturation. This is likely due to increasing duplicate count resulting in order-preserving questions being more likely to target the duplicates, which have an increased signal-to-noie ratio (less attention competition) in the scene.
Left/Right Orientation. In Figure 4, we notice left/ questions scaling strongly with scene congestion and moderately with trajectory length. An intuitive potential explanation for the scene congestion scaling is provided by increased occlusion, object attention compe- tition, and more non-target objects appearing between the targets. Peerformacce degradation with trajectory length is similarly intuitively explained by increasing scene complexity.
Counting Questions. Counting question performance is predominantly dependent on target object ground truth count. As shown in Figure 7, o3 progressively undercounts objects as ground truth increases. In fact, counting questions with target object class ground truth > 2 have an overall accuracy of only 35.7%.
Additionally, Figure demonstrates that, independent of ground truth count, increasing trajectory length does not seem to consistently affect undercounting. This is perhaps counterintuitive, as longer trajectories include more diverse viewpoints of persistent objects, which would naively result in relative overcounting.
Furthermore, we observe in Figure 8 that while the ground truth number of target objects seen throughout the entirety of trajectory is typically greater than the number observed by models in any single frame, the amount predicted by o3 closely tracks the latter value. Based these observations alone, it is not clear whether counting considering overall
7

---

Published as a conference paper at COLM 2025
1.0
ligh Bicaa and Nalk wTT cunee medirts
6.0
6.00
Figure 8: o3 counting predictions vs. max- imum number of target objects a sin- gle frame. Includes both single-frame and multi-frame trajectory ground truths, both with 95% confidence interval.
Figure 7: 03 average undercounting fraction as function of target object ground truth count. Different lines indicate different trajec- tory lengths.
trajectory object counts (requiring some notion of object permanence and continuity) or if the model is simply selecting some subset (potentially the maximum object frame) of the trajectory, and returning value correlated with object count in that subset.
is also notable that while absolute counting performance is increasingly poor beyond ground truth counts 0 - 1, o3 does retain strong notion of more vs less objects in trajectory, explaining why object comparison results are superior absolute counting in higher duplicate count (and thus higher target object count) settings.
3.3 Single frame counting results
Similar to baseline counting, o3 exhibits systematic undercounting behavior in single-frame scenarios while maintaining strong correlation with true object counts. This pattern again indicates that although absolute counting accuracy is limited, models retain meaningful notion of relative quantity (Figure 8).
By averaging the fraction of undercounting observed with ground truth > 2 in Figure 8, we see that o3 undercounts with an average prediction-to-actur ratio of 0.65 (compared to 0.73 in multi-frame with same average ground truth), and progressively undercounts worse with increasing ground-trutt -oound-truth values. Notably, we never observe overcounting. This improved undercounting ratio for multi-frame trajectories (approximately 13% higher predictions), suggests that o3 is doing some kind of limited object count aggregation across frames.
single-frame trajectories
3.4 Full Rotation results
Table 3 shows that the best-performing model, OpenAl o3, consistently undercounts in both single and double duplicated object scenarios, with a slightly less severe bias when only single object is duplicated. This result is intuitive: the single-duplicated case features more distinct visual context after the 180 rotation, making the duplicated object scene easier to identify as different than the initial view. However, the dramatic undercounting when the ground truth is two (86% and 90% failure rates for single and double duplication, respectively) reveals critical failure in integrating the complete trajectory. o3 fails to use the explicit motion cues and clear object differences to understand that visually similar configurations at O' and 180 represent distinct sets of objects. Similar failure rates are seen for all other tested models, including GPT-4o and the Gemini-1.5 family.
8

---

Published as a conference paper at COLM 2025
Single Duplicated Object
Two Duplicated Objects
Ground Truth
Undercount
Correct
Overcount
Undercount
Correct
Overcount
0
0.00
1.00
0.00
0.00
0.95
0.05
0.09
0.91
0.01
0.06
0.93
0.01
0.86
0.14
0.00
0.90
0.10
0.00
Table 3: Impact of intentional object duplication counting accuracy in the Full Rotation dataset with OpenAl o3. Counting questions target objects with ground truth counts of either 1, 2.
4 Related work
Introducing MLLMs. Multimodal Large Language Models (MLLMs) typically combine pretrained vision encoders (Radford et al., 2021), (Dosovitskiy et al., 2021) with large language models (LLMs) (Brown et al., 2020), (Chowdhery et al., 2022), (OpenAl et al., 2024), (Touvron et al., 2023), (Vaswani et al., 2023). This integration enables them to process and reason about both visual and textual information, demonstrating strong capabilities in tasks such as visual question answering, image captioning, and generating descriptions grounded in visual input (Yin al., 2024). The potential of these models extends to applications requiring interaction with physical world, including robotics and embodied AI, where understanding dynamic scenes and spatial contexts essential (Driess et al., 2023), (Brohan et al., 2023), (Wang et al., 2023), (Liu et al., 2025).
MLLM Benchmarks for Static Images. A substantial body of work focuses on evaluating MLLMs using single static images. Benchmarks utilizing synthetic scenes like CLEVR (Johnson et al., 2017) assess compositional reasoning and spatial relationships in controlled settings, while those grounded in real-world images, such VQA v2 (Goyal et al., 2017), GQA (Hudson & Manning, 2019), and NLVR2 (Suhr et al., 2019), evaluate understanding single naturalistic visual input. Furthermore, comprehensive evaluation suites including MME (Fu et al., 2024a) and MMBench (Liu et al., 2024a) provide broader assessments across diverse perception and cognition tasks within single-image context. While invaluable for measuring progress in interpreting complex visual scenes, these static evaluations highlight the need for distinct methodologies capable of address- ing the challenges posed by temporal dynamics, viewpoint changes, and the continuous perception-action loop characteristic of embodied agents.
and reasoning based
MLLM Benchmarks for Video and Embodied Spatial Reasoning, To evaluate performance on dynamic inputs, numerous video-based benchmarks have been proposed, targeting various aspects of understanding. Early benchmarks often focused on temporal event com- prehension, action recognition, or causal reasoning within videos (Jang et al., 2017; Lei al., 2019; Yi et al., 2020; Girdhar & Ramanan, 2020), with diagnostic datasets further probing specific temporal concepts (Li et al., 2024b; Liu et al., 2024b). Subsequently, comprehensive benchmarks emerged to assess broader video understanding across diverse content and tasks (Li et al., 2021; Fu et al., 2024b; Li et al., 2024a; Ning et al., 2023; Fang et al., 2024). More recently, reflecting the push towards embodied AI, evaluations have increasingly incorporated embodied perspectives, revealing challenges in areas like long-form egocentric understanding (Mangalam et al., 2023), embodied question answering (Majumdar et al., 2024), and spatial reasoning (including explicit counting) using realistic trajectories from 3D scans (Yang et al., 2024). While these benchmarks effectively identify performance gaps on complex tasks, REM complements them by utilizing a controlled synthetic environment specifically designed to enable detailed analysis of how fundamental factors influence core spatial reasoning performance. By precisely manipulating trajectory length, scene clutter, and object duplication, REM allows for a systematic study of their impact on object permanence (critically tested via counting across views), spatial relationship tracking under viewpoint shifts, and numerical consistency during simulated movement. This focused ap- proach provides targeted diagnostics on where models might fail, offering insights beyond overall performance metrics on complex scenarios.

---

Published as a conference paper at COLM 2025
5 Discussion
We introduced REM, a benchmark leveraging controlled synthetic environments to probe MLLM embodied spatial reasoning across sequential egocentric viewpoints. Our analysis suggests a core limitation: MLLMs struggle to build and maintain stable internal world model that integrates visual perception with explicit discrete egomotion to track distinct object instances over time and viewpoint changes. This fundamental deficit manifests in the models' handling of object identity and quantity across dynamic scenes:
The poor world modeling results in deficient spatiotemporal context integration, meaning models fail to correctly update their internal scene representation using the explicitly provided actions alongside unfolding visual context (Section 3.4). This is best shown in the Fuull Rotation' experiment (Table 3), where models cannot leverage known motion and visual change to disambiguate visually similar scenes. They incorrectly merge distinct object instances seen before and after the 180 turn, demonstrating a failure to ground perception in movement and visual context to form viewpoint-invariant understanding.
This difficulty in individuating objects using spatiotemporal context possibly contributes to the observed poor numerical grounding (Sections 3.2, 3.3). When distinct instances erroneously merged due to context integration or object representation failures, systematic undercounting can naturally follow, extending observed under-counting biases in single- frame perception to multi-frame trajectories.
The impact of this core limitation additionally varies across other specific tasks. 'Counting' directly probes permanence (degrading with ground truth count, Figure 7). 'Numerical Comparison' reflects noisy object quantity representations (failing on small differences, Figure 5). 'Left/Right' reasoning shows decaying relational tracking with distance and clutter (Figure 4). "Temporal Ordering' highlights sensitivity to distinct instance tracking versus overall scene complexity (improving with duplicates, degrading with congestion, Figure 6). Unlike static or uncontrolled video benchmarks, REM's systematic variation isolates these specific failures in dynamic spatial reasoning. For example, REM shows that by simply increasing scene congestion from the Baseline dataset average to 36 viewed objects, and only including counting questions with ground truth > overall SOTA model performance drops from an impressive 80% to under 60%.
This inability to form integrated world models (cognitive maps) is primary bottleneck for deploying MLLMs in embodied Al. Future work must prioritize architectures and training paradigms that explicitly foster robust object permanence and the integration of spatiotemporal context, using targeted benchmarks like REM to measure progress towards spatially grounded intelligence.
Acknowledgments
This work was partially supported by funding from Lockheed Martin Corporation, the Dis- tributed Collaborative Intelligent Systems and Technology (DCIST) Collaborative Research Alliance, and Fujitsu Limited.
References
Anthony Brohan, Noah Brown, et al. Rt-2: Vision-language-cction models transfer web knowledge to robotic control, 2023. URL httpss:/aaaxxi..or////////0//// 15818.
Tom B. Brown, Benjamin Mann, et al. Language models are few-shot learners, 2020. URL https://arxiv argg http::://rxxv..oor/bb//228.. 14165.
Aakanksha Chowdhery, Sharan Narang, et al. Palm: Scaling language modeling with pathways, 2022. URL htt.tttts///aaiiii wwwgg//s///2....00 httppp/////axiii.oo///////..//.
02311.
Alexey Dosovitskiy, Lucas Beyer, et al. An image is worth 16x16 words: Transformers for image recognition scale, 2021. URL http:://arrvv.orr/abb///// 11929.
10

---

Published as a conference paper at COLM 2025
Danny Driess, Fei Xia, et al. Palm-e: An embodied multimodal language model, 2023. URL https://arxiv..
http:://arrxt.orr//bb/22223 wtg./bs///////////// http//bb//333.3333 03378.
Xinyu Fang, Kangrui Mao, et al. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding, 2024. URL RBRLLttp::/aankvv.rr//////// 14515.
Chaoyou Fu, Peixian Chen, et al. Mme: A comprehensive evaluttion benchmark for multimodal large language models, 2024a. URL URRLitttsss//aaxi..o/////////// 13394.
Chaoyou Fu, Yuhan Dai, et al. Video-mme: The first-ever comprehensive evaluation
benchmark of multi-modal Ilms in video analysis, 2024b. URL /tt///tt///ar 2405. 21075.
Rohit Girdhar and Deva Ramanan. Cater: A diagmostic dataset for compositional actions and temporal reasoning, 2020. URL https://arxii http:/ivwv.rgga/b////eea/// 04744.
Yash Goyal, Tejas Khot, al. Making the in vqa matter: Elevating the role of image understanding in visual question answering, 2017. URL 00837.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019. URL //appi// ort./bs///////////// 1902.09506.
Yunseok Jang, Yale Song, et Tgif-qa: Toward spatio-temporal reasoning in visual question answering, 2017. URL
Justin Johnson, Bharath Hariharan, et al. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
Jie Lei, Licheng Yu, et al. Tvqa: Localized, compositional video question answering, 2019. URL httttps//aaxivv wwtpr/iwwwoo/ab/////aeee 9ttp////bbb11900169.6..
Kunchang Li, Yali Wang, et al. Mybench: A comprchensive multi-modal video understand-
ing benchmark, 2024a. URL SBRL.ntpp//arrvv.or///////21...0
Linjie Li, Jie Lei, et al. Value: A hoalliiiakkkeecccmnnnoooooveeeee evaluation, 2021. URL TRLLnttp://arrvvv.r////////1/ 04632.
understanding
Shicheng Li, Lei Li, et al. Vitatecs: A diagnostic dataset for temporal concept understanding 17404.
of video-language models, 2024b. URL
Hao Liu, Wilson Yan, et al. World model on million-length video and language with blockwise ringattention, 2025. URL http:://arxiv.orr//as//2222 08268.
Yuan Liu, Haodong Duan, et al. Mmbench: your multi-modal model an all-around player?, 2024a. URL htttp::/aarxiv.org/ab////////// 20000//0/2200002211...1 06281.
Yuanxin Liu, Shicheng Li, et al. Tempcompass: Do video lims really understand videos?, wrtp:o////b/ 00476.
2024b. URL
Arjun Majumdar et al. OpenEQA: Embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, et Egoschema: diagnostic benchmark for very long-form video language understanding, 2023. URL httpps://arxi.. org. / 2308. 09126.
Munan Ning, Bin Zhu, et al. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models, 2023. URL 16103.
John O'Keefe and Lynn Nadel. The Hippocampus as a Cognitive Map. Oxford University Press, 1978.
11

---

Published as a conference paper at COLM 2025
OpenAI, Josh Achiam, et al. Gpt-4 technical report, 2024. URL http:://arx....og//dass 2303.08774.
Alec Radford, Jong Wook Kim, et al. Learning transferable visual models from natural 00020.
language supervision, 2021. URL
Alane Suhr, Stephanie Zhou, al. A corpus for reasoning about natural language grounded in photographs, 2019. URL https///aaxxvv..aaa//////// .00491.
Edward C. Tolman. Cognitive maps in rats and men. Psychological Review, 55(4):189-208, 1948.
Hugo Touvron, Thibaut Lavril, et al. Llama: Open and efficient foundation language models, 2023. URL
http:/vv..rr//////////e// 13971.
Ashish Vaswani, Noam Shazeer, et al. Attention is all you need, 2023. URL https://arxiv. org/abs/1706.. 03760002226.
Guanzhi Wang, Yuqi Xie, et al. Voyager: An open-ended embodied agent with large 16291.
language models, 2023. URL https: http://///axv..o/abb///////
Jihan Yang, Shusheng Yang, et al. Thinking in space: How multimodal large language models see, remember, and recall spaces, 2024. URL WWRLLtttps//arxxi.orr//abbss 14177.
14171.
Kexin Yi, Chuang Gan, et al. Clevrer: Collision events for video represeetttiio and reasoning, 2020. URL https://arxx wrgg 0142...
Shukang Yin, Chaoyou Fu, et al. A survey on multimodal large language models. National Science Review, 11(12), November 2024. ISSN 2053-714X. doi: 10.1093 00993//sr/nwaet.. URL http://dx.doi.orr www.dd/.oo..og///1/// org/10. 1893/nsr/nwae403.
http://dx.no
12

---

Published as a conference paper at COLM 2025
A
Dataset Statistics
Cannd Cannt-a Seals
00
25
58
15
95.
-2.5
150
17.5
200
222
Figure 9: Key Statistics for the Single Frame dataset. Left to right: Objects seen per frame (mean 17.3, range 7-30); Duplicates seen (mean 5.01, range 0-20); Ground truth counts for counting questions (2 2, log scale), showing long-tailed distribution relevant to undercounting analysis (see Figures 7 and 8). Note the log scale on the rightmost plot.
Oo Lue Bolo
Figure 10: Key Statistics for the Baseline dataset. Left to right: Objects seen per trajectory; Duplicates seen per trajectory; Ground truth counts for number questions (2 2, log scale). Note the long-tailed distribution for counts, also relevant to undercounting analysis (see Figures and 8). Note the log scale on the rightmost plot.
Namber CONica Seas
Total Murour www.oooiaiiaaaaac.. Seer
Gruad VAL
Figure 11: Key Statistics for the Mini Baseline dataset. Left to right: Objects per trajectory; Duplicates seen per trajectory; Ground truth counts for number questions. Note that the eight zero duplicate trajectories contain only left/right questions.
13

---

Published as a conference paper at COLM 2025
B Dataset generation and evaluation Details
Figure 12: Generation and evaluation pipeline for REM. Starting from trajectory param- eters (length, object count, duplicate rate), we synthesize a collision-free scene (sample shapes/ /colors and inject duplicates) and discrete egomotion plan, then render the ego- centric sequence in Blender while logging per-frame object IDs and pixel coverage. The annotations feed cata dictionary that aggregates counts and frame indices, from which we automatically instantiate QA templates (counting, comparison, left/ left/right positioning, temporal ordering). Models are queried with the images, actions, and questions, and rule-based verifier (numeric/keywer checks) scores answers for correctness.
System Prompt
You are an agent walking through an environment in Blender. You will receive a series of images, each taken after taking an action in the environment (either moving straight or turning 15 degrees left/right). You will also receive question that you must answer correctly after seeing all images. You will see objects with shape and color. The possible shapes include cuboid, cone, sphere. The possible colors include red, green, blue, yellow, purple, brown, black, orange. Please answer the question based on the set of images. Answer as concisely as possible, usually only single word. If you're asked about a true/false question, answer with 'yes' or 'no' only. If it's a question where you're asked to compare the number of objects, respond only with whichever object there are more of, or equal, if there are the same number of objects If you're 'r asked to count objects, answer only with the number (as number, not in english) of objects you see. you're asked whether you saw something before, after, or at the same time as another object, answer only with 'before', 'after', or 'same time' only. If the first time you see an object is in an image before another object, it comes before (and the other comes after). If two objects appear in the same frame together for their first viewing, its same time
Figure 13: System prompt and message format. Each model query contains two messages: the fixed system prompt (above), then a single user message whose content is, in order: the question text, the movement annotations, and all trajectory images, each resized to 960 x 640.
14


Reviews:
[{"review_id": "Nb0616CMG9", "reviewer": "Reviewer_iEfL", "summary": "This paper constructs a synthetic 3D dataset consisting of simple 3D shapes and the camera moves along a trajectory. Different views along the trajectory are saved, and a whole trajectory consisting multi-view images are used to evaluate MLLMs. They evaluated a few mainstream MLLMs and found their limitations.\n\n=== Update after author response ===\n\nI appreciate the authors put a lot of effort to create this dataset. While with a few limitations, this dataset still provides certain use to evaluate the 3D spatial perception and spatial memory of MLLMs. Therefore I'd like to raise the rating to accept.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It's kind of interesting to see how MLLMs perform under such challenging scenarios.\n2. This dataset may provide motivations for future MLLMs to incorporate long term memory and trajectory-wise consistent object representations.", "reasons_to_reject": "1. There are no object movements. Only the camera moves. This is a limitation in terms of simulating real world scenarios.\n2. The scene is not truly 3D, as all objects are at the same level, although with different heights. It may be more appropriate to be called \"2.5D\". One can argue that in many scenarios such as the street, the scenes are \"2.5D\" as well, however, making the scene multiple layers along the height would make the dataset more challenging and map to some real-world use cases. The authors should point out this limitation, and could view it as a future work.\n3. When too many frames are input to GPT-4o or other proprietary models, the results may not make much sense, because these models are not specifically optimized to take inputs of so many images.", "questions_to_authors": "N/A", "extra_scores": {}}, {"review_id": "Tg6jWNS3oO", "reviewer": "Reviewer_eEuK", "summary": "The paper presents a dataset called REM (Reasoning over Embodied Multi-frame trajectories) and multimodal large language models' shortcoming on spatial relationships, temporal ordering, and numerical tracking across egocentric images and trajectories. For this purpose, they have made three types of dataset: Baseline, Single Frame, and Full Rotation. Baseline benchmark general capacity. They have recorded different image views in a 3D simulation along with the trajectory. For the purpose of MLLMs assessment, they have utilized both multi-view images and trajectory together. Full rotation assess object permanence capability of multi-modal models. And, single frame investigate counting capability of these models. They have demonstrated the performance in the form of accuracy across five recent multi-modal large language models", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Enough models have been evaluated. \n2. Unlike others, the dataset has been captured in a 3D controlled environment", "reasons_to_reject": "1. It would have been better if authors would provide more detailed description on how the image and question pairs have been generated at scale. It is ambiguous to me that how question and images have been generated and make sure the questions have been remained valid to the ground truth \n\n2. Also, \"left of\" and \"right of\" are good aspect of spatial reasoning for evaluation. It is limited and simplified. Broader evaluation elaborates their weakness better and also make it more applicable to the real world situation\n\n3. There is no human evaluation presented in the paper to provide the baseline regarding how the perfoemance between multimodal language models and human differs", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RMoP7dDKra", "reviewer": "Reviewer_2bj9", "summary": "This paper introduces REM, a benchmark designed to evaluate spatial reasoning capabilities in multimodal large language models (MLLMs) within simulated 3D environments. REM offers a controllable and diagnostic Blender-based environment through tasks that test spatial relationships, temporal ordering, and numerical tracking across egocentric trajectories. The authors demonstrate that current state-of-the-art MLLMs struggle with REM, particularly under viewpoint shifts and increased scene complexity.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a new benchmark to probe the spatial reasoning ability of VLMs with sequential visual inputs. REM challenges existing LLMs with controlled environment simulation, calibrated trajectories and multi-dimensional questions. \n- The analysis reveals some interesting findings such as LLMs lack of an internal world model to be aware of its spatial positions or identify the same object across different frames.", "reasons_to_reject": "- The template-based questions may introduce ambiguities in ground truth answers. It is not clear enough to me how the question generation process guarantees the correctness of the ground truth answer without ambiguity. For example, in Fig.1's relative position question, what if the center moves to a new point with a different camera angle where the black sphere appears to the right of the green sphere? How is that avoided in the trajectory sampling or question generation stage? In addition, is object occlusion considered when generating questions? How do the questions guarantee that the involved objects are visible in the frames? \n- While REM comes with good motivation for probing LLM's awareness of its spatial positions, I am unsure if the provided frame inputs are sufficient to justify the argument. In the simulated environment with many similar/duplicated objects without natural background reference, even human beings may not be able to answer the questions correctly. \n- REM is limited to rendered symbolic objects based on limited colors and shapes. It may not reflect LLMs true capabilities when placed in a more realistic environment.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v6i7Yt2sYd", "reviewer": "Reviewer_gAU9", "summary": "This paper introduces REM, a new benchmark to evaluate long-horizon, multi-frame spatial reasoning. The benchmark comprises synthetic 3D scenes generated in Blender using primitive objects like cubes and cylinders. Random camera trajectories are simulated to capture the environment. Four types of question-answer pairs are generated using a template-based system, including object counting, numerical comparisons, relative positioning, and temporal ordering. The dataset includes three variants: (1) a baseline for general spatial reasoning, (2) a single-frame version, and (3) a multi-frame 360° rotation from a fixed location. Several state-of-the-art MLLMs are evaluated on this benchmark. Results show that current MLLMs struggle to build a stable internal world model that integrates spatiotemporal context.\n\n=== Update after author response ===\n\nThe rebuttal addressed some of my concerns including the benchmark construction and evaluation. However, I still have concerns regarding the unrealism of the benchmark and the lack of insights into the methodology. I increased my rating from 4 to 5.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a controllable benchmark for testing spatial reasoning capabilities in MLLMs.\n2. The analysis of GPT-4o’s performance across different question types and dataset variants is comprehensive.", "reasons_to_reject": "1. The benchmark relies on overly simplified, synthetic environments, which limit its realism and applicability to real-world scenarios. No human evaluations are conducted on the benchmark.\n\n2. There are concerns about benchmark construction and evaluation:\n- It is unclear whether the randomly sampled frames fully capture the scene to allow meaningful reasoning.\n- The keyword-based answer evaluation may be flawed. For example, if a model outputs both \"left\" and \"right\", it might be incorrectly considered fully correct.\n\n3. The usage of MLLMs lacks clarity.\n- Prompting strategies are not well-documented or explored.\n- The potential for leveraging spatial depth or camera poses is not discussed.\n- No method is proposed to improve model performance; the paper is purely evaluative.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Four reviewers provided diverging reviews for this benchmark paper. Concerns revolve mainly around the realism of the stimuli and the lack of detail about the methodology of dataset construction and of the evaluation of VLMs on the benchmark. That said, the AC tends to agree that while with faults, the suggested benchmark has its merits for testing spatial relations and could be of use to the community. The authors are encouraged to include the details of the discussion with reviewers in their final manuscript.
Decision: Accept