SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models
Authors: Arijit Ray
Jiafei Duan
Ellis L Brown II
Reuben Tan
Dina Bashkirova
Rose Hendrix
Kiana Ehsani
Aniruddha Kembhavi
Bryan A. Plummer
Ranjay Krishna
Kuo-Hao Zeng
Kate Saenko
Venue: COLM 2025 (2025)
Keywords: spatial reasoning
vqa
multimodal language models
TL;DR: dynamic simulations improve spatial reasoning in MLMs - both for static relationships and on our complex tasks that require reasoning about actions

Abstract: Reasoning about motion and space is a fundamental cognitive capability that is required by multiple real-world applications. While many studies highlight that large multimodal language models (MLMs) struggle to reason about space, they only focus on static spatial relationships and not dynamic awareness of motion and space---i.e. reasoning about the effect of egocentric and object motions on spatial relationships. Manually annotating such object and camera movements is expensive. Hence, we introduce SAT, a simulated spatial aptitude training dataset comprising both static and dynamic spatial reasoning across 175K question-answer (QA) pairs and 20K scenes. Complementing this, we also construct a small (150 image-QAs) yet challenging dynamic spatial test set using real-world images. Leveraging our SAT datasets and 6 existing static spatial benchmarks, we systematically investigate what improves both static and dynamic spatial awareness. Our results reveal that simulations are surprisingly effective at imparting spatial aptitude to MLMs that translate to real images. We show that perfect annotations in simulation are more effective than existing approaches of pseudo-annotating real images. For instance, SAT training improves a LLaVA-13B model by an average 11% and a LLaVA-Video-7B model by an average 8% on multiple spatial benchmarks, including our real-image dynamic test set and spatial reasoning on long videos---even outperforming some large proprietary models. While reasoning over static relationships improves with synthetic training data, there is still considerable room for improvement for dynamic reasoning questions.

Published as a conference paper at COLM 2025
SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models
Arijit Ray,' Jiafei Duan, 2t Ellis Brown, 5t Reuben Tan, 14 Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Ranjay Krishna, 2.3+ Kuo-Hao Zeng,"' Kate Saenko
Bryan A. Plummer,
Boston University, Universiity of Washington, 3 Allen Institute for AI, *Microsoft Research, NNew York University
Simulated SAT Training
Real-world improvements on spatial reasoning
Stock and damomis (recsoning obout office ofego olaect mosion)
Static (Existing Bench
Perspective
Action Consequence
Goal Alm
Static: Where
Counter window? Dynamic any object moved
the motorcydle? SAT (ust: fwo, itying. beside..
lok the digg and lnce the serr the clsg, be onright?
Is turn move
need to the back SAT
picture?
fright
Object Movement
Egocentric Movement
SAT Tuning Improve ments
LLaV--M
SATRee
CV8
BJNK
Sensh
any the olpaeds misorit from to SAT Chain terty
How carmenta aali acene
10
-ATRee OV3
BUNK
VSR 301 5g
Figure 1: We propose Spatial Aptitude Training (SAT), which generates simulated spatial training data that allows us to go beyond simple static relationships in existing datasets. Inspired by cognitive science, we add more challenging dynamic questions that require reasoning about ego and object motion. SAT improves performance on both existing spatial benchmarks and dynamic reasoning on our new real-image benchmark.
Abstract
Reasoning about motion and space is fundamental cognitive capability that required by multiple real-world applications. While many studies highlight that large multimodal language models (MLMs) struggle to rea- son about space, they only focus on static spatial relationships, and not dynamic awareness of motion and space-ie., reasoning about the effect of egocentric and object motions on spatial relationships. Manually annotating such object and camera movements is expensive. Hence, we introduce SAT, a simulated spatial aptitude training dataset comprising both static and dynamic spatial reasoning across 175K question-answer (QA) pairs and 20K scenes. Complementing this, we also construct small (150 image-QAs) yet challenging dynamic spatial test set using real-world images. Leveraging our SAT datasets and 6 existing static spatial benchmarks, we systemati- cally investigate what improves both static and dynamic spatial awareness. Our results reveal that simulations are surprisingly effective at imparting spatial aptitude to MLMSs that translate to real images. We show that per- fect annotations in simulation are more effective than existing approaches pseudo-annotating real images. For instance, SAT training improves a model by an average 8% on multiple spatial benchmarks, including our real-image
LLa VA-13B model by an average 11% and
"equal advising, joint author

---

Published as a conference paper at COLM 2025
dynamic test set and spatial reasoning on long videos-evee even outperforming some large proprietary models. While reasoning over static relationships improves with synthetic training data, there is still considerable room for improvement for dynamic reasoning questions.
Introduction
Reasoning about space and motion is a fundamental cognitive capability that allows humans and animals to survive and operate in the real world (Gardner, 2011; Vasilyeva & Lourenco, 2012; Blades & Spencer, 1994). While simple visual relationships (such left-right) may be trivial to recognize for humans, more dynamic reasoning involving motion such as taking perspectives and reasoning about the effect of ego-motion or object movement requires more cognitive support (Gardner, 2011; Tversky, 2019). Hence, it is not surprising that such "dynamic" spatial reasoning capabilities in children correlate with their aptitude in geometry, physics, and even linguistic reasoning (Taylor & Tversky, 1992; Mallot, 2024).
Despite the promise of multimodal language models (MLMs) as general-purpose intelligent agents (Achiam et al., 2023; Zhu et al., 2023; Liu et al., 2024), recent studies reveal that these models still struggle with spatial reasoning (Chen et al., 2024; Cheng et al., 2024; Tong al., 2024; Fu et al., 2024b). Specifically, MLMs often fail to predict the relative positions of objects in static images a limitation attributed to the scarcity of spatial relationship annotations in their training data. To address this, recent methods (Cheng et al., 2024; Chen et al., 2024) introduce dgeudo-annooations to encode static object relationships in real images (Cheng et al., 2024), which require extensive engineering and refining to scale. Furthermore, many real-world applications require reasoning that extends beyond static object positions. For example, smart glasses and embodied AT applications need to reason about dynamic scenes, where reasoning about the effect of object and camera movements are essential (Duan et al., 2024; Yuan al., 2024a; Liu al., 2025). Generating gpeudo-annotatinn methods for such dynamic settings is non-trivial due to the complexity of predicting action causality on real images.
using existing
We explore the possibility of using simulation data as simple solution to impart both static and dynamic spatial reasoning into MLMs. We propose Spatial Aptitude Training (SAT), an approach to generate spatial question-answer (QA) data without any human supervision. Manually annotating 3D movements is expensive; hence, SAT leverages 22K ProcTHOR (Deitke et al., 2022) scenes composed of iK assets to generate 175K QA pairs. With perfect 3D information and control of the assets, SAT goes beyond static object relationships to questions that require reasoning about egocentric and object movements. Since our data is generated procedurally by composing assets, it can be scaled up arbitrarily. Hence, SAT is programmatically controllable, unlike existing 3D (Brazil et al., 2023; Azuma et al., 2022) or spatial datasets (Cheng et al., 2024), which are also not compositional.
With SAT, we analyze what types of training data improve spatial reasoning. We focus on two types of spatial reasoning data. First, spatial-QA about object relations in static scenes, shown in Fig. (under "Existing Spatial Bench"), to impart reasoning about the relative locations of objects in the scene (e.g., where is object X with respect to object Y?). Next, we evaluate the effect of dynamic spatial tasks, Fig. (shown under "Our SAT Dynamic Bench"). This includes questions about egocentric movement, object movement, allocentric perspective, goal aiming, and action consequences. These dynamic spatial reasoning skills are well-studied in human cognitive development children understand the consequences of self-motion (egg,, the moving room test (Anderson et al., 2013)), track how entities move in scene (Anderson et al., 2013), and can adopt others' viewpoints when planning actions (Brucato et al., 2023; Vasilyeva & Lourenco, 2012; Blades & Spencer, 1994).
We use two widely adopted open-source MLMs, an image-based LLaVA-1.5-13B and as our base models for evaluations. To test static spatial
video-based LLaVA-Video-77 reasoning, we use four contemporary real image benchmarks: CV-Bench (Tong et al., 2024), BLINK (Fu et al., 2024b), Visual Spatial Relations (VSR) dataset (Liu et al., 2023), and GQA-Spatial (Hudson & Manning, 2019a). In addition, we test the effect of SAT on spatial
2

---

Published as a conference paper at COLM 2025
Table 1: Comparison of existing datasets to ours. Our training dataset is synthetic and interactive, allowing us to generate spatial QA and extend to dynamic (e.g,, object movement) reasoning data for free. can also be easily extended to generate new scenes and tasks.
SAT
2D Vision-Language GQA, VG, Obj365
3D Lisian-Langauge Omni3D, ScanQA
Spatial Rel VSR, 2D5V
Spatial QA CVBench, BLINK, Sp VLM, RGPT
(Ours)
2D Annotations
3D Annotations
x
((25D)
X
Static Spatial
/
Dynamic Spatial QA
X
X
X
New Scene/Task Gen
X
Object Interaction
X
X
X
reasoning on longer videos using the recent VSI-Bench (Yang et al., 2024a). Since no dataset exists for dynamic spatial reasoning, we construct a difficult SAT test set on real images.
Challenging conventional wisdom about sim-to-real transfer in model training, we find simulations to be surprisingly effective at imparting spatial intelligence to MLMs. Our simulated SAT training improves the baseline LLaVA-13B model by 11% and LLaVA-video- 7B by 8% (absolute) on average on a wide range of spatial benchmarks, including our challenging dynamic test set-even outperforming some larger closed-source and spatially- tuned models (Achiam et al., 2023; Team et al., 2023). Interestingly, SAT also improves spatial reasoning on long videos (Yang et al., (e24ailespeciallyy in route planning by 4%, showing promise for aiding embodied applications. However, while simple static relationships in existing datasets easy to improve, our best model performs at 62% on our SAT dynamic questions (random chance is
leaving considerable room for improvement.
Our work shows promise that training with data generated using embodied movements and interactions simulators can indeed help instill spatial intelligence in MLMs.
2 Related work
Our work draws inspiration from fundamental schools of thought in neuroscience that suggest spatial intelligence core foundation for most cognitive abilities (Tversky & Suwa, 2009; Mallot, 2024; Duan et al., 2022b).
Spatial understanding benchmarks. We visualize a comparison of our dataset with existing 3D and spatial benchmarks (Table 1). 3D understanding is vital for various computer vision tasks, including segmentation (Jatavallabhula et al., 2023; Hong et al., 2023a; Kerbl et al., 2023; Nie et al., 2020; Kundu et al., 2022; Wang, 2023), object localization (Qi et al., 2019; Sajjadi et al., 2022; Kamath et al., 2021), tracking (Agia et al., 2022; Kurenkov et al., 2023; Li et al., 2022), fine-grained scene captioning (Chen et al., 2021b; Su et al., 2021), open- vocabulary classification (Chen et al., 2020; Hong et al., 2023a; Achlioptas et al., 2020; Shao et al., 2019), and question answering (Ye et al., 2021; Azuma al., 2022; Linghu et al., 2024). Most use 3D scans of real-world data (Hong et al., 2023b), but 3d scans are expensive to recompute for dynamic scenes and most MLMs are trained to input 2D images-hence we focus on 3D spatial reasoning on 2D inputs. In light of potential applications in embodied Al (Duan et al., 2022b), we align our dataset to be compatible with physics simulators, allowing us to go beyond datasets which only leverage image annotations (Caesar et al., 2020) and have static spatial reasoning (Fu al., 2024b; Duan et al., 2022a; Chen et al., 2024; Kamath et al., 2023; Su et al., 2021; Liu et al., 2023; Goyal et al., 2020; Yang et al., 2019; Du et al., 2024).
Synthetic training data. Many have investigated whether synthetic information can boost reasoning in real environments Geng et al. (2024), finding that synthetic images sometimes help in classification (Chen et al., 2021a), semantic understanding (Mishra et al., 2022), correcting biases (Qraitem et al., 2023), and teaching navigation capabilities to embodied AI (Ehsani et al., 2024; Silwal et al., 2024). Inspired by this, our work explores if synthetic data with perfect 2D 3D/3D information
improve spatial reasoning.
Vision-language models. Our task is heavily influenced by the emergence of multimodal foundation models (Radford et al., 2021; Jia al., 2021; Wang et al., 2022a; Yuan et al., 2021;
3

---

Published as a conference paper at COLM 2025
Canenilo
Orject Toueen do of
Which direciin caraae saliing the move?
Ane Ratared
Choose:
Rotate gigh
Tours lod
the socondt
Teleppot X
And. Tooster moved left awey from coreers
Mové object to (pas1
Figure 2: Method of generating our SAT dynamic data: we take actions in 3D simulator and check the 3D locations of assets. We generate template-based QA pairs based on how the 3D nature of the scene changes with actions taken.
Fu et al., 2021; Wang et al., 2022b; Ye et al., 2023). VLMs have been adapted for a wide range of downstream image (Isimpoukelli et al., 2021; Driess et al., 2023; Zhu et al., 2023; Li et al., 2022; Alayrac et al., 2022; Dai et al., 2024) and video (Zhang et al., 2023; Maaz et al., 2023; Tan et al., 2024; Wang et al., 2024; Li et al., 2023; Luo et al., 2023) understanding tasks. Many tasks require spatial understanding of the scene (Brohan et al., 2023; Zhang et al., 2024b). Multiple recent works (Fu et al., 2024a; Hong et al., 2023a; Cho et al., 2024) have noted their weakness in 3D pose and location estimation. Adjacent to works like Ray et al. (2023); Zhang et al. (2024a); Hsieh et al. (2023); Thrush et al. (2022) that point to deficiencies in understanding compositions of spatial relationships with objects and attributes, we explore perfect 3D information in synthetic images can improve spatial understanding.
3 SAT: Spatial Aptitude Training
Our goal is to improve the spatial reasoning capabilities of MLMs. Since obtaining varied 3D annotations with interactive movements of the camera and objects real images is expensive and tedious, we procedurally generate photo-realistic environments and auto- matically curate training data. The resulting data generation pipeline, SAT, serves both instruction-tuning data for MLMs and as a benchmark to test the dynamic spatial reasoning capabilities not present in existing benchmarks.
In total, our training dataset contains 175K questions across 22K procedurally generated scenes from ProcTHOR-10K dataset of indoor apartment buildings. Next, we outline the generation process for the two kinds of spatial questions and dynamic.
3.1 Generating Static Spatial QAs.
Aligning with contemporary benchmarks for spatial reasoning, we first generate instruction- tuning data for static spatial reasoning that deal with relative relations objects. Overall, we generate 127K static spatial QA pairs across 8K images across the following types:
Relative spatial relations. We generate questions about the relative location of one object in the scene to other objects. We form two kinds of questions- (1) judging if object X is to the left, right, above, or below object Y. For example, Is the wine bottle to the left or right of the plate? (2) judging if object A or B is closer to another object C. For example, Which object is closer to the wine nottle-the cup, or the plate? Given the camera parameters, we first project the objects' poses into the camera coordinate system and then generate the corresponding answers. More details about the camera coordinate normalization are in the appendix.
Relative Depth. We generate questions about judging whether object X is closer to the camera than object Y by calculating the distances to the objects in the simulator. For example, Is the wine bottle closer to the camera than the plate?
Count. Since many MLMs (Goyal et al., 2017) struggle with counting, we include counting questions using the number of object instances from the metadata of our simulator.
4

---

Published as a conference paper at COLM 2025
3.2 Generating Dynamic Spatial QA
Grounded in spatial cognitive tests (Anderson et al., 2013; Brucato et al., 2023; Vasilyeva & Lourenco, 2012), we outline five different complex tasks that require reasoning about egocentric movements, object movements, and allocentric perspectives. We generate 48K QAs on 13K images. The high-level idea behind generating such QAs is illustrated in Figure 2. Given a frame in a simulated environment, we take an action and formulate QAs based on how the 3D orientation of objects changes based on the action taken. Below, we outline the specific approach for each of the question types.
Egocentric Movement. This is based on the "moving room test" Anderson et al. (2013), fundamental test designed to assess and improve spatial cognitive development in children. This test aims to measure if an agent can judge how they moved given two frames. This is useful beyond just measuring spatial cognition since this task can help judge navigation trajectories from egocentric videos in embodied applications. We take random action from the choices of rotating left or right by a random angle, or moving left/ /rigtt by a random distance. Note that moving and rotating are different since camera can be moving left while rotating right. Based on the action taken, we formulate question of the type: How did the camera taking the video likely rotate/nove? with the answer being the action sequence taken. We have 6.9K training image-QA pairs of this type. This is denoted as EgoM in the tables. More details of the exact simulator actions taken to generate this are in the appendix.
Here, we randomly choose an object and move it by randomly chosen distance and direction, ensuring that the object is still in the frame of view. Next, we compare the updated 3D position of the object with the original position normalized by the camera coordinates to decide if the object got closer, further, more left, or more right from the original position, or if it did not move. Based on that, we form QA pairs of the type: Did any of the objects move from the first frame to the second frame? with the answer being the way the object moved. Note that sometimes objects may move in conjunction with camera movement. To answer the questions accurately, the agent needs to learn to distinguish between egocentric movement and objects moving. We have 6.9K training image-QA pairs of this type. This is denoted as ObjM in the tables.
Object Movement.
Allocentric Perspective. Inspired by spatial cognitive test for humans and animals Brucato et al. (2023), this test checks if the agent is able to take the perspective of another viewer and judge the relative locations of objects according to the other viewer. To make such reasoning QAs, we first choose 2D point in the scene and mark it as "X". Next, we teleport that agent to the 3D location corresponding to "X" (determined by ray tracing). We check the relative positions of objects according to the camera view from "X" (similarly as described in 3.1. We make QA's of the type: For someone at the mark 'X' facing left/right by 90 degrees, would the <object> be to their left or right? We have 6K training image-QA pairs of this type. This is denoted as Pers in the tables.
Goal Aiming. Aiming is a prerequisite for efficient navigation to objects Franz& & Mallot (2000), a fundamental spatial cognitive capability. Hence, we design QAs that check how well agents can aim to the desired object. We pick random object and calculate the angle of the object to the camera using the 3D location of the object and camera, assuming looking forward is 0 degrees (exact equations in the supplementary). Based on the angle, we formulate questions of the type: need to go to the countertop. Which direction should turn to face it? Since precise angles are hard to judge from a single image, we formulate answers as choices of rough angles to turn towards the left or right. We have 6.8K training image-QA pairs of this type. This is denoted as Aim in the tables.
Action Consequence. Here, the agent needs to reason about how the spatial relationships change when it takes a movement action, inspired by how humans can reason about the consequences of their movements in an environment Franz & Mallot (2000). Here, we show the first frame and ask the agent to judge if we would move closer/ closer/further, further, or look towards or away from an object if it took that action. e.g., If rotate left by 90 degrees and move forward, would move further from the sofa? Note that in most cases, moving forward would get us closer to object. To make the distribution of answers even, sometimes rephrase the question as to whether we would be facing the object or not. We have 15K training image-QA pairs of this type. This is denoted as EgoAct in the tables.
5

---

Published as a conference paper at COLM 2025
Table Both large proprietary and open-source MLMs struggle on our dynamic SAT spatial test set, including strong models like Geminil .5-pro and spatially-tuned Robopoint. SAT training improves LLaVA models significantly on SAT test set.
SAT Real
SAT Synthetic
Avg
EgoM
ObjM
EgoAct
GoalAim
Pers
Avg
a. GPT4-V
50.7
54.7
32.7
52.0
50.5
34.2
44.8
b. GPT4-0
57.5
61.5
33.2
47.6
67.5
37.5
49.4
Geminil.5-fla
57.6
67.1
33.1
52.9
64.0
32.7
50.0
Geminil.5-pro
64.8
57.7
29.8
55.5
56.9
49.5
49.9
e. Robopoint-13B
46.6
50.2
69.4
48.8
72.6
25.5
53.3
f. LLaVA-1.5-13B 41.6
46.6
73.8
49.7
45.6
39.9
51.1
8-
+ SAT
54.9
61.7
90.2
91.4
96.8
98.5
87.7
Á Improvement
+13.3
+15.1
+16.4
+41.7
+51.2
+58.6
+36.6
h. ILAVA-Video-7B
53.5
56.4
82.7
48.0
52.9
47.1
57.4
i. SAT
63.4
79.6
80.4
85.3
56.4
88.4
78.0
Improvement
+9.9
+23.2
-2.3
+37.4
+3.5
+41.3
+20.6
Real-image Test Set. Existing image spatial datasets only test for static relationships and not the above-mentioned dynamic capabilities. A recent video spatial dataset (Yang al., 2024a) also focuses on static object relations in video input where the scene doesn't change. Hence, we construct a small yet challenging test set of dynamic QAs that focus on how relationships change given ego or object motions and varying perspectives. We use expert annotators to annotate 150 maage-question-ansvee
pairs (~ 30 each of the above dynamic types). We also generate larger synthetic test set 647 object movement, 647 egocentric movement, 592 goal aim, 1336 action consequence, and 778 perspective questions on 805 images for analysis individual splits.
3.3 Finetuning MLMs with SAT
We provide the MLM with the required image(s) and the question and ask to choose between two answers- correct answer and distractor answer, following existing bench- mark standards (Fu et al., 2024b; Cai et al., 2024). The distractor answer is generated by switching the correct answer to the opposite/ distractor word (e.g, left - right, "did not move"). To prevent catastrophic forgetting, we mix in some of the original pre-training data during tuning from the LLaVA Instruct-tune dataset (Liu et al., 2024). More tuning and prompt details are in the appendix.
Experiments
Using our SAT data as a training set and existing spatial benchmarks along with our SAT real-image dynamic test set, we investigate what encourages spatial aptitude in MLMs.
Setup. We use two popular open-source MLMs: LLaVA-1.5-13B (Liu et al., 2024) and LLaVA-Video-77B (Zhang et al., 2024d) for experiments. We report the standard accuracy metric used for question-answering evaluations by checking the predicted answer matches the GT answer. We notice better performance for off-the-shelf MLMs when we provide the options in text (e.g., Choose between right or left) as opposed to option numbers (e.g., Choose between option A B). Hence, sometimes report higher performances for the baselines than those reported by the original papers. Below, we present the key experimental questions we wish to ask.
4.1 Can SAT improve spatial reasoning in MLMs?
To evaluate the effect of SAT training on spatial performance, we use existing static spatial Denchmarks-CCVBer (Tong et al., 2024), BLINK (spatial subsets) (Fu et al., 2024b), Visual

---

Published as a conference paper at COLM 2025
Table 3: SAT tuning improves performance of open MLMs on existing static spatial relation-
ship benchmarks- often outperforming large proprietary and spatially-tuned Robopoint.
CVBench (Tong al., 2024)
BLINK 2024b)
2DRel
3DDep
Avg
MV
RelDep
SpRel
Avg
a. GPT4-V
62.4
71.1
79.8
68.3
70.2
55.6
59.7
72.7
62.7
GPT4-o
85.7
78.9
58.6
82.5
70.2
Geminil 5 5-tash
66.0
76.9
75.3
68.3
71.4
51.1
62.9
62.9
59.0
Geminil 5-pro
77.4
Robopoint-136
53.6
79.4
74.7
71.3
69.1
48.1
51.6
75.5
58.4
E. LLaVA-1.5-138
58.2
46.6
53.0
47.8
51.7
45.1
56.4
69.9
57.1
8
SAT
61.5
89.7
80.7
73.0
75.6
44.4
76.6
72.7
64.6
Improvement
+3.3
+43.1
+27.7
+23.2
+23.9
-0.7
+20.2
+2.8
+7.4
LaVA-Vid-7B
39.1
35.6
25.5
36.7
L SAT
66.2
81.2
88.2
79.3
78.4
48.1
66.1
73.4
62.6
A Imprommentt
+6.9
+4.2
+16.9
+24.6
+13.2
+9.0
+10.5
-2.1
+5.8
Table 4: SAT also improves spatial performance on videos, VSI-Bench (Yang et al., 2024a)
Rel Dist
Rel Dir
Rt. Plan
App. Order
MC Avg
a. GPT4-o
37.0
41.3
31.5
28.5
34.6
b. Gemini-1.5-flash
37.7
41.0
31.5
37.8
36.9
c. Gemini-1..-pro
51.3
46.3
36.0
34.6
42.1
d. LLaVA-Video-7B
43.9
42.0
33.5
32.3
37.9
e. + SAT
47.9
39.6
38.7
40.6
41.7
A Improvement
+4.0
-2.4
+5.2
+8.3
+3.8
Spatial Relations (VSR) (Liu al., 2023), and GQA-Spatial (Hudson Manning, 2019a). For dynamic reasoning, we use our SAT test sets on both real and synthetic images. Since SAT real is small, we only report the overall average and report individual split performance on our synthetic test set for more detailed analysis. Finally, we evaluate the effect of SAT tuning on a recent long-video spatial benchmark, VSI-Bench (Yang et al., 2024a), by tuning LLaVA-Video-7B (Zhang et al., 2024d).
Open MLMs struggle on spatial reasoning. While proprietary MLMs perform better on static reasoning, they still struggle with dynamic reasoning. In Tables 3 and 2, we see that open-source MLMs (rows and h) struggle on both static relationships and dynamic reasoning. Larger closed-source models like GPT4-o (Achiam et al., 2023), Gemini-1.5- Robopoint (Yuan et al., 2024b) are better at static
pro (Team et al., 2023) and spatially-tunaa reasoning (in Table 3 row b, d), but still struggle on dynamic QAs (Table rows a-e). On SAT synthetic, aiming at the goal is generally easier for spatially stronger models like RoboPoint and GPT4-o since it also mostly requires judging object position.
Training on SAT significantly improves static spatial reasoning real images for MLMs. As shown in Table 3, tuning on SAT improves performance on static spatial questions for both LLaVA-1.5 and LLaVA-Video models by significant amounts (rows f vs g and vs i). Comparing rows a-e, we observe that SAT tuning makes both LLaVA models match/ aatch/outperform some closed-source models on zero-shot performance on CVBench and BLINK. Compared to an existing apatially-tuned baseline, we outperform RoboPoint- 13B (Yuan et al., 2024b) (row e). This shows promise that SAT instruction-tuning may push performance further for these stronger models.
Training on SAT improves dynamic spatial reasoning real images. Tuning on simulated SAT improves performance on SAT real test set as shown in Table (column 1) by 12% for LLaVA-1.5 and by 10% for LLLVA-Viiees This shows strong sim-to-real transfer. Once again, our strongest model matches Gemini-1.5-pro and outperforms GPT models.
Camera movement and out-of-domain relations are hard to improve BLINK SpRel has abstract relationships (especially with people) not present in our synthetic data (e.g., "looking away", "surrounding"), Hence, in Table 3, gains modest on SpRel (row vs g), with
7

---

Published as a conference paper at COLM 2025
Table SAT also improves out-of domain spatial relation (such as outdoor and video reasoning with humans on MME-Reaaloorl--Ltt) despite having only indoor scenes.
Model
Avg
Motion
Interaction
Position
Reasoning
Perception
LLaVA-Vid-7B
35%
26.8%
24.5%
34%
36.8%
35.0%
+ SAT
39%
33.2%
31.5%
42%
39.7%
39.6%
A Improcement
+4%
+6.4%
+7%
48%
+2.9%
+4.6%
Table 6: Performance on some traditional spatial benchmarks- VSR (Liu et al., 2023), GQA- Sp (Hudson & Manning, 2019b)- and other VQA benchmarks- GQA (Hudson & Manning, 2019b), OK-VQA (Marino et al., 2019), VQAv2 (Goyal et al., 2017)- not focused on spatial relationships. Our spatial tuning remembers pre-training commonsense on other capabilities while improving spatial reasoning.
VSR
GQA-Sp
GQA
OK-VQA
VQAv2
LLaVA-15-13B
65.9
53.1
78.6
30.7
60.5
b. SAT
70.4
55.8
718
35.1
60.9
LLaVA-Video (row h vs i) showing improvement. Camera movements are also hard (BLINK MV). While more investigation is needed, we believe this may be due to translation invariance in ViT (Dosovitskiy et al., 2021) leading to minimal feature changes between images with subtle camera movement.
Training SAT improves spatial reasoning in videos. Even though SAT only has up to 2 frames in its training data, we evaluate its impact on longer videos on Yang et al. (2024a). The results are shown in Table 4. We see that SAT training helps significantly on route planning capabilities (column Rt. Plan) while also improving other spatial splits. This shows promise that SAT tuning can benefit embodied AI applications.
SAT training also improves on outdoor videos including humans. SAT dataset is rendered using indoor simulators without any humans. However, when trained on SAT, we see improvements on outdoor video reasoning that includes humans on MME-Realworldd-Lee (Zhang et al., 2024c) as shown in Table 5. While MME-RealWorid has numerous splits related to reasoning and perception, we highlight some of the splits where we see the most improvements when tuned on SAT along with the overall average (Avg column) and the average for all "Reasoning" and "Perception" "-baedd splits.
Our SAT-tuned model maintains its non-spatial reasoning capabilities. We run an eval- uation some standard VQA benchmarks-nand
GQA (Hudson & Manning, 2019a), VQAv2 (Goyal et al., 2017), and OK-VQA (Marino et al., 2019) (9K image-QA pairs, random 3K from each). We maintain overall performance on them compared to the off-the-shelf LLaVA (Liu et al., 2024) as shown in Table 6. This suggests that we remember pre-trained vision-language commonsense while adding stronger spatial capabilities.
SAT synthetic is a useful diagnostic test set The lack of improvement on object relations (SpRel and 2DRel) in LLLVA- VVdeo is also reflected by lesser gain the GoalAim (row h vs Table 2) on SAT synthetic test, where the model needs to judge object position. Similar for SAT EgoM (Table 2) and BLINK MV (Table 3) dealing with camera movements-LLave Video improves more than LlaVA on both. Hence, despite being synthetic, SAT synthetic is also a useful diagnostic test set.
L.LVA--Vieo
4.2 What are effective types of spatial training data?
Sources of instruction tuning Since contemporary work on spatial understanding uses pseudo annotations (Chen et al., 2024; Cheng et al., 2024), we wish to compare to such baselines as well. Since their datasets are not easily available yet, we reproduce a similar instruction-turiin seneiiiinntuingggg set using pseudo annotations to create spatial QAs. We keep formats the same as SAT to ensure the performance differences are attributed to the source of spatial information and not the question formats.
8

---

Published as a conference paper at COLM 2025
Table 7: Table showing the effectiveness of SAT over existing sources of spatial data. Dy- namic spatial reasoning data improves performance over just static spatial relationship data.
CVBench
BLINK
SATReal
VSR
3DDep
Avg
MV
RelDep
SpRel
Avg
a. LLaVA-1.5
41.6
58.2
46.6
53.0
47.8
51.7
45.1
56.4
69.9
57.1
66.0
GQAPseudo
45.1
64.0
54.5
53.4
57.3
53.1
62.3
1SR/25VRD
45.4
60.8
64.2
54.3
44.7
55.9
3.0
61.3
69.9
44.7
67.9
SAT
81.7
72.5
66.4
68.0
Dynamic
54,9
61.5
89.7
80.7
73.0
75.6
44.4
76.6
72.7
64.6
70.4
a Sim-only
Sim + Real
Sim-onlly
A Sim + Real
15
10
13.2
143
0
SAT Real
CVB
BLINK
Avg
SAT Real
CVB
BLINK
USI.Bencch
Avg
(a) Improvement over LLaVA-1.5-13B
(b) Improvement over LLaVA-Video-7B
Figure 3: Simulation-only training vs mixing in some original pre-training data. Sim-only is very effective, mixing in pre-training data improves further.
GQAPseudo: We create SAT-Static like instruction-tuning data by inferring the depth (Yang et al., 2024b) on images from GQA (Hudson & Manning, 2019a) and Vi- sualGenome (Krishna et al., 2017) and the 2D bounding box annotations. We filter out potential incorrect relationships using simple heuristics (more details in the appendix). We create 225K static spatial image-QA Maage-QA tuples. Generating pseudo-annotated dynamic QAs is not possible since we cannot easily infer the result actions on real images, hence, we only generate static QAs.
VSR/25VRDD We use spatial relationship datasets like VSR (Liu et al., 2023) and 2.5VRD (Su et al., 2021) to produce more real spatial QAs. These datasets contain relation- ships such as "touching",
"behind", allowing formulate QAs such "Is the cat touching the sofa?" We combine VSR and 2.5VRD to create 200K image-QA tuples.
SAT simulated data outperforms pseudo-annotations or human annotations. In Table 7, we see that pseudo-annotation can improve performance on certain splits like Count and 2DRe on CVBench (row b). However, noise in annotations (egg, bounding boxes often not accounting for the entire object) leads to errors in judging depth the entire object relative to other objects (e.g., part of the annotated "lake" may be in front of "tree", but the majority of it may lie behind it). Following more complex careful curation similar to Cheng et al. (2024) may improve this performance, which we leave to future work since their data is not yet easily available. Human-annotated spatial relations (using Su et al. (2021); Liu et al. (2023)) tend to perform well in-domain VSR test. While the higher diversity of nuuman-annottted relations also helps prevent forgetting on BLINK SpRel (row vs d), they are finite and not easily composable generate varied 3D and dynamic data. Hence, we observe no improvements on SAT dynamic and minimal gains in 3D perception splits.
Adding dynamic QAs further improves performance over just static QAs. We notice improvements on both dynamic and static performance when we add dynamic data as noted in Table (row d vs e). However, BLINK MV remains challenging and we observe no performance gain. While static-only seems to perform better on BLINK MV, we observe that simply predicting "moved right" all the time achieves that performance. We also notice gains if dynamic mixed data matches the amount of static-only (in appendix Table 10).
SAT training scales with more data, especially when dynamic splits added. Since SAT is generated procedurally, we wish to explore whether spatial performance of MLMs scale with more data. Hence, we vary the amounts of training data (train for epoch). As shown in Figure 4, we see that while performance tends to saturate with static-only training (likely due to overfitting to simpler relationships), the performance scales positively with dynamic reasoning questions mixed in.

---

Published as a conference paper at COLM 2025
LLaWA-1
5-138
LLNVA--Mee--S BLINK
HO91 Ooreenon
SVSi-Sndhh EAT Real -O/tand --
feal CVteret BLNN --
30 11.0%
10%
70 0.9
02.0%
80.0%
72.0%
00.00 13.0%%
00.00%
62.0%
10.0%
91.0%
30.00%
4110%
42.00%
47.0%
3110%
126
30K
GBC
10K
55.
127K
153K
17K
20.05
SAT
Stateo
5A1
State-Dyumm
GA1 Sturec
SAI Blate-yranre
Figure 4: Ablations with training data size, We see that performance generally scales with data, with dynamic showing better scaling.
Mixing in some original pretraining data when training in simulation helps performance We check whether simulation data alone suffices to impart spatial reasoning to MLMs. For sim-only training, we use our full SAT statictdynamic data. We compare this with sim+rral training, where we mix random samples of the original pretraining data 60% of the time during the training along with our SÄT data. For both cases, the model sees the same amount of new SAT spatial data. As shown in Figure 3, surprisingly, simulation alone is still effective at improving the spatial awareness on real images. However, mixing in pretraining data improves further.
5 Discussion
Limitations. We instruction-tune two kinds of MLMs for spatial reasoning, an image- only LLaVA (Liu et al., 2024) and a video LLaVA-Video (Zhang et al., 2024d). While we remember pretraining commonsense as noted in Table 6, we haven't explored improving other capabilities like math and science reasoning Yue et al. (2024). The scope of the paper, however, is to analyze what kinds of data improve spatial performance, and not a large-scale training of a new MLM. Additionally, further analysis on more MLMs (Dubey al., 2024; Deitke et al., 2024) might be beneficial.
Future work. Although our study focuses on evaluating the spatial reasoning capabilities of MLMs, it can be extended in various avenues. For instance, to determine the kinds of embodied applications that benefit from improved complex spatial reasoning. We do see gains in accuracy in route planning in VSI-Bench (Yang et al., 2024a), which should help embodied navigation. As a more direct evaluation, we also check the action prediction accuracy (from choice of going left right/ forward) for given frame the SPOC Easy- ObjectNav benchmark (Ehsani et al., 2024). Our model (SAT Dynamic) scores an accuracy of 51% compared to 40% for training only on static spatial questions. This suggests that embodied navigation might benefit from improved dynamic spatial reasoning. We leave more thorough evaluation in this direction as future work. Further, leveraging the interac- tive nature of our scenes with perfect 3D information could facilitate more explorations in dynamic and their corresponding image frames could also advance language-controlled world models Ball et al. (2025), a direction we leave for future work.
causal reasoning. Our causal annotations in language and
Conclusion. We propose training and testing dataset of dynamic motion-based spatial pro tasks that go beyond simpler static perception of relationships existing datasets. This improves MLMs numerous spatial benchmarks while maintaining pre-trained common- sense. We hope that SAT paves the way for bridging the gap between passive perception and active interaction reasoning in MLMs, making them more suitable for deployment in real-life applications.
10

---

Published as a conference paper at COLM 2025
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arviv preprint ar xiXi::3003.08774 2023.
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Vision-ECCV 2020: 16th European Conference, Glasgoro, UK, August 23-28, 2020, Proceedings, Part I 16, pp. 422-440. Springer, 2020.
Christopher Agia, Krishna Murthy Jatavallabhula, Mohamed Khodeir, Ondrej Miksik, Vib- hav Vineet, Mustafa Mukadam, Liam Paull, and Florian Shkurti. Taskography: Evaluating robot task planning over large 3d scene graphs. In Conference on Robot Learning, pp. 46-58. PMLR, 2022.
Bean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-2277777 2022.
David I Anderson, Joseph J Campos, David C Witherington, Audun Dahl, Monica Rivera, Minxuan He, Ichiro Uchiyama, and Marianne Barbu-Roth. The role of locomotion in psychological development. Frontiers in psychology, 4:440, 2013.
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding, In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19129-19139, 2022.
Philip Ball, Jakob Bauer, Frank Belletti, Bethanie Brownfield, Ariel Ephrat, Shlomi Fruchter, Agrim Gupta, Kristian Holsheimer, Aleksander Holynski, Jiri Hron, Christos Kaplanis, Marjorie Limont, Matt McGill, Yanko Oliveira, Jack Parker-Holder, Frank Perbet, Guy Scully, Jeremy Shar, Stephen Spencer, Omer Tov, Ruben Villegas, Emma Wang, Jessica Yung, Cip Baetu, Jordi Berbel, David Bridson, Jake Bruce, Gavin Buttimore, Sarah Chakera, Bilva Chandra, Paul Collins, Alex Cullum, Bogdan Damoc, Vibha Dasagi, Maxime Gazeau, Charles Gbadamosi, Woohyun Han, Ed Hirst, Ashyana Kachra, Lucie Kerley, Kristian Kjems, Eva Knoepfel, Vika Koriakin, Jessica Lo, Cong Lu, Zeb Mehring, Alex Moufarek, Henna Nandwani, Valeria Oliveira, Fabio Pardo, Jane Park, Andrew Pierson, Ben Poole, Helen Ran, Tim Salimans, Manuel Sanchez, Igor Saprykin, Amy Shen, Sailesh Sidhwani, Duncan Smith, Joe Stanton, Hamish Tomlinson, Dimple Vijaykumar, Luyu Wang, Piers Wingfield, Nat Wong, Keyang Xu, Christopher Yew, Nick Young, Vadim Zubov, Douglas Eck, Dumitru Erhan, Koray Kavukcuoglu, Demis Hassabis, Zoubin Gharamani, Raia Hadsell, Aäron van den Oord, Inbar Mosseri, Adrian Bolton, Satinder Singh, and Tim Rocktäschel. Genie 3: A new frontier for world models. 2025.
Mark Blades and Christopher Spencer. The development of children's ability to use spatial representations. Advances in child development and behavior, 25:157-199, 1994.
Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13154-13164, 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-languagg-action models transfer web knowledge to robotic control. arv preprint arXiv:2307.15811, 2023.
Maria Brucato, Andrea Frick, Stefan Pichelmann, Alina Nazareth, and Nora S Newcombe. Measuring spatial perspective taking: Analysis of four measures using item response theory. Topics in Cognitive Science, 15(1):46-744 2023.
11

---

Published as a conference paper at COLM 2025
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving, 2020. URL httpp: http:///aaaiii.ooaa//////// /900// 11027.
Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, et al. Temporalbench: Benchmarking fine-grained temporal understanding for multimodal video models. arXiv preprint ar arXiv:2410.18818, 2024.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. arXiv preprint ar vrXiv:2401.12166 2024.
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In Luropean conference on computer vision, pp. 202-221. Springer, 2020.
Wuyang Chen, Zhiding Yu, Shalini De Mello, Sifei Liu, Jose M. Alvarez, Zhangyang Wang, and Anima Anandkumar. Contrastive syn-to-real generalization, 2021a. URL https: //arxiv. org/ abs 02290.
Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context- aware dense captioning in rgb-d scans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3193-3203, 2021b.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language model, 2024. URL https://arxet http:://aaxiw..og/bbbb///ss org/abs/2406. 01584.
Jang Hyun Cho, Boris Ivanovic, Yulong Cao, Edward Schmerling, Yue Wang, Xinshuo Weng, Boyi Li, Yurong You, Philipp Krähenbühl, Yan Wang, et al. Language-image models with 3d understanding. arviv preprint ar Xiv::4405.00688 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large- scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 35:5982-5994, 2022.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli Van- derBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao
Kuota,KuoHHao Zeng, Jon Borchardt, Dirk Groeneveld, Dumnas Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Anirud- dha Kembhavi. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models, 2024. URL htttp:://arriiii /tg/////////////////o///eeeeaa// 2409. 17146.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. Xiv preprint ar arXin:2303.03378 2023.
12

---

Published as a conference paper at COLM 2025
Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. Embspatial--ench: Benchmarking spatial understanding for embodied tasks with large vision-language models, 2024. URL https://arxin: https://arxi...rr/bbsss orgg 05756.
Jiafei Duan, Samson Yu, Soujanya Poria, Bihan Wen, and Cheston Tan. Pip: Physical interaction prediction via mental simulation with span selection. In European Conference on Computer Vision, pp. 405-421 425. Springer, 2022a.
Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2):230-24) 2022b.
Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. aanipulate-anything Automating real-world robots using vision- language models. In Conference on Robot Learning, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint ar Xiv:2400.22220 2024.
Kiana Ehsani, Tanmay Gupta, Rose Hendrix, Jordi Salvador, Luca Weihs, Kuo-Hao Zeng, Kunal Pratap Singh, Yejin Kim, Winson Han, Alvaro Herrasti, et al. Spoc: Imitating shortest paths in simulation enables effective navigation and manipulation in the real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16238-16250, 2024.
Matthias O Franz and Hanspeter A Mallot. Biomimetic robot navigation. Robotics and autonomous Systems, 30(1-2):133-153, 2000.
Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending lan- guage model for 3d visual understanding and reasoning. ar preprint Xix::2003.11400 2024a.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token modeling. ar Xiv preprint ar XrXiv:2111. 126688 2021.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. Xr preprint vrXto:2404.12390 2024b.
Howard E Gardner. Frames of mind: The theory of multiple intelligences. Basic books, 2011.
Scott Geng, Cheng- Yu Hsieh, Vivek Ramanujan, Matthew Wallingford, Chun-Liang Li, Pang Wei W Koh, and Ranjay Krishna. The unmet promise of synthetic training images: Using retrieved real images performs better. Advances in Neural Information Processing Systems, 37:7902-7929, 2024.
Ankit Goyal, Kaiyu Yang, Dawei Yang, and Jia Deng. Rel3d: benchmark for grounding spatial relations in 3d. Advances in Neural Information Processing Systems, 33, 2020.
A minimally contrastive
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering, 2017. URL //axxvv h/tg//bs/1/ 2.. 00837.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, (6620482-20494. 2023a.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models, 2023b.
13

---

Published as a conference paper at COLM 2025
Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality, 2023.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019a. URL https: http::///xxvvvoreeeeeee orgp:iii.orrr/roooo / 1902.09505
Drew A. Hudson and Christopher D. Manning. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering, May 2019b. URL http::/trxiii..o/ abs/1902. 09506. arXiv:1902.09506 [cs].
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba. Conceptfusion: Open-set multimodal 3d mapping, 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, PP. 4904-4916. PMLR, 2021.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr modulated detection for end-to-end multi-modal understanding, 2021. URL https: //axxiv http:/iwwwwr///b.////eeee/// htrrg/abs21044/ 12763.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's "up" with vision-language models? investigating their struggle with spatial reasoning, 2023. URL https: htt.v.oogg/////////// 2310. 2978.19785.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4):1-14, 2023.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AL. ArXio, abs/1712.05474,, 2017.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Con- necting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic abject-aware neural scene representation. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12871-12881, 2022.
Andrey Kurenkov, Michael Lingelbach, Tanmay Agarwal, Emily Jin, Chengshu Li, Ruohan Zhang, Li Fei-Fei, Jiajun Wu, Silvio Savarese, and Roberto Martn--arrtn. Modeling dynamic environments with scene graph memory. In International Conference Machine Learning, pp. 17976-17993. PMLR, 2023.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXit:2305.06355. 2023.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De- An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive mecision-making. Advances in Neural Information Processing Systems, 35: 31199-31212, 2022.
14

---

Published as a conference paper at COLM 2025
Xiongkun Linghu, Jiangyong Huang, Xuesong Niu, Xiaojian Ma, Baoxiong Jia, and Siyuan Huang. Multi-modal situated reasoning in 3d scenes, 2024. URL hts//449.022388
Benlin Liu, Yuhao Dong, Yiqin Wang, Yongming Rao, Yansong Tang, Wei-Chiu Ma, and Ranjay Krishna. Coarse correspondence elicit 3d spacetime understanding in multimodal language model. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2025.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning, 2023. URL https: //arxiv. wwwpiiv.orr/b/////////e/o 22///205.2222/ 00363.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. ar arviv preprint ar axXi::2306007207, 2023.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video- chatgpt: Towards detailed video understanding via large vision and language models. ar Xiv preprint ar arXtv:230.005424 2023.
Hanspeter A Mallot. From geometry to behavior: An introduction to spatial cognition. MIT Press, 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019. URL 0306600067.
https://axxiv.org/aa//
Samarth Mishra, Rameswar Panda, Cheng Perng Phoo, Chun-Fu Richard Chen, Leonid Karlinsky, Kate Saenko, Venkatesh Saligrama, and Rogerio S Feris. Task2sim: Towards effective pre-training and transfer from synthetic data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 919999949999 9204, 2022.
Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang, TotalBdunderssanddngg Joint layout, object pose and mesh reconstruction for indoor scenes from single image. In Proceedings of the IEE.//CV Conference on Computer Vision and Pattern Recognition, pp. 55-64, 2020.
Charles Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9277-9286, 2019.
Maan Qraitem, Kate Saenko, and Bryan A Plummer. From fake to real (ffr): A two-stage training pipeline for mitigating spurious correlations with synthetic data. ar preprint ar arXiv:2308.0455 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, Pp. 8763. PMLR, 2021.
Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan A. Plummer, Ranjay Krishna, and Kate Saenko. Cola: A benchmark for compositional text-to-image retrieval, 2023.
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd Van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas] J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. Advances in Neural Information Processing Systems, 35:9512- 9524, 2022.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the I.F.//VV international conference computer vision, pp. 2019.
15

---

Published as a conference paper at COLM 2025
Sneha Silwal, Karmesh Yadav, Tingfan Wu, Jay Vakil, Arjun Majumdar, Sergio Arnaud, Claire Chen, Vincent-Pierre Berges, Dhruv Batra, Aravind Rajeswaran, et al. What do we learn from large-scale study of pre-trained visual representations in sim and real environments? In 2024 IEEF. International Conference on Robotics and Automation (ICRA), pp. 17515-17521. TEEE, 2024.
Yu-Chuan Su, Soravit Changpinyo, Xiangning Chen, Sathish Thoppay, Cho-Jui Hsieh, Lior Shapira, Radu Soricut, Hartwig Adam, Matthew Brown, Ming-Hsuan Yang, and Boqing Gong. 2.5d visual relationship detection, 2021. URL https: org/ 2ttg//////200111111// 12727.
Reuben Tan, Ximeng Sun, Ping Jui-hsinn Wang, Hanieh Deilamsalehy, Bryan A Plummer, Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long video-llm. arXiv preprint ar Xi0:2404.04334, 04346, 2024.
Holly Taylor and Barbara Tversky. Spatial mental models derived from survey and route descriptions. Journal of Memory and langurge, (1(2):261-292, 1992.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. ar Xiv preprint ar 2rXi::2312.11800 2023.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality, April 2022. http: arXiv:2204.03166 [cs].
htrrg//bb/2224... 03162. 0311226...........
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal Ilms, 2024. URL https: wogg htrrg/abs/440.... 16860.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212, 2021.
Barbara Tversky. Mind in motion: How action shapes thought. Hachette UK, 2019.
Barbara Tversky and Masaki Suwa. Thinking with sketches. 2009.
Marina Vasilyeva and Stella F Lourenco. Development of spatial cognition. Wiley Interdisci- plinary Reviews:: Cognitive Science, 3(3):349-36) 140344993,,220 2012.
Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for image- language and video-lane rideo-language tasks. Advances in neural information processing systems, 35: 5696-5710, 2022a.
Peng-Shuai Wang.
Octformer: Octree-based transformers for 3d point clouds. ACM Transactions on Graphics (TOG), 42(4):1-11, 2023 2023.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. ar preprint ar axXiv:2212.03111,, 2022b.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvide02: Scaling video foundation models for multimodal video understanding. arXiv preprint 2rXia:2403.15377 2024.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arviv preprint ar Xrxi:22412.14177 2024a.
16

---

Published as a conference paper at COLM 2025
Kaiyu Yang, Olga Russakovsky, and Jia Deng. Spatialsense: An adversarially crowdsourced benchmark for spatial relation recognition, 2019. URL https://arxvv.orr/abb////// 02660.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024b.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint ar arXio:2304.14178 2023.
Shuquan Ye, Dongdong Chen, Songfang Han, and Jing Liao. 3d question answering. arXiv preprint arXiv:2112.088399 2021.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. Xrv preprint arXi::2111.11432 2021.
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics. In Conference on Robot Learning, 2024a.
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-languuage model for spatial affordance prediction for robotics, 2024b. URL 10721.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tundd audio-visual language model for video understanding. arXiv preprint ar arXiv:2306.022 008858, 2023.
Jieyu Zhang, Weikai Huang, Zixian Ma, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, and Ranjay Krishna. Task me anything. Advances in Neural Information Processing Systems, 2024a.
Jimuyang Zhang, Zanming Huang, Arijit Ray, and Eshed Ohn-Bar. Feedback-guided autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15000-15011, 2024b.
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal Ilm challenge high-resolution real-world scenarios that are difficult for humans? ar Xiv preprint ar Xiv:24408. 18215777, 2024c.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video Instruction Tuning with Synthetic Data. arXiv preprint ar xrXi::2410.02713, 2024d.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv22304.1052 2023.
A Appendix
Acknowledgements
This project was partially funded by Amazon Science and NSF award number 2120322. We wish to thank Devdeep Ray and Jang Hyun Cho for helpful discussions on 3D and depth estimation. The contents of the paper do not represent the views of the US Government or the funding agencies.
17

---

Published as a conference paper at COLM 2025
A.1 Discussion
Limitations. We instruction-tune two kinds of MLMs for spatial reasoning, an image- only LLaVA (Liu et al., 2024) and video LLaVA-Video (Zhang et al., 2024d). While we remember pretraining commonsense as noted in Table 6, we haven't explored improving other capabilities like math and science reasoning Yue al. (2024). The scope of the paper, however, is to analyze what kinds of data improve spatial performance, and not a large-scale training of a new MLM. Additionally, further analysis on more MLMs (Dubey et al., 2024; Deitke et al., 2024) might be beneficial.
Future work. Although our study focuses on evaluating the spatial reasoning capabilities of MLMs, it can be extended in various avenues. For instance, to determine the kinds of embodied applications that benefit from improved complex spatial reasoning. We do see gains in accuracy in route planning in VSI-Bench (Yang et al., 2024a), which should help embodied navigation. As a more direct evaluation, we also checked the action prediction accuracy (from choice of going left /rtght/tttttttee forward) for given frame on the SPOC Easy- ObjectNav benchmark (Ehsani et al., 2024). Our model (SAT Dynamic) scores an accuracy of 51% compared to 40% for model trained only on basic spatial questions. This suggests that embodied navigation might benefit from improved dynamic spatial reasoning. We leave more thorough evaluation in this direction future work. Further, leveraging the interac- tive nature of our scenes with perfect 3D information could facilitate more explorations in dynamic and causal reasoning.
A.2 Tuning details
We tune a widely used open-source MLM, LLa Liu et al. (2024) for fine-tuning experiments. We LoRA-tune (Hu et al., 2021) with rank 256 and alpha 512. We full fine-tune the LLaVA-Video-7BB model since it is smaller. We see minimal differences in trends by training with more lora parameters or with full fine-tuning as shown in Figure 7. To prevent catastrophic forgetting, we randomly sample examples from the LLaVA Instruct Tuning dataset Liu et al. (2024) with 40% probability while tuning with our synthetic QA pairs. We train all models to around 1 epoch of synthetic data. Due to memory constraints, we use batch size of 8 (using gradient accumulation). We set a small learning rate of 5e-6 (due to our small batch size) with cosine annealing with 1K warm-up steps and a weight decay of 0. Training requires two 48GB NVIDIA GPUs, while inference is possible with one GPU. To demonstrate effect on a video benchmark, we tune another widely used video MLM, (Zhang et al., 2024d). FOr LLa LLaVA-Vided VAVVe-odd---b, the paper results are solely by training on SAT, without mixing any real orginal pretraining data. However, we also do compare the effect of mixing in real pretraining data as show it in Figure We also show the sacling and
A.3 Dataset Details
Human Performance our SAT We conduct human study with experts to measure the quality of our synthetic test data which is automatically generated. We observe that spatial awareness demands more mental power since one has to pay more attention and reason about how the orientation of the scene changed or would change based on an action. We conduct an expert human study, where we ask anonymized graduate students to answer 200 randomly sampled questions from our test set using the interface showed in Figure 10. We see that humans are 92.8% accurate on our SAT dataset. This is still a significant gap compared to the performance of best existing zero-shot MLM (around 53%). We note that zero-shot SAT accuracies on the synthetic test set are lower since synthetic image are often out of domain for MLMs. Hence, we most use SAT-synthetic to as diagnostic set to analyze performance gains after tuning the MLMs. We see tha gains on the synthetic test set are indicative of performance Huggingface. Our SAT real dataset was annotated by careful cross-checking by 4 graduate students. Our datasets will all be released publicly.
the related task on real images. We will release the dataset
18

---

Published as a conference paper at COLM 2025
More details on dataset creation
We first take an apartment from ProcTHOR-10K and place the camera position where many objects are visible. We do this by randomly choosing 20 points to place the camera and then choosing the point with max objects visible.
Normalizing the camera coordinates In ProcTHOR Deitke et al. (2022), in the camera view, the y coordinate the height coordinate, which means the y increases pointing upwards (e-g,, the ceiling has greater than the floor). Hence, from the bird's ss eye view, the coordinates are x and The rotation of the camera is such that always parallel to the x-2 plane. Hence the rotation is described as angle clockwise around the y-axis with the camera pointing to the positive z-axis as O-degree rotation.
Given a camera rotation, we normalize the view by translating to (0,0) for and z by subtracting the camera Xo and yo- Further, we rotate the x-z plane such that the camera points to the positive z-axis.
For rotation, we use the formula:
cos a) san(a)
sin(a) (a)
R
Hence, the normalized x', z' for any object is computed using:
xo yo
R.
The y value remains unchanged since it is the height, which is not affected since do not change the camera height.
Hence, finally, x' goes negative to the left and positive to the right, z' goes positive towards the depth and y goes positive upwards from the floor level. We use the values of x' and z' to calculate relative relationships (left, right, in front of, and behind) as described below.
SAT Static Spatial QAs
Relative spatial relations. For instance, if the value of x' for "chair" is lower than that of "table", the chair to the left of the table. We can also compute the distance between objects. We randomly choose 3 objects. We compute the pairwise distances using their y, 3D coordinates. Based on whether object is closer or further to object 2, we make QAs like "Is the couch closer to the lamp the table?"
Relative Depth. Similarly, if the value of z' for, say, "lamp" is greater than that of "couch", we say the "lamp" is further away from the camera than the couch.
SAT Dynamic Spatial QAs
Egocentric Movement. We first choose an image frame. Next, we first choose to rotate left or right from angles 20, 30, 40, 50, 60 chosen randomly. We use the controller.step(actionn wwwpptttiion='tteeeeeeeee RotatteRigh'', degrees=angle) function in the AI2THOR Kolve et al. (2017) platform. Next, we move forward with probability 0.5 by random distance wwtpppaction
from 20 to 40 centimeters
moveMagni tu))=dist).
We capture the next frame from this final position. We also formulate camera movement questions. Note that camera movement and rotation are separate since a camera can be moving left while rotating right. For camera movement, we follow translate the camera beft/right in a random direction by a random distance. Specifically, we first rotate by an angle between 45 to 90 degrees to the left /right, then move forward by random distance and then rotate right/let (right if previously rotated left in the first step and vice versa) by an angle between 90 and 135 degrees. Sometimes, we do not move rotate.
Object Movement. We first choose an object visible in the scene with at least a certain bounding box area to make sure the object is not too small non-salient. Next we decide to
19

---

Published as a conference paper at COLM 2025
move the object by random distance sampled from 0.25 to 0.5 meters in random direction possible. We use the PlaceObjectAtPoiie platform. Sometimes, we do not move any object. Sometimes the camera well as object moves. The agent needs to disentangle the two to answer questions correctly.
function in the A12THOR Kolve et al. (2017)
Allocentric Perspective. We choose point the 2D image that is not too close to the margsns (in between 0.2-0.8 normalized width and height of the image). We use action in A12Thor Kolve al. (2017) to get the 3D location of the point. We try few points since some points may not possible to navigate to and pick one possible point. We randomly turn left right by 90 degrees. Next, we check whether an object is left/ riihtt based on the method described in Section A.3.
Goal Aiming. We compute the angle between randomly chosen object with the camera assuming looking straight is 0 degrees. If the angle of an object is - we say we have to "turn left by degrees" and "right by degrees" otherwise. When |a - 0 I e, we say we have to look "roughly straight". We define e 10. Note it is very hard for human or machine to judge the exact degrees to turn from one single image, hence, we give multiple choices where the difference in choices are between left /right and hence, the machine really only has to decide between them and not the exact angles to turn. We use the following equation to calculate the angle of an object with the camera. First normalize the object coordinates to the camera coordinates, (Xo, yo), and then calculate the angle, a.
x²
X0
= R
y/
x
arctan
z
Action Consequence. Here, we just compute at the objects got close to or further away while taking the action for the action sequence task or the perspective task. We formulate questions based that. Note that most of the time get close to objects in the scene and hence we random subsample such cases.
Pseudo-annotatar QAs
We use DeythAnything (Yang et al., 2024b) to estimate depth of the scene and bounding box annotations in GQA (Hudson & Manning, 2019a) and VG (Krishna et al., 2017) to formulate static spatial QAs. To make spatial QAs, we first choose three objects in the scene based on which we make questions as defined in Section 3.1. We use the following heuristics first choose the three objects to reduce noise:
Too small objects: We do not choose objects where the bounding box area occupies less than 10% of the area. We observe this removes lot of noisy annotations and very non-salient objects.
Partially occluded objects: We do not choose objects if they are partially occluded by another object (based on the bounding box annotations).
Hence, we choose three clearly visible objects in the scene, estimate it's depth and and the center point location. Based on this 2.5D information, we formulate questions about their depth and relative locations (left/right, behind or in front of).
Evaluation details
Evaluating on VSR and GQA-Sp
For VSR (Liu et al., 2023), we have captions like "truck in front of airplane." " We re-formulate them to into quetsion-answer pairs like: "Is truck in front of airplane? Answer yes or no" to keep all evaluation formats consistent. This ensures performance differences due to the difficulties of spatial relationships in the datasets and not the question-answent format.
Evaluating on VSI-Bench
20

---

Published as a conference paper at COLM 2025
Table 8: Datasource for LLaVA-Video on static and dynamic
CVBench
BLINK
SAT Real
Count
2DRel
3DDep
3DDist
Avg
MV
RelDop
SpRel
Avg
LLaVA-Video 53.5
59.3
77.0
71.3
54.7
65.2
39.1
55.6
75.5
56.7
+ SAT Static
51.6
66.2
85.7
90.5
84.3
81.2
54.9
62.9
73.4
63.7
SAT Dynamic
63.4
66.2
81.2
88.2
79.3
48.1
66.1
73.4
62.6
Table 9: Datasource for LLaVA-VIdeo on long video VSI-Bench. Row b and c are trained purely on simulations, while row d is simulation and orignal pre-training data (I.LVA-OV data) mixed in during training to prevent forgetting.
VSI-Bench (vid)
Rel Dist
Rel Dir
Rt. Plan
App. Order
Avg
a. LLaVA-Video
43.9
42.0
33.5
32.4
38.0
b. + SAT Static
43.5
43.4
34.5
30.3
37.9
c. + SAT Static + Dyn
47.7
42.0
35.6
36.1
40.4
d. SAT Dyn LLaVA-OV
47.9
39.6
38.7
40.6
41.7
For VSI-Bench (Yang al., 2024a), we only use the MC split and not the numerical splits. We believe the numerical splits requiring estimation of exact distances and dimensions from RGB images alone without camera and depth information is difficult and out of scope for SAT since SAT focuses on relationships in images and dynamic scenes and not exact depth and distances.
Qussei format
We frame each questions binary choice following existing benchmark standards (Fu et al., 2024b; Tong et al., 2024), and to prevent the MLM from choosing shortcuts by eliminating obvious wrong choices in multiple choices (Cai al., 2024).
Following LLaVA Liu et al. (2024) convention, we use <image> tokens to represent images. This the exact prefix we use.
chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human' s questions.
Next, we add the question prompt to the prefix:
## <im_start><image<iin
Human: Answer in natural language. the person facing the frisbee? Choose between the following options: yes, no. ##AAsiittan:
For questions with two images, we simply have <innsttrrbxiiimeeexiaaaxxxiimee in the image part of the prompt.
The prefix with A chat between ..." is something we found to be very important for LLaVA performance. Hence, we append this prefix to the question both, when tuning and testing. Further, we also found performance improvements when we specify the answer choices in text like "choose between 'let'' or 'right" than asking the model to choose an answer option letter (like A or B) number (like option or 2). We randomize the answer choice order during evaluation. Since SAT Real is small, we perform circular eval, where we prompt the model with both ordering of choices and average the performance. This reduces randomness and robust. We also note higher variance in performance between different training seeds on BLINK due to the small size of the dataset. However, the trends remain the same. We will release all checkpoints, the training script, and the best tips and tricks in the training schedule.
21

---

Published as a conference paper at COLM 2025
Table 10: Effect of dynamic data keeping the data size the same as static different scales. This is to judge the effect of types of simulated spatial data. We see that adding dynamic maintains static spatial reasoning while improving dynamic and video spatial reasoning.
CVBench
BLINK
VSI-Bench
SAT Real
CVB 2D
CVB
Avg
Avg
MC Avg
LLaVA-13B
41.58
52.95
50.40
51.68
57.13
+Stat 127K
40.08
69.40
76.42
73.40
39.84
+Stat+Dyn 127K
51.26
73.37
77.33
63.26
LlaVA-Video-7B
53.45
67.30
63.00
65.15
56.73
37.95
+Static SUK
49.94
74.55
85,00
79.77
59.39
38.73
+Stat + 50K 50K
59.28
74.13
82.08
78.11
62.34
40.36
LlaVA-Video-7B
53.45
67.30
63.00
65.15
56.73
37.95
+Static 100K
51.58
75.03
87.42
81.22
63.74
37.94
+Stat Dyn 100K
59.13
73.50
85,25
79.38
61.09
38.63
Table 11: Most frequent correct and incorrect words
Rank
Correct Words
Incorrect Words
Incorrect Words (excluding count)
stationery
4x4 iced wheeler donuts
2
pan
3
books
4
my
4x4
5
into
12
cookies
6
vase
tea poles office game dogs
7
straight
iced
8
magazine
10
9
signs
10
flowerpot
A.4 Extra ablations and analysis.
Analyzing the semantic concepts where models fail after SAT tuning. We wish to un- derstand the semantic concepts where SAT tuning doesn't suffice. Hence, we we analyze the words that most often occur in the failure cases in contrast to the correct cases and show them Table 11. Unsurprisingly, we see high fraction of count words with higher numbers than 2 being mostly present. If we exclude the coutn words, we see a higher fraction of outdoor words. This is also unsurprising since our simulator only has indoor scenes. Using world models and more diverse simulators to improve our dataset is an exciting area of future work.
Errors in relationship judgment compounded with lack of reasoning causes MLMs to perform worse than random on dynamic tasks For certain splits such as perspective, or overall performance 50%). In dynamic examples, the spatial position of an object often switches when new perspective is taken-egg, a cup might be in the left side of the image, but switches to the right side after taking a new perspective. Weaker MLMs have bad spatial perception in general. However, even for MLMs with strong static relationship performance, they often answer what is currently visible (i.e., cup is on the left) rather than what actually happens after the new perspective (i.e., cup on the right). Examples are shown in Figure 5. For example, let's look at the performance of a model which has strong static spatial performance Gemini-1.5-pro, If we look the second row example, the surfer is currently the left of the X marked point and hence the model answers left instead of reasoning about how might switch when viewing from that perspective. Similarly, in row 3, and in row 7 where the door is currently visible the left. While this analysis is from qualitative analysis, a more thorough analysis is left as future work.
SAT real, some accuracies are below random chance (random is
dynamic and video spatial reasoning As shown in Tables 8 and 9, adding in dynamic data helps LLaVA- on video spatial reasoning (Table 9) and on dynamic SÄT Real in Table 8, while maintaining performance
Adding in dynamic helps LLaVA-Vides
22

---

Published as a conference paper at COLM 2025
CVBench and BLINK. While scaling works better with mixing in dynamic shown in Figure in the main paper, we also analyze the effect of adding dynamic data while maintaining the same data size static in Table 10. We see that for both models, adding in dynamic helps significantly on SAT Real. For LLa it helps overall. For LLaVA- LLaVA-Video, it helps video and SAT real while maintaining performance BLINK and CVBench.
Quality of data is important. Hence, simulations are an effective source. We wish to answer if the gains trivially come from just simply learning the formats. Hence, we noise baseline where we flip the answers 20% of the time. We see performance degrades significantly. We see reduction of 10% on CVB and 18% on BLINK when training with just 20% noise. This means MLMs are not trivially learning the format of questions or object/attribute -ame--ased biases. Rather, the spatial answer quality is important.
23

---

Published as a conference paper at COLM 2025
Image(s)
Question
Answer
LLavA Answer
LLavA
Spro Answer
Answer
were any of the objects in the initial frame that you still see the second frame moved from their original positions?
chair was moved left away from the camera
For someone at the x marked point and facing 90 degrees to left, will person surfing be to their left right?
right
If stand where the and face the same her, is the truck to her left or right?
right
climbed lop mountain left, would the people in the scene appear bigger or smaller?
smaller
X
If stand in the shower facing faucol, will able see the towels without the mirror?
no
how should bears at the back rotate face bear in front? Rotate left by 90 degrees or right by 90 degrees?
Rolate left by 90 degrees
i am standing in the centre the room, facing camera. door left or right?
right
I rotate right move forward, will the window move closer to me?
yes
Figure 5: Some qualitative results of spatial question answering on some of our SAT Real Dynamic Benchmark
24

---

Published as a conference paper at COLM 2025
Which objjett disee to book, the box??
cbjecti
Where &s the bow locoted with respect to
How many persons are the mage?
box??
LLa
cycle Motoroyck
LLaVA:
SAT: There is only 220 person In the image, and he standing 1g nest to the elephant.
SAT
The
Where i the mouse /sed beol located respect laptopp/ right7
Which to cammea??
Air 17?
dining touching
Wheee
ball localed with renpact the persor?
Speckdl tuekereee
LLaVAc no
LLa above
SAT The diring coble
SAT The sports ball is Ehioacth person an they are hitlng the bal with ther turtis nechee.
suuching the cat thee
locil
aoiey
bicydo
X
LLaVA: SAT:
2 SAT:S
CT
SAn hhe bicycle EAS bux placte en markin froms ofthe bus
GT:
og
Figure 6: Some qualitative results of spatial question answering comparing baseline LLaVA on real benchmarks.
CVBench BLINK -AT real
80.0%
70.0%
60.0%
KEOM
50.0%
40.0%
30.0%
Freeze-Vis
Lora 256
Lora 512
Train Params
Figure 7: Effect of scaling up more tuning parameters. Tuning vision encoder is critical showing visual features current bottleneck for spatial reasoning. However, tuning more LLM parameters yields incremental returns.
25

---

Published as a conference paper at COLM 2025
Image(s)
Question
GT Answer
were any of the objects in the initial frame that you still in the second frame moved from their original positions?
Chair was moved left and towards camera
need to go gray secure safe (near the mark in the image). Which direction should turn to face the object?
Left by 50 degrees
If turn right by 38 degrees, will be facing away from SideTable (near the mark 2 in the image)?
Yes
The first image from the beginning of the video and the second image is from the end. How did the camera likely rotate when shooting the video?
Rotated Left
someone at "X" facing left by 90 degrees, will the armchair (mark 4) be to their left or right?
Right
Figure 8: Qualitative examples of our synthetic SAT dynamic data
26

---

Published as a conference paper at COLM 2025
Spatial Relationship
Depth Q: Which object closer camera, countertop garbage can?
Q: is wine bottle (mark A] to the left or right of dresser (mark C)?
Answer: garbage can
Answer: right
Figure 9: Qualitative examples of our synthetic SAT static data
Instructions
for partcippting in
For the imagett shown below, please answer the queetio to the best your abritty
There
depending
question seems ambigucus, answer you most liety Pleaze take your time and co the HIl's ailigenty Wi will be montoring for rapid random clicks and w reject you we find ev deeneee for rapid clicking.
Answer the question to the best of your ability for the image(s) shown.
Select the correct answer from the choices.
Firetinntial/Oriinnall Image
Second
Questior: The frist Image from the beginning of the video and the second Image from the endi How did the camera Ikayy move when shooting the video?
forward
rotated right and rotated right
Figure 10: Interface compute human accuracy for SAT.
27


Reviews:
[{"review_id": "6YIUJFl9ET", "reviewer": "Reviewer_Q2hV", "summary": "This paper proposes a Spatial Aptitude Training (SAT) dataset to train vision-language models (VLMs) that can handle static and dynamic questions about images and videos. Using the dataset, they fine-tuned the LLaVA-13B and LLaVA-Video-7B models and demonstrated that the fine-tuned models outperform the baselines and commercial VLMs (GPT-4o/v, Gemini1.5-flash, and Pro).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed dataset contains dynamic questions that are new to the community.\n- The experiment results show significant improvement compared to the baselines and commercial VLMs.\n- Good qualitative examples to show the performance of different fine-tuned models and baselines.", "reasons_to_reject": "- Besides the creation of synthetic datasets, the papers contain limited novelty in terms of the algorithm.\n- The dataset is created from CVBench, BLINK, and VSI-Bench with pseudo-annotated Q&As. The paper lacks a detailed description of the dataset and its comparison to other similar datasets.\n- No model novelty presented", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tXQcO2AS0C", "reviewer": "Reviewer_2SN7", "summary": "The paper investigates whether simulation data can help improve static and dynamic spatial reasoning ability in LLMs. A data generation approach named Spatial Aptitude Training (SAT) is proposed to generate data in simulation environments characterizing both static and dynamic spatial reasoning. Training on SAT data shows that it helps with multiple aspects of spatial reasoning and even helps long-video reasoning. Some ablation studies are carried out to reveal the effective part of the SAT data.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Extending spatial reasoning into dynamic settings is an important yet underaddressed direction. The data generation pipeline is carefully designed to address both static and dynamic scenes. Sampling data from simulation environments also mitigates the data scarcity issue in real-world VQA datasets. \n- Comprehensive experiments and insights: experimental results justify the effectiveness of SAT data and shows its improvement in multiple aspects. The ablation study also provides interesting insights such as the importance of dynamic scenes and hints of scaling with these data samples.", "reasons_to_reject": "- Limited Real-World Data: While SAT shows promising results in synthetic environments, the real-world test set is relatively small (150 image-QAs), which may not fully capture the complexities of real-world spatial reasoning. While there is also a synthetic split SAT-Synthetic, evaluation results on these two splits in Table 2 show different ranks and significant gaps in metric values. \n\n- Generalization to Diverse Environments: The paper could further explore how well models trained on SAT generalize to a wider range of real-world scenarios, including outdoor environments or dynamic human interactions. Besides, evaluation could be improved by comparing to a base model that is fine-tuned on a similar amount of spatial visual questions to reflect the uniqueness and superority of SAT over general VQA generation pipelines.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j5BUQ4VfX9", "reviewer": "Reviewer_MPJk", "summary": "This paper introduces SAT (Spatial Aptitude Training), a simulated dataset for training multimodal language models (MLMs) on spatial reasoning tasks. The authors create 175K QA pairs across 20K scenes, including both static spatial relationships and dynamic spatial reasoning tasks that involve reasoning about how egocentric and object motions affect spatial relationships. They also create a small real-image test set and evaluate on existing spatial benchmarks. Their results show that training MLMs on simulated data improves spatial reasoning capabilities on real images.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Important research direction that addresses a real gap in MLMs. Spatial reasoning is crucial for many applications but current models struggle with it, especially with dynamic spatial relationships. The authors make a good case for why this matters based on cognitive science literature.\n-  The results look impressive, with significant gains over baselines and sometimes even over proprietary models like GPT4-V and Gemini. Comprehensive evaluation across multiple datasets (both their own and existing ones) and model types. They test on four static spatial benchmarks, their own dynamic test set, and a video benchmark. The ablation studies reveal interesting insights.", "reasons_to_reject": "- Limited discussion of failure cases or analysis of where their approach breaks down. While they mention that camera movements remain challenging (lines 266-269), a deeper error analysis would help understand the limitations of simulation-based training.\n- Concerns about sim-to-real transfer and out-of-distribution scenarios. The paper doesn't adequately address how their simulated training data might not cover the diversity of real-world spatial scenarios. The test set is small (only 150 image-QAs), which makes me wonder about robustness.\n- The paper presents better performance on their own dataset than existing baselines, which is somewhat expected. While they do show gains on external benchmarks as well, it would be helpful to see a more detailed analysis of the types of questions/scenarios where their approach fails on those benchmarks.", "questions_to_authors": "- How well would your approach generalize to real-world applications with more complex scenes than those in your simulator? Have you tested on more challenging real-world images? How much do you think the improvements from dynamic data would transfer to embodied AI applications like navigation in the real world?\n- Your synthetic test accuracy is lower than real-world test accuracy for many models. Why do you think this is the case?\n- Have you considered using more diverse simulators beyond ProcTHOR to increase the variety of your training data?", "extra_scores": {}}, {"review_id": "5fCWLQzTLK", "reviewer": "Reviewer_VDnM", "summary": "This paper introduces new data for training and evaluating spatial understanding of MLMs.\n\nThe authors generate a large dataset with synthetic images, and collect a very small (150 QAs) test set with real images.\n\nThe goal is to improve and evaluate the abilities to understand static and dynamic (i.e. effect of motions) spatial properties.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n--\n\n(S1) Sound motivation. Fills a gap in the literature and space of available datasets.\n\n(S2) Scalable approach. The data is generated procedurally without manual per-example explicit supervision.\n\n(S3) The paper examines of the data, in particular which types of data improve different skills (static and dynamic understanding).\n\n(S4) The results show benefits on real-image benchmarks (despite the new training data being synthetic).\n\n(S5) Generated data validated by testing humand performance (Appendix A.3)\n\n---\n\nReasons to accept:\n\n- valuable contribution as detailed above\n\n- I do not see any clear flaw to this work, although I might have missed something (I am not an expert on this specific topic).", "reasons_to_reject": "Weakness/suggestion for improvementy:\n--\n\n(W1) What the authors generate are **QAs for instruction tuning**. This is only made clear in Section 3 but should be clarified earlier. I could imagine a similar pipeline to be used to generate structured annotations or full-text captions describing the same synthetic scenes.\n\n---\n\nMinor:\n--\n\nL148 \"*MLMs struggle with counting*\": the reference of (Goyal et al., 2017) doesn't seem appropriate/relevant, certainly not up to date.\n\nL263, 270, 288, 323, 329, etc. (also in the Appendix): need a dot at the end of the paragraph headers.\n\nL676: unresolved reference (\"*Figure ??*\")\n\nL680 \"*batch size of 8 (using gradient accumulation*\": is this 8 before accumulation? then what is the effective batch size?\n\nL695 \"*we most use ... to as ...*\" -> mostly ... as\n\nL696 \"*We see tha gains*\" ???\n\nL811: no dot at the end of the section title.\n\nPlease proof-read the whole appendix.", "questions_to_authors": "None.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces SAT, a dynamic spatial aptitude training dataset that requires reasoning about ego and object motion, extending beyond the simple static relationships found in existing datasets. All reviewers have expressed positive opinions about the paper. We suggest that the authors provide a more detailed description of the dataset and further explore its generalization to real-world applications involving more complex scenes. Given the context, we recommend a clear acceptance of this paper.
Decision: Accept