Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
Authors: Diankun Wu
Fangfu Liu
Yi-Hsin Hung
Yueqi Duan
Venue: NeurIPS 2025 spotlight ()
Keywords: Spatial intelligence
Multimodal large language model
3D Understanding
Primary area: deep_learning
TL;DR: In this paper, we present Spatial-MLLM, a novel framework for spatial understanding and reasoning with only 2D inputs.

Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced performance on 2D visual tasks. However, improving their spatial intelligence remains a challenge. Existing 3D MLLMs always rely on additional 3D or 2.5D data to incorporate spatial awareness, restricting their utility in scenarios with only 2D inputs, such as images or videos. In this paper, we present Spatial-MLLM, a novel framework for visual-based spatial reasoning from purely 2D observations. Unlike conventional video MLLMs which rely on CLIP-based visual encoders optimized for semantic understanding, our key insight is to unleash the strong structure prior from the feed-forward visual geometry foundation model. Specifically, we propose a dual-encoder architecture: a pretrained 2D visual encoder to extract semantic features, and a 3D spatial encoder—initialized from the backbone of the visual geometry model—to extract 3D structure features. A connector then integrates both features into unified visual tokens for enhanced spatial understanding. Furthermore, we propose a space-aware frame sampling strategy at inference time, which selects the spatially informative frames of a video sequence, ensuring that even under limited token length, the model focuses on frames critical for spatial reasoning. Beyond architecture improvements, we construct a training dataset from multiple sources and train the model on it using supervised fine-tuning and GRPO. Extensive experiments on various real-world datasets demonstrate that Spatial-MLLM achieves state-of-the-art performance in a wide range of visual-based spatial understanding and reasoning tasks.

Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
Diankun Wu
Fangfu Liu
Yi-Hsin Hung Tsinghua University
Yueqi Duan't f Tsinghua University
Tsinghua University
Tsinghua University
-Thidinng
clsee window From par klop tefu thold wolng anord the soll befera the bad alng
Example
and
You want
nurigato the hightttnd cleser the window. You will purffct hellwinggenn setions howw her exch jjleas itll in] chasse ether trom back' "sun left "urniigle']1 Sowwad unill the ou bel. (pleas in forward unti wall (pacee filli]] forwand until the nightstadd You
destiration
à Turn Lwf, Leb
C. Tum Bock Turalof
a.mum Tern
Turn Right Tur Left
Current Position?
Get whene? window.. How?
Walk wall window
nightssund
2 Seatiel MLLM
Tread
Lamket wen nuthed whichis affere
Absolute Diennce
Distance
Relative Birwttis
frast-right
Tlunking
Requsase with znower
Dewil
Toal La
Musded Performnncc sas Baradh
Figure 1: We propose Spatial-MLLM, method that significantly enhances the visual-based spatial intelligence of existing video MLLMs. As shown, Spatial-MLLM is capable of understanding and reasoning about the underlying scene from video input, achicving kate-of-the-as performance across wide range of tasks.
Abstract
Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced performance on 2D visual tasks. However, improving their spatial intelligence remains challenge. Existing 3D MILMS always rely on additional 3D or 2.5D data to incorporate spatial awareness, restricting their utility in scenarios with only 2D inputs, such as images videos. this paper, we present Spatial-MLLMM novel framework for visual-based spatial reasoning from purely 2D observations. Unlike conventional video MILMs which rely on CLIP- based visual encoders optimized for semantic understanding, our key insight is unleash the strong structure prior from the reedfforward visual geometry foundation model. Specifically,
propose dual-encoder architecture: pretrained 2D visual encoder to extract semantic features, and a 3D spatial mecoder-initializz the backbone of the visual geometry model to extract 3D structure features. A connector then integrates both features into unified visual tokens for enhanced spatial understanding. Furthermore, propose space-aware frame sampling strategy at inference time, which selects the spatially informative frames of a
from
"Equal Contribution. Conresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

video sequence, ensuring that even under limited token length, the model focuses on frames critical for spatial reasoning. Beyond architecture improvements, we construct training dataset from multiple sources and train the model on it using supervised tine-tuning and GRPO. Extensive experiments on various real-world datasets demonstrate that Spatial-MIIL achieves state-of-the-ar performance in a wide range of visual-based spatial understanding and reasoning tasks. Project page: https://diankun- wu. mou.githubb github..oo i Lto/SpaaiallMMnne
1 Introduction
Multimodal Large Language Models (MLLMs) I 2 3 have achieved significant progress in processing multimodal inputs to generate contextually aware and semantically coherent responses. While proprietary models such as Gemini [4] and GPT-4o [5] exhibit state-of-the-ar performance, the open-source community continues to advance the field by improving these models' ability to
interpret diverse content modalities, including images 6 7 S. videos 10 11
12 [3] [4], and
audio [15] 17. Although these models excel wide range of 2D tasks, their capacity perceive, understand, and reason about 3D scenes, i.e., 3D spatial intelligence, remains limited [8 19].
The requirement of spatial understanding and reasoning typically arises in two scenarios. In the first scenario, the model has access to additional 3D or 2.5D data (e.g.. point clouds, camera parameters, or depth maps) alongside 2D visual imputs (e.g., images videos). These supplementary modalities enhance the model's spatial awareness, enabling more accurate spatial reasoning. However, this setup limits the model's applicability in many real-world scenarios where only monocular video of the scene is available, which is the second scenario. The model's 's ability to perform spatial understanding and reasoning under such conditions is referred to as visual-based 3D spatial intelligence [18 20] A major challenge in this setting is that each frame provides only a partial observation of the scene, and no global representation (e.g., the point clouds 22 23 or posed depth maps (24 (25 is available
as input. This requires the model to infer the global spatial layout from incomplete cues and internally integrate these partial observations into coherent and implicit global representation, which demands strong spatial awareness. However, most existing video MLLMs pretrain their visual encoders on
following the CLIP (27 paradigm. This makes the visual encoder excel at capturing high-level semantic content but lack structure and spatial information when only 2D video inputs are available 28 29 30). Consequently, current video MLLMs generally perform worse on spatial reasoning tasks than on other tasks, such as temporal understanding. Moreover, their performance still significantly lags behind human capabilities [18].
image-text pairs- prrmmrily image-caption data [13 14 26
In this paper, we introduce Spatial-MLLM, a method that significantly improves the visual-based 3D spatial intelligence of existing video MLLMs. To address the limitations of visual encoders in general-purposs video MILMMs, our key insight is to unleash the strong structure prior provided by the feed-forward visual geometry foundation model [31 32 33 34). These models, typically trained on pixel-point pairs, complement the general-purpose purpose video MLLM visual encoders that are trained primarily on image-text data [14]. Based on this insight, we design dual-encoder architecture consisting of a 2D encoder- initializei from the visual encoder of a general-purpos. video MILM to extract 2D semantic information, and spatial encoder- leveraging the VGGT feature extractor 332 to recover implicit 3D structural information from 2D video inputs. We then use connector to integrate features from both branches into unified visual tokens. The resulting representation enables the Large Language Model (LLM) backbone to perform effective spatial reasoning without requiring explicit 3D data as input.
Furthermore, we fully exploit the additional information provided by the introduced feed-forward visual geometry model [32], and propose space-aware frame sampling strategy at inference time, which selects the most spatially informative frames from the video sequence when the total number of input frames is limited (e.g., due to the VRAM limitation). Specifically, we first feed a relatively large number of frames into the spatial encoder and decode the resulting 3D features into voxels. The frame selection task then reformulated as a maximum coverage problem over these voxels. which we solve using a greedy algorithm. To train Spatial-MLLM, we construct a iisual-based 3D spatial
dataset and perform supervised fine-tuning on it. We further apply a simple cold-start 35 to help the model adapt to the correct reasoning format, and then train it using Group Relative Policy Optimization (GRPO) Rd 35) to enhance its sofg--hhi--of-thuugee (long- CoT) spatial reasoning capability 37
We conduct extensive evaluations the VSI-Bench (18).

---

ScanQA [8), and SQA3D B3 benchmarks and demonstrate that the proposed Spatial-MLLM achieves state-of-the-- art performance in wide range of disual-based spatial understanding and reasoning tasks.
In summary, main contributions are:
We introduce -patial-MLLM, a method that significantly enhances the visual-based 3D spatial intelligence of existing video MILMs,, demonstrating strong spatial understanding and reasoning capabillities without requiring any 3D or 2.5D input.
We design dual-encoder and connector that effectively integrates semantic information from standard 2D visual encoder with structural information extracted by spatial encoder, which is initialized from a feed-forward visual geometry foundation model.
We fully exploit the additional information provided by the feed-forwe worddfo visual geometry model and design a space-aware aware frame sampling strategy that selects spatially informative frames, thereby improving model performance under input length constraints.
We train our model on the constructed dataset with a two-stage pipeline. Extensive experiments demonstrate that our method achieves state-of-the- performance on wide range of visual- based spatial understanding and reasoning tasks.
2 Related Work
2.1 MLLMs for Video Understanding
Multimodal Large Language Models (MLLMs) have made significant progress in integrating vision and language. Early works such as BLIP-2 [2] and Flamingo [1] introduce token-level fusion (e.g., g.. Q- Former) and feature-level fusion (e.g. cross-attention layers) to bridge modalities. Other approaches, including the LLaVA series 3 40), MiniGPT-4 [T.. and subsequent models [3 42 43]. leverage MLPs to project visual features into the language space. Recent advancements in MILMs have extended their capabilities from images to videos, typically by introducing video-language alignment through large-scat scale pretraining ( 44 Later models, such as Qwen2.5-VL. [14, enhance temporal reasoning via dynamic resolution and absolute time encoding. Although existing video MLLMs excel at capturing high-level semantics and temporal patterns, they struggle to interpret the underlying 3D scene from video input, which inspires our work to enhance their spatial understanding capabilities.
2.2
3D MLLMs for Scene Understanding
Recent advances in MLLMs have sparked interest in extending their capabilities from 2D to 3D scene understanding 23 2 25
45
48 49 50
51 52]. LL3DA 23 extracts scene-level
features from point clouds using a Q-Former, while Grounded 3D-LLM 45 integrates 3D detectors to generate object proposals. Methods like Chat3D [46). LEO (47) and Chat Shat-Scene 22 first segment 3D objects and encode object-centric features for fusion. Alternatively, 3D-LLM
48 and Scene- LLM 44] aggregate CL.IP features from pre-segmented multi-view object patches into 3D point representations, leveraging multi-view images and camera parameters. LLLVAA3D 24 projects 2D multi-view patch features into voxel space for 31D-aware aggregation, and GPT4Scene (50) enhances 3D reasoning by first reconstructing scenes and then using BEV images as input. While these methods advance 3D scene understanding, most of them require additional 3D or 2.5D inputs, which is difficult to acquire in real-world scenarios. In contrast, our approach only requires 2D videos as input.
2.3 Visual-based 3D Spatial Intelligence
Visual-based 3D spatial intelligence aims enable video MLLMs to perceive, infer, and reason about 3D structures and spatial relationships purely from 2D visual inputs. Despite recent advances, most existing video MILMs are still primarily designed for 2D understanding tasks, and their extension to visual-bar based 3D reasoning, e.g., 3D question answering (3] and robotic manipula- tion [54), remains relatively underexplored. To address this limitation, a new wave of specialized benchmarks has emerged to systematically evaluate the spatial reasoning capabilities of these mod-
els [8] 20 55 56 55] 58). Among them, VSI-Bench [88 serves as pioneering benchmark that
comprehensively assesses visual-spatial intelligence across multiple dimensions. STI-Bench (20)
3

---

Large Language Model
Anawer
Connector
Tukenizz
Encoder
23 Fesaree
coona
aig0a Kanae
8
[eforaan)
Longuage
"Mendaiig from the clese pain sajerry (hisls, aola ahool char). The cleat the datwaieer
Indes Selection
Spacesanee Frinco Samepl
tike sefo
2D Encoder
3D Spatial Encoder
Spediel
Inforesic
Figure 2: Overview of SLLtial-MLLLM. Our model is composed of a 2D visual encoder Exp, a 3D spatial encoder Espanial, which is initialized from feed-forward visual geometry foundation model, connector, and large language model backbone. At inference time, we incorporate space-awar aware frame sampling strategy to select spatially informative frames when the number of input frames is limited due to GPU memory constraints.
introduces physics-aware questions, such as velocity estimation, to quantify model's spatial and kinematic reasoning abilities. Ego-ST Bench (55) evaluates the model's s spatial intelligence from an egocentric perspective, while VLM4D [56] emphasizes motion dynamics, such trajectory prediction, to probe 4D spatiotemporal interactions. Collectively, these benchmarks signify a shift toward more holistic evaluation of visual-based spatial intelligence in video MIL.Ms.
3 Method
In this section, we introduce Spatial-MLLM. Given video of N frames depicting a scene, denoted as V == [fr)/=]. where f; E #W x3 Spatial-MLLM is designed to understand spatial relationships, perform spatial reasoning, and generate appropriate responses. We begin by describing the model architecture in Section 3.1 which comprises a 2D visual encoder, a 3D spatial encoder, a connector, and large language model backbone. Then we present the space-aware frame sampling strategy in Section 3.2 which selects N& spatially informative frames { (f/) J ;= where N₂ & N. Finally, we introduce the training dataset construction process and two-stage training pipeline in Section 3.3
3.1 Spatial-MILLM Architecture
In this section, we present the architecture of Spatial-MLLM, which shown in Figure 2 We adopt Qwen2.5-VL-3B [14] as our base model and explore strategies to enhance its spatial understanding and reasoning capability. Before diving into the details, we first briefly introduce the key insights that motivate our design.
What hinders visual-based spatial intelligence in existing video MLLMs? Existing video MLLMs 14 13 12 typically employ a pre-trained 2D visual encoder &aD to extract 2D patch features e2p- These features then projected into visual tokens through lightweight connection module. A large language model backbone Se subsequently generates the final response by conditioning on both visual and textual tokens. A critical bottleneck in this process lies in the nature of the visual features extracted. The required type of information varies by task: high-level semantic representations are essential for 2D recognition and understanding, whereas fine-grained structural cues are crucial for spatial reasoning. However, the visual encoders used in current video MLLMs are primarily pre- pre-trained on image-text datasets (mainly image-caption pairs) 14 26 following the CLIP [27] paradigm. As result, these models predominantly capture semantic content and often lack spatial awareness when no additional 3D or 2.5D data are available (28
29
30 To address this, our key
insight is to unleash feed-forwart visual geometry foundation models 32 which are trained on
pixel-point pairs and can recover rich 3D structural information from 2D inputs, which complements the semantic features extracted by the 2D visual encoder. We design dual-encoder architecture that

---

exploits the strengths of both models and a connector to fuse semantic and structural information into unified visual tokens. Below, we introduce the core components of our design.
Dual-Encoder:
The proposed dual-encoder consists of a 2D encoder & and 3D spatial encoder Epaaiall For the 2D encoder branch, we adopt the same design the visual encoder of Qwen2.5- to encode input frames into semantically rich features;
VL [4]
e2n = E2D (()))
e2D E R
No
x
where P2D and dap denote the patch size and feature dimension of the 2D visual encoder, respectively, The two consecutive frames are grouped for video input, thus NNx [[Nx/2].
For the spatial encoder branch, we utilize the feature backbone of VGGT (32). Specifically, given N&
frames of the scene video, we first patchify the input and then extract 3D features with alternating
frame-wise self-attention and global self-attention [59]. This process allows Expatial to aggregate
spatial information across different frames to get the final 3D features:
C3D; Cc. Oregister = Espusial ((E))=²))
X
C3D E
(2)
where e3D. e.. and eregister represent the dense 3D feature, the camera feature for each frame, and the register tokens [60], respectively. We only use eap in the feature fusion stage it captures the dense structure information of the input frames.
Connector. After obtaining the 2D and 3D features, we use connector to integrate the semantic and structural information from both branches. Specifically, we first align esp with eap in both spatial and temporal dimensions:
X
e Rearrange(es)); eg E
Here, the spatially and temporally adjacent information in esp is aggregated into the feature channel dimension, enabling alignment with e2p. Next, we employ a lightweight connector to fuse the information to obtain the unified visual tokens:
Connector(eeD. egp);
where e E Sxdum denotes the final visual tokens and 5 = N₂' x
x
the sequence Although more complex
W
length. In practice, we adopt MLP--based design (detailed in Section B.2
feature fusion methods, e.g., cross-attention (59, 51], could be applied, we find that this approach effective to enhance the model's spatial understanding and reasoning capabilities. We leave the exploration of more advanced fusion strategies for future work.
3.2 Space-Aware Frame Sampling
Due to GPU memory constraints, video MLLMs can process only limited subset of frames from scene video sequence. For example, in the VSI-Bench setup [18] only 8 to 32 frames are sampled as input the video MLLM, while typical scene video in VSI-Bench contains over 2,000 frames. A widely adopted solution is uniform frame sampling [13] 14 I8), which is effective for general- purpose video understanding. However, as spatial videos represent 3D scenes, the sampling strategy for spatial understanding tasks should focus on capturing most information of the underlying scene, which uniform sampling fails achieve.
Benefiting from the feed-forward visual geometry foundation model, we design a straightforward space-aware frame sampling strategy at inference time. Specifically, given scene video V {f. (f our objective to select N₁ frames, {f.h (r]),N ) that have most coverage of the underlying scene. To achieve this, we first uniformly subsample Nm, frames, {f." 1 No. where Nn. satisfies N₂- < Nm < N, li=1,
and is determined by the available GPU memory. In practice, we choose N,, 128 and N₂ 16. We then leverage EaD to extract their corresponding 3D features e and camera features e._ Subsequently, use the pretrained camera head fa and depth head fa of the VGGT model 32 to decode a set of camera parameters and depth maps:
E.
K.
J=
No. Nn fe(e.), and (D J Nm
fa(ew)
(5)
5

---

This allows us to calculate the voxels (f." covered by each frame f." and formulate frame selection maximum coverage problem 662 i.e. select N& frames {f,5 Jiml S { N.
Jizl such that the total number of unique covered voxels Nv V(f. is maximized. In practice, we apply greedy algorithm to accelcrate computation [63 25
Once the N₂ frames are selected, it is not necessary to recompute their 3D features ešD and the corresponding features from the precomputed set esp can be directly reused. We provide the complete algorithm and detailed explanation in Section B.1
3.3 Training
Training Data Construction. We first construct visual-based 3D spatial question-answering dataset. The dataset has approximately 12k QA pairs and constructed from three sources: the training set of ScanQA [38. SQA3D [39], as well as additional self-created spatial QA data. All items in our training dataset are derived from scenes in the training set of ScanNet [64] and are each represented as quadruple I (Qr, A,, V., MM,). denoting the question, answer, video ID, and meta-information (e.g., task type). respectively. For the self-created QA data, we follow the data processing pipeline proposed in VSI-Bench [18]. Specifically, we first convert ScanNet scenes into continuous video clips at 24 FPS and 640 x 480 resolution. Then we generate spatial reasoning QA pairs leveraging of Scannet. The generated QA pairs cover various spatial understanding and
the meta-annotatio reasoning tasks, including object counting, object size, room size, absolute distance, appearance order, relative distance, and relative direction. Since the QA pair construction process is similar to that of VSI-Bench [88) we exclude the QA pair I, if its scene video V; is used in VSI-Bench (these videos sourced from the validation of Scannet)
prevent data leakage. Finally, the self-created data contains approximately 70k QA pairs in total. We provide additional details on shows brief summary of key statistics of the
training data construction in the section B.3 Figure 3
training dataset.
Supervised Fine-tuning. Leveraging the constructed training dataset, we first perform supervised fine-tuning (SFT) our model. Since Eno and Espatial are pre-trained on large-scale image-text and pixel-point pairs, respec- tively, we freeze them preserve their ability to extract rich semantic and structural information. We jointly train the connection module and the LLM backbone enable the model to adaptively fuse 2D and 3D features and en- hance its spatial understanding and reasoning capability. During this stage, we employ the standard cross-entropy loss Lee between the model-generated answers and the ground-trun -tuth annotations:
SR/AD 21.27%
ScanoA 21 25%
ScanQA
Spatial-MLLM 120K
SQA3D
Self-Created
Self-Created
57.47%
Appest Order
Rel. Dir.
15.32%
Rel./Abs. Dist.
(0) = -
- E'ogoo log ((ol)) (1)--1)
,q. (G)
Ste gomm Oite 25,9%%
Self- Created Data
Appear. Order
Room/Obj. Size
where (f.) N2 denotes input video frames, denotes the system prompt and question, (0 represents the i-th to- ken in the ground-truth answer, and (1:--1) denotes the preceding answer tokens.
50.60% Als
Obj. Cnt.
Figure 3: Basic statistic of training dataset.
RL Training. Following the SFT stage, we first perform a simple cold start 35 to help the
model adapt the correct reasoning format. Then we train the model using Group Relative Policy Optimization (GRPO) [36] to enhance its long-Co'l (37) spatial reasoning capability. During training, we first sample a set of output (01. .2. .G} for each question g from the policy model Tea.. Then we optimize the policy model by maximizing the following objective:
1
G Σ
To(os 4) A., clip( (ol 9) To (o: 9)
+t)).)) (Fol
JGRPO Egq.o
min
(7)
G
Now (o: 9)
where A, mwwn(rcannn....mmm (taamnrrraaa...... Pal is the advantage function computed using the group rewards.
ad(rr1,"2.....1
In GRPO, the design of the reward function is critical. addition to formatting reward applied to all task types, introduce task-dependent reward modeling to ensure that it accurately reflects

---

Table I: Evaluation Results VSI-Bench [18]. For Spatial-MLLLM and Qwen2.5-VL. series
use 16
frames as input and report miero average scores. For other open open-source methods GPT-4o []. the number of frames is the same as VSI-Bench setting (ranging from to 32 frames). For Gemini- 1.5 Pro [4) samples
video frames at FPS. Bold and underline denote the best-performing and second- weaddd--ss-ssfrrteee performing open-soures models, respectively.
Methods
Numerical Question
Multiple-Chuice Question
Avg. Rank
Obj. Room Size
RRl..
Route Plan Appr. Order
Proprictory Models
GPT-4o [5]
46.2
5.3
43.8
38.2
37.0
41.3
31,5
28.5
34.0
Gemini
Open-tource source Models
V.tem0112-4088
46.5
31.8
42.1
34,0
39.6
36.0
LongVILA-SB
29.1
9.1
16.7
0.0
29.5
30.7
32.5
25.5
21.6
VILA--155408 67
22.4
24.8
48.7
22.7
40.5
25.7
31,5
32.9
31.2
LongVA-7B
22.2
LLaVA-Oneeisioon 6
OneVision
43.5
23.9
57.6
37.5
42.5
39.9
32.5
44.6
40.2
48.9
22.8
37.4
35.3
42.4
36.7
35.0
48,6
40,9
Spotial-MII.M
Qwen2.5-VL-Srri
Qwen2.5 L-B-2.5-VL-3B [4
24.3
24.7
31.7
22.6
38.3
41.6
26,3
21.2
30.6
10
Owen2.5.VL-7B 14
40.9
14.8
43.4
10.7
38.5
38.5
33.0
29.8
33.0
Qwen2.5-V1
29.3
57.9
29.4
41.7
37.0
Spatial-MLLMM-8
65.3
34.8
63.1
45.1
41.3
46.2
33.5
46.3
48.4
the proximity between the predicted and ground-truth answers. Specifically, we categorize the data into three types based on answer format: numeric answer questions, multiple-choice questions, and verbal answer questions. For numeric questions, we compute the mean relative accuracy [18]. For multiple-choice questions, we employ an exact match reward. For verbal answer questions, we use fuzzy matching based on Levenshtein distance. Further details on reward calculation are provided in Section B.5
4
Experiments
4.1
Implementation Details
Training details. Spatial-MLLM is built on Qwen2.5-VL [14 and VGGT [32 and has approximately 4.9B parameters in total. We use the visual encoder of Qwen2.5-VL [1] to initialize εTD, and the LLM backbone of it to initialize fe. We then use the feature backbone of VGGT 332 to initialize Espatill During training, we use 640 x 480 resolution and limit video frames to 16. In the SFT stage, train the model using Adam optimizer [65] for epoch. We set the global batch size to 16 and use a linear learning-rai rate schedule, with a peak value of 10 In the cold start stage, we first construct small CoT dataset. Specifically, we prompt Qwen2.5-VL--73
[14] to generate multiple thinking processes and answers according to the scene video and question. Then we use the GT answer filter correct thinking-answer pair (more details provided in Section B.4h. We use similar setting as in the SFT stage to train the model for 200 steps. In the RL stage, we perform rollouts per question and the default sampling temperature 1. The KL divergence coefficient, B, is set to 0.04. Due to computational resource limitations, we train the model for 1,000 steps with a learning rate of 10-6 We show the training curve of SFT Stage and RL Stage in Figure
Inference Details. During inference, we set Nm,. = 128 and N₂- = 16 for aware frame sampling. Since spatial reasoning requires certain level of determinism, set the temperature to 0.1 and the top-p to 0.001. The default input resolution from the scene video is 640 x 480.
4.2 Comparisons on VSI-Bench
Setup. VSI-Bench 18 contains more than 5,000 question-answer pairs derived from egocentric videos sourced from ScanNet [64]. ScanNet++[69), and ARKitScenes 77 The task types are divided into Multiple-Chon Choice Answer (MCA) and Numerical Answer (NA). For the MCA tasks, we compute mean accuracy, and for the NA tasks, we calculate relative accuracy across confidence thresholds (0.5, 0.55 ... 0.95). We report the final average score and individual metrics on eight task types of VSI-Bench, including: (1) configurational reasoning tasks (object counting, relative direction, absolute direction, and route planning), (2) measurement estimation tasks (object size, room size, and absolute distance), and (3) spatiotemporal reasoning tasks (appearance order). For Spatial-MLLM

---

Table 2: Evaluation Results ScanQA 38 and SQA3D
We the val of ScanQA and test of Bold and underline denote best pest-perfornng and second-best best--erforrinng models in each model category. respectively.
SQA3D for evaluation following common practice +7
ScanQA (val)
SQA3D (test)
Methods
Video-Input Only
BLEU-1 BLEU-4 METEOR ROUGE-L CIDEr EM-1 EM-RI
Task-Specific Models
ScanQA 38
30.2
10.1
13.1
33.3
64.9
47.2
SQA3D 39
30.5
11.2
13.5
34.5
46.6
3D-Vista
13.9
35.7
48.5
X
3D/2.5D-Input Models
3D-LLM [48]
39.3
12.0
14.5
35.7
69.4
LL3DA (23
13.5
15.9
37.3
76.8
Chat-Scene
22
43.2
14.3
18.0
41.6
87.7
$4.6
$7.5
3D-LLaVA
21
17.1
18.4
43.1
92.6
54.5
56.6
Video-3D LLM
25
47.1
16.2
19.8
49.0
102.1
58.6
X
Video-Input Models
Qwen2.5-V1-338 [14
26.4
33.2
62.7
43.4
45.9
Owen2.5-VI--78 14
26.2
12.7
34.2
64.9
46.5
49.8
Owen2.5-VL-72B [4]
26.8
35.2
66.9
47.0
50.9
LLaVA-Videos Video-7B 12
39.7
3.1
44.6
88.7
48.5
Oryx-34B [33
38.0
37.3
72.3
Spatial-MLLLM-BB
44.4
45.0
91.8
55.9
58.7
and Qwen2.5-VL. series, we report micro average scores in Table 1 and macro average scores in Table
Baseline Models. We compare our model with a broad range of video-input MILMs. For proprietary model, we include GPT-4o [5] and Gemini-1.5 Pro [4] for comparison. For open-source video-input
MLLMs, we compare our model with InternVL2 LLaVA-NeXT-VE
LongVILA [66], VILA [67, Long VA (68),
12]. LLa VL-VV-OneVisio 6 and the V
14 series. The parameter
count of the baseline models is reported in Table
Results. We present the quantitative results on VSI-Bench [18 in Table | and Table Despite having 4.9B parameters, SpLtial-MLLM significantly outperforms all proprietary and open-source MLLMs. including those with substantially larger parameter counts (e.g. 32B or 72B). Among the remaining models, the best- rest-performing one is the proprietary Gemini-1.5 Pro 4 Notably, Spatial-ML. MLLM
provided with only 16 input frames per video, while Gemini-1.5 Pro samples videos at FPS (i.e.,
an average of 85 frames per video on VSI-Bench) according to its API instructions [18]. Despite the
significantly lower number of input frames, Spatial-MLLM still achieves higher average accuracy than Gemini-1.5 Pro 4.
4.3 Comparison on ScanQA and SQA3D
benchmarks built upon ScanNet 64 Since the authors did not provide a test set for ScanQA, we evaluate using the validation set, which consists of 4,675 QA pairs focused on understanding spatial relationships such object alignment and orientation, as well as the ability accurately identify objects in 3D scenes based on textual questions. We follow standard practice 25 50 by evaluating answer quality using the following metrics: CiDEr, BLEU-1, BLEU-4, METEOR, and ROUGE-L. For SQA3D, we evaluate the model on its test set, which contains 3,519 QA pairs. The task requires the model to first understand its position and orientation within the 3D scene, as described by text, then reason about its environment and answer question under those conditions. Since SQA3D contains definitive answers, we use exact match accuracy (EM) and refined version (EM-R) as evaluation metrics. We provide the evaluation results using additional metrics for both benchmarks in Section C
Setup. ScanQA 38 and SQA3D 39 are two 3D
Baselines. Since both the ScanQA [8 and SQA3D 39 benchmarks provide additional 3D anno- tations (e.g., point clouds and depth maps of the scene), we compare Spatial-MLLM, with several other model types addition video-input MILM. These includes task-specific models designed for 3D question-ansserrigg tasks, such as ScanQA (38), SQA3D 39 3D-V 77., and LLMss that require point clouds or depth maps as input, such as Chat-Scene Video-3D LLM and 3D-LLaVA 21

---

Figure 4: Visualization of Training Curves in the SFT and RL Stages. For SFT stage, we present the mean loken accuracy and loss curves. For the RL stage, we show the dynamics of completion length and reward.
Results. We present the quantitative results on the ScanQA [38] and SQA3D [39] benchmarks in Table 2 As shown, Spatial-MLLM significantly outperforms all video-input models across all metrics both ScanQA and SQA3D. Our model also surpasses all task-specific models. Among models utilizing 3D or 2.5D input, only VD-LLaVA [21] (on ScanQA) and Video-3DD 3D-LLM [25] (on ScanQA and SQA3D) achieve better performance than Spatial-MLLM However, 3D-LLaVA requires additional point cloud input, and Video-3D-LLM depends on depth maps. Despite not relying on any additional 3D 2.5D input, our model still outperforms other 3D-dependent models
such as 3D-LLM 48, LL3DA 23
and Chat-Scene 22
4.4 Ablation Study and Analysis
Ablation on Input Frame Number. We evaluate the effect of the number of input frames on VSI-Bench across different models, including Spatial-MLLM, Gemini-1.5 Pro 14) and Qwen2.5-VL. 3B [14]. The result shown in Table. For the 1 fps setting of Gemini-1.5 Pro, we upload the entire video to the model following the VSI-Bench [88. where the video is sampled at fps according to the API instructions. For the 0.1 fps and 0.25 fps settings, we first manually sample the video frames and then upload these sampled frames to the model. As shown, all models exhibit improved performance as the number of input frames increases, particularly when the number of frames is small.
Effectiveness of Space-aware Frame Sampling. We evaluate different frame sampling configu- rations in Table including 8, 16, and 32 frames using uniform sampling and space-aware frame sampling. As shown, increasing the number of sampled frames improves performance for both space-aware frame sampling and uniform sampling. Compared with uniform sampling, space-aware frame sampling consistently outperforms it when the number of input frames is the same.
We further provide a visualization of our space-aware frame sampling in Fig. [] which shows the point maps (predicted by the VGGT (32)) corresponding to the frames selected by different sampling strategies. As shown, the proposed space-aware frame sampling strategy consistently yields more spatial coverage than uniform sampling, which often overlooks transient regions that appear briefly in the video and tends to produce redundant viewpoints when the camera remains static.
Effectiveness of RL Train- ing. We evaluate Spatial- MLLM's performance be- fore and after GRPO train- ing on VSI-Bench. The re- sults are presented in the second (SFT GRPO) and third (SFT) rows of Table As shown, even though we conduct only small scale RI. training (i.e., 1,000 steps), the GRPO-trained model still achieves performance gains, suggesting that long chain-of-thought reasoning enhances the spatial reasoning capabilities required by VSI-Bench (18).
Table 3: Ablation Study. We report micro average results for numerical ques- tions and multiple-choiee questions on VSI-Bench [18 in different settings.
Methods
Numerical Multiple-Choice Avg.
Spatial-MLLA
52.7
43.8
48.4
Spatial-MLLM (w/o sa sampling)
$1.6
42.3
47.1
Spatial-MILLM (w/o sa sampling & GRPO)
51.5
40.4
46.1
Qwen2.5-Vi We-32.5-VL-3BB (SFT)*
49.2
40.3
44.9
Qwen2.5-VT. VL- (SFT)
47.1
32.6
40.0
Qwen2.5-VL-3B
26.9
34.4
30.6
Effectiveness of the Spatial-MLLMM Architecture and Training Dataset. We compare the su- pervised fine-tuned version of Spatial-MLLMM two supervised fine-tuned versions of Qwen2.5-VL. 3B 14 (the base model of Spatial-MLLM) and original Qwen2.5-VL-33 VL-3BB model in Table denotes results obtained with the R1-V 72 training framework. & denotes results which we further apply question token mask during the loss computation process within RI-V (72), which aligns better with MPLL training process. As shown both SFT versions of Qwen2.5-VL-3B

---

Uniform Frame Sampling
Uniform Sampling
Space-aware Sampling
Space-aware Sampling
Figure Visualization of different frame sampling strategies. For clarity of visualization, we set Nn. 128 and N₂ 8 in the visualization example.
Table 4: Ablation on Input Frame Number. We report micro average results VSI-Bench [8] using
different input frame numbers and frame rates (FPS), For Gemini-1.5 Pro, the input frame number is averaged over questions.
Methods
Frames
FPS
Numerical
Multiple-Choic
Avg.
8
N/A
50.8
41.2
46.1
Spatial-MLLM
16
N/A
$2.7
43.8
48.4
32
N/A
53.1
45.3
49.3
Spatial-MLLM
8
N/A
48.2
39.2
43.8
16
N/A
51.6
42.3
47.1
(wlo sa sampling)
32
N/A
52.4
44.2
48.4
12.2 (avg.)
0.1
43.1
35.7
39.5
Gemini-1.5 Pro 4
29.6 (avg.)
0,25
48.8
37.8
43.5
117.1 (avg.)
1
49.7
44.0
46.9
8
N/A
20.2
33.1
26.5
Qwen2.5-VL. VL-3B
14
16
N/A
26.9
34.4
30.6
32
N/A
28.3
35.7
31.9
show improvements, indicating the effectiveness of our proposed dataset to enhance the model's spatial reasoning capabilities. Furthermore, both models underperform compared to the supervised fine-tuned version of Spatial-MLLMM which validates the effectiveness the proposed architecture.
Conclusion
We introduce Spatial-MLLMM, a method that enables effective spatial understanding and reasoning from purely 2D visual inputs. By combining a semantic 2D encoder with a structure-aware spatial encoder initialized from visual geometry foundation model, our dual-encoder design captures both semantic and spatial cues. Additionally, our proposed space-aware frame sampling strategy further enhances performance under limited input constraints. Trained on the collected dataset, our model achieves state-of- the-art results across multiple benchmarks.
Limitations and Future Work. Although Spatial-MLLA, demonstrates significant improvements over previous video MLLMs across a wide range of visual-based spatial understanding and reasoning tasks, there remains room to scale Spatial-MLLM further in terms of model size and training data, Moreover, as this work primarily addresses visual-based spatial intelligence, we have trained and evaluated our model specifically on relevant datasets and benchmarks. An interesting direction for future work would to explore how integrating spatial structural information might further benefit general video understanding and reasoning tasks.
10

---

Acknowledgments
This work was supported in part by the Beijing Natural Science Foundation under Grant L252011, and by the National Natural Science Foundation of China under Grant 62206147.
References
[1] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al., "Flamingo: visual language model for few-shot learning," NeurlPS, 2022.
[2] Li, D. Li, S. Savarese, and Hoi, "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models," in ICMI,, 2023.
[3] H. Liu, C. Li, Q. Wu, and Y. J. Lee, "Visual instruction tuning," " NeurIPS, 2024.
[4] G. Team, P. Georgiev, V. L. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, al., "Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context," arXiv preprint arXiv:2403.055.0 2024.
[5] A. Hurst, Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al.. "Gpt-40 system card," arXiv preprint arXiv:2410.21276, 2024.
[6] B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang. Y. Li, Z. Liu, and C. Li, oneviinon Easy visual task transfer." arXiv preprint arXiv:2408. 03326, 2024.
[7] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al., "Intervv:: Scaling up vision foundation models and aligning for generic visual-linguistic tasks," in CVPR, 2024.
[8] J. Bai, S. Bai. S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, "Qwen-vl: A frontier large vision-language model with versatile abilities." arXiv preprint arXiv:2308 10088.2940 12966. 2023.
[9] B. Lin. B. Zhu, Y. Ye, M. Ning. P. Jin, and L. Yuan, Vidco-lllvaa Learning united visual representation by alignment before projection," arXiv preprint arXiv:2311 10122, 2023.
[10] Z. Cheng, S. Leng, H. Zhang, Y. Xin, X. Li, G. Chen, Y. Zhu, W. Zhang, Z. Luo, D. Zhao, et al., "Videollama 2: Advancing spatial- spatial-temporal modeling and audio understanding in video-llms," arXiv preprint AXXv:2406.004720 2024.
[11] R. Qian, X. Dong, P. Zhang. Y. Zang, S. Ding, D. Lin, and Wang, "Streaming long vidco understanding with large language models," arXiv preprint arXiv:2405. /6009, 2024.
[12] Y. Zhang, J. Wu, W. Ii, B. Li, Z. Ma, Z. Liu, and C. Li, "Video instruction tuning with synthetic data," ArXir, vol. abs/2410.027113 2024.
[13] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K.-Y. Chen, X. Liu, J. Wang. W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin, "Qwen2-vl: Enhancing vision- langua-a model's perception of the world at any resolution." ArXiv, vol. 095/2449.121911.. 2024.
[14] S. Bai, K. Chen, X. Liu, J. Wang. W. Ge, S. Song. K. Dang, P. Wang. S. Wang. J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang. W. Ding. Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, 2025.
Z. Yang, H. Xu, and J. Lin, "Qwen2.5-vl technical report," " ArXiv, vol. abs/2502.
[15] R. Huang, M. Li, D. Yang, J. Shi, X. Chang. Z. Ye. Y. Wu, Z. Hong, J.-B. Huang. J. Liu, Y. Ren, Z. Zhao. and S. Watanabe, "Audiogpt: Understanding and generating speech, music, sound, and talking head," ArXiv, vol. abs/2304. 12995, 2023.
[16] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. W.L.,, L. Lu, Z. Ma, and C. Zhang, "Salmonn: Towards generic hearing abilities for large language models." ArXiv, vol. abs/2310. 132899 2023.
[17] Z. Liu, Y. Dong, J. Wang, Z. Liu, W. Hu, J. Lu, and Y. Rao, "Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment," ArXiv, vol. abs/2502.00438.. 2025.
[18] J. Yang, S. Yang, A. W. Gupta, R. Han, F.-.. Li, and S. "Thinking in space: How multimodal large language models see, remember, and recall spaces," ArXiv, vol. 141721 2024.
[19] B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Driess. P. Florence, D. Sadigh, L. J. Guibas, and F. Xia, "SpatialvIm: Endowing vision-languag models with spatial reasoning capabilities." 2024 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp.
2024.
11

---

[20] Y. Li, Y. Zhang. T. Lin, X. Liu. W. Cai, Z. Liu, and B. Zhao, "Sti-bench: Are milms ready for precise spatial-temporal world understanding?." ArXiv, vol. abs/2503.23765, 2025.
[21] J. Deng, T. He, I.. Jiang. T. Wang. F. Dayoub, and I. Reid, "3d-llava: Towards generalist 3d Imms with omni superpoint transformer"" ArXiv, vol. abs/2501.01163, 2025.
[22] H. Huang, Y. Chen, Z. Wang, R. Huang, R. Xu, T. Wang, L. Liu, X. Cheng, Y. Zhao, J. Pang, et al., "Chat-scene: Bridging 3d scene and large language models with object identifiers," in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[23] S. Chen, X. Chen, C. Zhang, M. Li, G. Yu, H. Fei, H. Zhu, J. Fan, and T. Chen, "I.Ida: Visual interactive instruction tuning for omni-3d understanding reasoning and planning," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26428-26438 2024.
[24] C. Zhu, T. Wang, W. Zhang, J. Pang, and X. Liu, "Llava-3d: simple yet effective pathway to empowering Imms with awwaaereess,," arXiv preprint 24..::2409.18222 2024.
[25] D. Zheng, Huang, and L. Wang, "Video-3d NI: Learning position-aware video representation for 3d scene understanding." ArXiv, vol. abs/2412.00493, 2024.
[26] S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smymis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang, F. Orgad, R. Entezari, G. Daras, S. Pratt, V. Ramanujan, Y. Bitton, K. Marathe, S. Mussmann, R. Vencu, M. Cherti, R. Krishna. P. W. Koh, O. Saukh, A. J. Ratner, S. Song, H. Hajishirzi, A. Farhadi, R. Beaumont, S. Oh, A. G. Dimakis, J. Jitsev, Y. Carmon, V. Shankar, and L. Schmidt, "Datacomp: In search of the next generation of multimodal datasets," ArXiv, vol. abs/2304. 14108, 2023.
[27] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, G. Krueger, and I. Sutskever, "Learning transferable visual models from natural language supervision." in International Conference on Machine Learning, 2021.
[28] Z. Tang, L. Lian, S. Eisape, X. Wang, R. Herzig, A. Yala, A. Suhr, T. Darrell, and D. M. Chan, "Tulip: Towards unified language-image pretraining." ArXiv, vol. abs/2503. 2025.
[29] Qi. Liu, H. Tang, and Z. Zhu, "Beyond semantics: Rediscovering spatial awareness in vision-language models," ArXiv, vol. abs/2503. 2025.
[30] B. Zhang, P. Zhang, X. wen Dong, Y. Zang, and J. Wang. "Long-clip: Unlocking the long-text capability of clip." in European Conference on Computer Vision, 2024.
[31] S. Wang. Leroy, Y. Cabon, B. Chidlovskii, and J. Revaud, "Dust3r: Geometric 3d vision made easy." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20697-20709, 2023.
[32] J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny, "Vggt: Visual geometry grounded transformer," in Proceedings of the IEEE/CVF ICnfeeCCCConfeennc on Computer Vision and Puttern Recognition, 2025.
[33] Li, R. Tucker, F. Cole, Q. Wang, L. Jin, V. Ye, A. Kanazawa, A. Holynski, and N. Snavely, "Megasam: Accurate, fast, and robust structure and motion from casual dynamic videos." ArXiv, vol. abs/2412.04463, 2024.
[34] Y. Wang, J. Zhou, H. Zhu, W. Chang, Y. Zhou, Z. Li, J. Chen, J. Pang, C. Shen, and T. He, "x3: Scalable crnnuutatio-equivarinnn visual geometry learning," " 2025.
[35] DeepScek-At, D. Guo, D. Yang. H. Zhang, J.M Song, R. Zhang, R. Xu, Q. 7hu, S. Ma, P. Wang. X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou. Shao. Z. Li. Gao, Liu. Xue, B.-L. Wang, B. Wu.
B. Feng. C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen. D.-L. Ji, E. Li. F. Lin, F. Dai,
F. Luo, G. Hao, G. Chen, G. Li, H. Zhang. H. Bao, H. Xu, H. Wang. H. Ding, H. Xin, H. Gao. H. Qu, H. Li,
J. Guo, J. Li, J. Wang, J. Chen, J. Yuan, J. Qiu, J. Li, J. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang. L. Zhao, L. Wang, L. Zhang, L. Xu, Xia, M. Zhang,
M. Zhang, M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q Wang, Q Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, J. Chen, R. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S.-K. Wu, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng. W. Zhao. W. Liu, W. Liang, W. Gao, W.-X. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang. X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X.-C. Shen, Chen, X. Sun, X. Wang. X. Song. X. Zhou, X. Wang. X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li. Y. Zhao, Y. Sun, Y. Wang. Y. Yu, Y. Zhang, Y. Shi, Y. Xiong. Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma,
Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y.-. Zou, Y. He, Y. Xiong, Y.W. Luo, Y. mei You, Y. Liu,
12

---

Y. Zhou, Y. X. Zhu, Y. Huang, Y. Li. Y. Zheng. Y. Zhu, Y. Ma, Y. Tang. Y. Zha, Y. Yan, Z. Ren, Z. Ren,
Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. guo Zhang, Z. Hao, Z. Ma. Z. Yan, Z. Wu, Z. Gu, Z. Zhu. Z. Liu, Z.-A. Li,
Z. Xie, Z. Song, Z. Pan. Z. Huang. Z. Xu, Zhang, and Zhang, "Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning," ArXiv, vol. abs/2501.12245, 2025.
[36] Z. Shao, P. Wang, Q. Zhu, R. Xu, J.-M. Song, M. Zhang, Y. K. Li, Y. Wu, and D. Guo, "Deepseekmath: Pushing the limits of mathematical reasoning in open language models." ArXiv, vol. abs/2402.03300,, 2024.
[37] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. H. Chi, F. Xia, Q. Lc, and D. Zhou, "Chain of thought prompting clicits reasoning in large language models," ArXiv, vol. 1e./2201.11905 2022.
[38] D. Azuma, Miyanishi, S. Kurita, and M. Kawanabe, "Scanqa: 3d question answering for spatial scene understanding." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19107-19117, 2021.
[39] X. Ma, S. Yong, Z. Zheng. Q. Li. Y. Liang, S.-C. Zhu, and S. Huang, "Sqa3d: Situated question answering in 3d scenes," ArXiv, vol. abs/2210.07474, 2022.
[40] H. Liu, C. Li, Y. Li, and Y. J. Lee, "Improved baselines with visual instruction tuning." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024.
[41] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, "Minigpt-4: Enhancing vision-languse understanding with advanced large language models," arXiv preprint wxXx:2334410599 2023.
[42] Y. Su, T. Lan, H. Li, J. Xu, Y. Wang, and D. Cai, "Pandagpt: One model to instruction-follow them all." arXiv preprint arXiv:2305. 16355, 2023.
[43] R. Pi, J. Gao, S. Diao, Pan, H. Dong, Zhang, Yao, J. Han, Xu, L. Kong, et al., "Detgpt: Detect what you need via reasoning," " arXiv preprint arXiv:2305.14100 2023.
[44] K. Li, Y. He, Y. Wang. Y. Li, W. Wang, P. Luo, Y. Wang. L. Wang, and Y. Qiao, "Videochat: Chat-centric video understanding," arXiv preprint 2rXiv:2305.0635. 2023.
[45] Y. Chen, S. Yang, H. Huang, T. Wang. R. Xu, R. Lyu, D. Lin, and J. Pang, "Grounded 3d-llm with referent tokens." arXiv preprint arXiv:2405. 10370, 2024.
[46] Z. Wang. H. Huang. Y. Zhao, Z. Zhang, and Z. Zhao, "Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes," arXiv preprint arXiv:2308.08763 2023.
[47] J. Huang, S. Yong, X. Ma, X. Linghu, P. Li, Y. Wang, Q. Li, S.-C. Zhu, B. Jia, and S. Huang, "An embodied generalist agent in 3d world," arXiv preprint arXiv:2311. 128711,, 2023.
[48] Y. Hong, H. Zhen, P. Chen, S. Zheng, Y. Du, Z. Chen, and C. Gan, "3d-llm: Injecting the 3d world into large language models," Advances in Neural Information Processing Systems, vol. 36, pp. 20482-20494, 2023.
[49] R. Fu, J. Liu, X. Chen, Y. Nic, and W. Xiong, "Scene-Ilm: Extending language model for 3d visual undorstanding and reasoning," arXiv preprint 2403::::403.1101. 2024.
[50] Z. Qi, Z. Zhang, Y. Fang. J. Wang, and H. Zhao, "Gpt4scene: Understand 3d scenes from videos with vision- language models." arXiv preprint arX1v:2501.01428, 2025.
[51] Z. Fan, J. Zhang, R. J. Zhang, R. Chen, H. Hu, K. Wang, H. Qu, D. Wang, Z. Yan, al., "Vlm-3r: Vision- language models augmented with ansructtoo-aligned 3d reconstruction," arXiv preprint arXiv:2505.20279, 2025.
[52] K. Ouyang, Y. Liu, H. Wu, Y. Liu, H. Zhou, J. Zhou, F. Meng, and X. Sun, "Spacer: Reinforcing milms in video spatial reasoning." arXiv preprint 251400250..01805, 2025.
[53] Z. Liu, Y. Dong. Z. Liu, W. Hu, J. Lu, and Y. Rao, "Oryx milm: On-demand at arbitrary resolution." arXiv preprint arXiv:2409.12961 2024.
understanding
[54] X. Wang, Y. Zhang, O. Zohar, and S. Ycung-Levy, "Vidcoagent: Long-form vidco understanding with large language model as agent," in European Conference on Computer Vision, pp. 58-76, Springer, 2024.
[55] P. Wu, Y. Liu, M. Liu, and J. Shen, "St-think: How multimodal large language models reason about 4d worlds from ego-centric videos." arXiv preprint MxXiv:2503.12542 2025.
13

---

[56] S. Zhou, A. Vilesov, X. He. Z. Wan. S. Zhang, A. Nagachandra, D. Chang. D. Chen, E. X. Wang. and A. Kadambi, "VIm4d: Towards spatiotemporal awareness in vision language models," in Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025.
[57] X. Linghu, J. Huang, X. Niu, S. Ma. B. Jia, and S. Huang. "Multi- situated reasoning in 3d scenes," Advances in Neural Information Processing Systems, vol. 37. pp. 140936, 2024.
[58] B. Yin, Q. Wang. P. Zhang. J. Zhang, K. Wang, Z. Wang, J. Zhang. K. Chandrasegaran, H. Liu, R. Krishna, et al., "Spatial mental modeling from limited views," arXiv preprint arXiv:2506.2/458 2025.
[59] A. Vaswani, N. M. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez. L. Kaiser, and Polosukhin, "Attention is all you need," in Neural Information Processing Systems. 2017.
[60] T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski, "Vision transformers need registers," ArXiv, vol. atb//3399165855 2023.
[61] X. Pan, S. N. Shukla, A. Singh, Z. Zhao, S. K. Mishra, J. Wang. Z. Xu, J. Chen, K. Li, F. Juefei-Xu, Hou, and S. Xie, "Transfer between modalities with metaqueries." " 2025.
[62] G. I.. Nemhauser, I.. A. Wolsey, and M. I.. Fisher, "An analysis of approximations for maximizing submodular set functions i," Mathematical Programming. vol. 14, no. 1. pp. 265-294. 1978.
[63] D. S. Hochbaum, Approximating covering and packing problems: set cover, vertex cover, independent set, and related problems, p. 94-143, USA: PWS Publishing Co., 1996.
[64] A. Dai, A. X. Chang. M. Savva, M. Halher, T. A. Funkhouser, and M. Nießner, "Scannet: Richly- 3d reconstructions of indoor scenes," 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2432-2443, 2017.
[65] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," " CoRR, vol. thss/4412.6988 2014.
[66] F. Xue, Y. Chen, D. Li, Q. Hu, Zhu, X. Li, Y. Fang, H. Tang, S. Yang. Z. Liu, He, H. Yin, P. Molchanov, Kautz, L. Fan, Y. Zhu, Lu, and S. Han, "Longvila: Scaling long-context visual language models for long videos," ArXiv, vol. abs/2408. 10188, 2024.
[67] Lin, H. Yin, W. Ping, Y. Lu, P. Molchanov, A. Tao, H. Mao, J. Kautz, M. Shoeybi, and S. Han, "Vila: On pre-training for visual language models," 2024 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26679-26689, 2023.
[68] P. Zhang, K. Zhang, B. Li, G. Zeng, J. Yang, Y. Zhang, Z. Wang. H. Tan, C. Li, and Z. Liu, "Long context transfer from language to vision," ArXiv, vol. abs/2406. 16852, 2024.
[691 C. Yeshwanth, Y.-.. Liu, M. Nicßner, and A. Dai, "Scannet++: A high-fidelity dataset of 3d indoor seenes," 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12-22, 2023.
[70] G. Baruch, 7. Chen, A. Dehghan, T. Dimry, Y. Feigin, P. Fu, T. Gebauer, B. Joffe, D. Kurz, A. Schwartz, and F. Shulman, "ARKitscenes diverse real-world dataset for 3d indoor scene understanding using mobile RCB-d data." in Thirty-fiiih Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[71] Z. Zhu, X. Ma, Y. Chen, Z. Deng, S. Huang, and Q. Li, "3d-vista: Pre-trained transformer for 3d vision and text alignment," 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2899-2909, 2023.
[72] L. Chen, L. Li, H. Zhao, Y. Song, and Vinci, "RI-v: Reinforcing super generalization ability in vision- language models with less than $3." "3.""h.tt::///// 02-02.
2025. Accessed: 2025-
[73] Q.-Y. Zhou, J. Park, and V. Koltun, "Open3d: A modern library for 3d data processing," ArXiv. vol. hts/1801109847, 2018.
[74] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, "Indoor segmentation and support inference from rghd images," in European Conference on Computer Vision, 2012.
[75] S. Gupta, P. Arbeláez, and J. Malik, "Perceptual organization and recognition of indoor scenes from rgb-d images," 2013 IEEE Conference Computer Vision and Puttern Recognition, pp. 564-571, 2013.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The main claims made in the abstract and introduction (Sec. accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
We discuss possible limitations of our method in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer to our method (Sec. and implementation details (Sec. 4) to reproduce our work step by step.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will release the data and code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
soteeeesubmmsiioooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https c/iips.cc/public//uue
CodeeSumiiii wettubbmmssiiioi ssionpolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We specify all the details in Sec.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow conventions observed in certain prior studies, and they don't 't include error bars in their experiments. To ensure a fair comparison, we similarly have not reported error bars our paper.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:p///aaars...cccbbbbll//tttt
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide Broader impacts in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
functioning correctly, harms that could arise when the
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
19

---

Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All code bases and datasets used in our work are open-source, and we strictly follow the licenses or terms of use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi twwowethhde...m caree.com/aee com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
20

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [Yes]
Justification: Our experiment poses no potential risks to study participants. We also adhere to the NeurlPS Code of Ethics and the guidelines for our institution.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

Technical Appendices and Supplementary Material
A
Broader Impacts
This work advances spatial reasoning in multimodal large language models by enabling 3D under- standing purely from 2D visual inputs. Such capability may broaden the accessibility of spatially aware Al in domains such as robotics, autonomous systems, and visual content understanding, without requiring costly 3D data, As with other vision-language models, considerations of data privacy and ethical deployment remain important to ensure positive social outcomes.
B
Additional Method Details
B.1 Details of Space-Aware Frame Sampling
Our space-aware frame sampling algorithm consists of three stages: (1) Scene geometry preprocessing,
(2) Voxelization and coverage calculation, and (3) Greedy maximum coverage selection. Beginning with the original video sequence V )
first perform uniform subsampling to obtain Nm 128 candidate frames {f.Th N For each subsampled frame, we leverage the backbone and Nm and (D Nm as illustrated in the main paper. Then we reconstruct 3D point maps p." through depth reprojection:
head of VGGT [32] to compute (E.T ( /i=1
p
= D
[uv]] -ET
(8)
where (u, v) denote pixel coordinates. In practice, we also obtain a confidence value c(p) E [o. for each point pm from the depth head. Although VGGT (32) also directly decode point maps from 3D dense features, we find that using depth and camera produces more accurate results.
The voxelization and coverage calculation process first establishes a 3D bounding box encompassing all valid scene points:
No. Pvllid ULDEP P.
dpp 0.1 c(p) 2 Percentle((((p))
i=1
We then discretize the bounding box into voxels. To handle relative scales in VGGT 32 outputs, use an adaptive way to set the voxel size A to 1 of the minimum dimension of the scene's bounding box:
A= A= X
min (max (Patia) - min
(10)
where hyperparameter and we set it 20. Each frame's voxel coverage V(f,m) then calculated by discretizing its valid points:
{
p min min
((fTT)
p P P!" n Praiid
(11)
A
Finally, we can formulate frame selection as the typical maximum coverage problem [62]
Uv(C")
max SC SC..........)))
s.t.
|S] = N₁:
(12)
iES
In practice, we set N₂ 16 and use a greedy approach provides the maximum new coverage, which is illustrated in Algorithm 1
663 25 to iteratively select the frame that
B.2 Details of Feature Fusion
Both the 2D and 3D encoders use a spatial patch size of 14. The 2D encoder further reduces the token sequence length by merging tokens spatially (2x2 adjacent tokens) and temporally (every 2 consecutive frames). As result, the 2D encoder outputs exactly one-eighth the number of tokens compared to the 3D encoder (we exclude register and camera tokens). To align these tokens, we first apply the same spatial-temporal merging strategy as used the 2D encoder. After merging, we rearrange the tokens into sequence, ensuring the two sets of tokens are precisely aligned in both position and number. Then we project both tokens into language model's hidden dimension with two-layer MLP and fuse them by dlement-wise addition.
22

---

Algorithm Greedy Maximum Coverage Sampling
Input Frame voxel sets (V(""") Nm,
j= target selection size N₂
Output Selected frame indices S S { (1 N
1: S +-
D Selected frames Covered voxels
2: C +-00 0
3: R + (1...., Nm
> Remaining candidates
4: for <- 1 to N₂ do
5: R = then
6: break
No remaining candidates
7: end
8:
1" argmax [V(".) Ac
D Max coverage gain
iR
9:
if (f..") \CI 0 then
10:
break end if
D. No additional coverage
11:
12:
S 5- fi
P Update selection
13:
C C-CUV
Update covered voxels Remove from candidates
14:
R
end for
16: return S
B.3
Details of Dataset Construction
We follow similar approach to that used in 18] construct the self-created part of training dataset. Specifically, the construction involves three main processes: video preprocessing, metadata computation, and QA pair generation.
Video Preprocessing. In this stage, we extract frames from the raw ScanNet (64 scans and convert them into videos at 24 FPS with resolution of 640 480.
Metadata Computation. In this stage. we extract spatial and semantic metadata from raw ScanNet scans and their associated semantic annotations. First, we align each raw scene mesh using the provided axis alignment matrices and convert it to the Open3D 73 point cloud. At the room level, we compute the room size using the alpha-shape algorithm and determine the center coordinates. At the object level, we generate oriented bounding boxes (OBBs) for each valid object instance and assign semantic labels from the annotations, excluding structural elements (e.g., walls, floors) and ambiguous categories (e.g. otherstructure). To ensure consistency across categories, we remap the
original ScanNet semantic labels to new label set based on the NYU40 classes [74] 75 (which we manually add and remove some categories align with VSI-Bench (T8). In addition, we collect the projected 2D semantic annotation of each scene video for the appearance order task. The final metadata for each scene includes: (1) room size and center coordinates; (2) the projected 2D semantic
annotation of the scene video; (3) object instances and their OBB parameters, including rotation matrices, extents, and centers; and (4) semantic labels for each object.
QA Pair Generation. Finally, we generate QA pairs of different tasks, including object counting, object size, room size, absolute distance, appearance order, relative distance, and relative direction.
Object counting (numerical): We first count how many times each object category appears in the scene, then randomly sample a category that appears at least twice. Question template: "How many <category> are in this room?"
Object size (numerical): We randomly sample a unique object in the scene and take the longest side of its oriented bounding box (OBB) the ground-truth length (in em). Question template:
"What is the length of the longest dimension (length, width, or height) of the <category>, measured in centimeters?"
Room size (numerical): We use the pre-computed room size (in m²) as the ground-truth value. Question template: "What is the size of this room (in square meters)?"
Absolute distance (numerical): For a pair of objects, we uniformly sample points inside each OBB and take the minimum Euclidean distance between the two point clouds as the ground-truth
23

---

(in m). Question template: "Measuring from the closest point of each object, what is the direct distance between the <category_ A> and the <category B> (in meters)?"
Appearance Order (multiple choice): We calculate the first appearance timestamp of each cate- gory, which the timestamp when its visible pixel count exceeds predefined threshold. Using these timestamps, we generate the correct order of appearance among the categories, along with other options. Question template: What will be the first-time appearance order of the following categories in the video: acategory_Az. <category B>, <category C>, <category_ D_>
Relative distance (multiple choice): We use an "anchor" object that is unique in the scene and then select four additional objects while enforcing 15-30cm separation thresholds between options. Question template: "Which of these objects (<category_As <category__i B> <category_ <category_De D>> is closest to the <anchor paaccorrcategory>??
Relative direction (multiple choice): For triple ppasition.facing query) of unique categories,
compute the horizontal angle between the vectors position -> facing and position + query. The angle is then discretized into directional classes (easy: left/right, medium: left/right/back, hard: rron-llefffroo--rgghhccciiiiiiif right). Question template (easy example): "If am standing by the Epesition-categoryy and facing the Stacing-category>, is the <query-category> to the left or the right?"
B.4 Details of Cold Start
To align the model with the desired reasoning format, we perform a simple cold start for 200 steps before GRPO training. The key to this stage is the construction of spatial reasoning dataset with chain-of-thought (CoT) annotations. The construction process is as follows:
Subset Sampling. We begin by sampling a subset Do = (I) (12,, M,) from training dataset.
Multi-path CoT Generation. For each item I4 Do. we utilize Qwen2.5-VL-722
14 generate
K independent reasoning processes (k)) and corresponding answers A4) We then compute reward
r Reward (A((A) AAi) for each reasoning-amd
pair, where Reward the reward function
described in Sec B.5
Consequently, we obtain set of outputs O = {(7. (k) (1)) (1)) ) for
each E Du-
Adaptive Filtering. Since VLL-2255 14] may generate incorrect reasoning processes and answers, we apply filtering process based the computed rewards. While using global reward threshold is straightforward, it often results in an imbalance across question types in the selected subset. To mitigate this, adopt adaptive filtering strategy. Specifically, for each item (k")
I E Do, we first keep the output with the highest reward to get O = (f,(("" ' , (x"))) where
k- arg max; ri (k) L; = r (kK) denote the maximum reward. We then categorize all items based on their question type and compute a question type-dependent threshold T((i), where t(i) denotes the
type of problem i. The item is added into the cold start set if and only if:
f Z Te(i) and A > 0,
where the type-dependent threshold satisfies Te(s) == Quantile(f?) t(f) = t(i)), 0.5). This rule preserves approximately the top 50% of generations per question type while discarding degenerate (zero-reward) outputs. In practice, we set N, = 5000 and = 3, and finally we get 2459 items in the cold start set. We provide pseudocode for this process in Algorithm 2
B.5
Details of SFT and GRPO Training
Reward Calculation. Given predicted answer Apred and ground truth answer Agt. the reward function Reward( Apod, Ag) consists of a format reward Rim and task-specific reward:
RMC.
multiple-choice
Reward[AArree Ag) >i Rtoemat + A
RMRA
numerical
(13)
R
verbal
24

---

Algorithm 2 Cold Start Dataset Construction
Input Original dataset D
1: Qwen2.5-VL model M
2: Reward function Reward(.:)
Sample size N,. Paths per item K
Output Filtered dataset coold
4: Initialize Do Sampling (D.
5: Deold t- V
6: for each item I, = (Q V., M,) E Do do
7: Generate K reasoning paths: (T, (T(()) KK
+ MM,(2. Vi)
sk=1
8: Compute rewards: r: +- Reward AAi,, vk
9: Select best path: k- arg maxe ri (k*)
(k)
10: Record fg +- 1 Ó. + (F,tk)) A(K))
11: end for
12: Group items by type: {Ge} + GroupBy CryopByType((ee Type
for each question type do
14:
Compute threshold: < Quantile ((ryli E Gr), 0.5)
15: end for
16: for each item I4 E Do do
if' ft 2 Te(e) and f; then
18: Doold t Doold U {O,}
end if
20: end for
return Deold
where A1 and d are hyperparameters, both of which are set to 1 in our implementation. For multiple-choice questions, we implement exact match criterion:
(Apeed Agi) (Arre)) (Agi))
(14)
where (-). performs answer normalization through whitespace stripping and I(-) denotes the indicator function. For numerical tasks, we compute mean relative accuracy (MRA) [1]
RMRA (Apreds Art)
[T7 [a(Apros) a(Ag)) (a(AAg)) t
1
Σ
<T)
(15)
where a(.) normalizes numeric values, c = 10-8 prevents division by zero, and T (0.50, 0.55, ..., 0.95} defines accuracy thresholds. For verbal answer questions, compute normalized similarity score using the Levenshtein ratio:
Rvertal (Apred Ag) = (er(@((ppc)))) ((Aa)) (0(A))))) + (0(Ag)
(16)
where Drev denotes the Levenshtein edit distance, and o6-) represents the text normalization function. In practice, the implementation provided by the Levenshtein library. In addition to the format and task-specific rewards, we also incorporate a reasoning length reward following Video-RI [2] which encourages the model to perform more thinking before generating the final answer.
Other Details. pressensss the prompts used in the SFT and GRPO stages. For both stages, we adopt the default system prompt of Qwen2.5-VI VL [44, namely, "You are helpful assistant." the SFT stage, the user prompt consists of question and type template. In the GRPO stage, the user prompt comprises question, a question post string, and a type template. conduct all experiments on Intel(R) Xeon(R) Gold 6430 platform with 80G NVIDIA A800 GPUs.
25

---

Question
GRPO Stage
Example 1:
many chair(s)
System Prompt
You a helphul
Example
User Prompt
Question If standing the sofa and facing the stove, the tv my left, right, or back? 'nAn object to my back would have turn least 135 degrees in order to faice Options: back right lett"
[Question)
humen pondering deeply. Engage an internal dialogue using expressions such as liet me think;, 'walt, "Hmm', 'oh, see', break down', etc, language thought expreesions. encouraged include self-reflection verification the reasoning process. Type tempiate["problen problem type"]
"Plesse think sbout queetion
SFT Stage
System Prompt
Type Template
"You are helpful scsistant."
Multiple Please provide your detailed reasoning between the option's latter from the given choices (e.g.. B, etc.) within the
User Prompt
<think> <ttinkk> lags,
(Question) Type Templato""problee problem type']
Type Template
<answer> <Janswer> tags."
Multiple choice:" Please with option's
provide your
reasaning
given choices A, B, etc.) within the <answer? <lanswer>
<think> <ithink> and then answer the question with the
tags"
numerical value (e.g. 42, 3.14, etc.) within the <answer> </answer>
Please answer the only numerical value (a.g-
lags.
3.14, etc.) within the <answer> <lanswer> tags.
reasoning between <think> </think> and then answer the question simply within the
Verbal:" provide your
Verbal: " Please answer the question simply within the <answer>
<lanswer> tags."
<answer> <janswer> tags"
Figure 6: Illustration of the prompts used the SFT and GRPO stages. We use the default system prompt of Qwen2.5-VL 14 (i.e.. "You are helpful assistant") for both stages, In the SFT stage, the user prompt consists of a question and a type template. the GRPO stage, user prompt
includes question, question post string, and a type template.
Table Maero average scores on VSI-Bench for Qwen2.5-V VI. series and Spatial-MLL.M
Numerical Question
MultipleChhoice Question
Methods
Avg.
Obj. Cnt.
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
Appr. Order
Owen2.5-VL-3B
14
24.3
24.7
31.7
22.6
38.3
42.6
26.3
21.2
29.0
Qwen2.5-VL-7B
14
40.9
14.8
43.4
10.7
38.6
40.1
33.0
29.8
31.4
Owen2.5-VI.-222
[4]
29.3
57.9
29.4
41.7
39.3
23.2
34.3
Spatial-ML.MM-BB
65.3
34.8
63.1
33.5
47.0
C
Additional Experiments
C.1 Additional Results on VSI-Bench
We present qualitative examples of Spatial-MLLM on the VSI-Bench [8] dataset in Figures to As illustrated, Spatial-MLLM is capable of reasoning with visual information across different task types and producing final answers accordingly. Furthermore, it demonstrates strong abilities in self-verification and task decomposition during the reasoning process.
C.2 Additional Results on ScanQA and SQA3D
39 benchmarks in Table 6 We present additional evaluation results the ScanQA 38 and SQA3D
and Table 7 As shown, our proposed method consistently outperforms all video-input models, including LLaVA-Video- [12] and Oryx-34B (53), both of which incorporate spatial reasoning
datasets such as ScanQA 38 during training.
Despite having only 4.9 billion parameters, SpLt significantly surpasses Qwen2.5-VL. 72B [4] on the ScanQA benchmark, achieving substantial gains across multiple metrics for instance, +2.3 EM-1, +17.6 BLEU-I, and +24.9 CIDEr. Similarly, on the SQA3D benchmark, Spatial-MLLM consistently outperforms Qwen2.5-VL--2BB across all question types and overall performance, including improvements of +4.2 EM-1 and +7.8 EM-R1, with notable gains in the Is (+15.3) and Which (+13.9) categories.
26

---

Table 6: Additional evaluation results on Scan 338 for task-specific models, 3D/2.5D input models, and video-input models. Reported metrics include EM-1, BLEU-1 BLEU-4, ROUGE-L, METEOR, and CIDEr.
ScanQA (val)
Methods
EM-1 BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-L METEOR CIDEr
Task-Specific Models
ScanQA
38
21.1
30.2
20.4
15.1
10.1
33.3
13.1
64.9
3D-Vista
71
22.4
10.4
35.7
13.9
69.6
3D/2.5D-Impp Input Models
3D-LLM [48]
20.5
39.3
25.2
18.4
12.0
35.7
14.5
69.4
LLB3A 22
13.5
37.3
15.9
76.8
Chat-Scene
22
21.6
43.2
29.1
20.6
14.3
41.6
18.0
87.7
3D-LLaVA 21
17.1
43.1
18.4
92.6
Video-3D LLM 25
30.1
47.1
31.7
22.8
16.2
49.0
19.8
102.1
Video-Input Models
Qwen2.5 3B 14
21.9
26.4
16.2
11.9
7.5
33,2
12.2
62.7
Qwen2.5-VL 7B 14
23.3
26.2
17.7
13.0
9.6
34.2
12.7
64.9
Qwen2.5-VL-72B [4
24.0
26.8
17.8
14.6
12.0
35.2
13.0
66.9
ILaVA-Video-7B 12
39.7
26.6
9.3
44.6
17.7
Oryx-34B [
38.0
24.6
37.3
15.0
72.3
Spatial-MLLLM44B
26.3
44.4
28.8
21.0
14.8
45.0
18.4
91.8
Table Additional evaluation results on SQA3D (39] for task-specifie models, 3D/2.5D input models, and video-input models. In addition the average EM-1 and EM-RI across all questions, we also report the average EM-1 for different question types, including What, Is, How, Can, Which, and Others.
SQA3D (test)
Methods
What
How
Can
Which Others
Avg. (EM-1) Avg. (EM-RI)
Task-Specific Models
SQA3D
[39
31.6
63.8
46.0
69.5
43.9
45.3
46.6
3D-Vista
34.8
63.3
45.4
69.8
47.2
48.1
48.5
3D/2.5D- Models
Scene-LLM
49
40.9
69.1
45.0
70.8
47.2
52.3
54.2
Chat-Scene
22
45.4
67.0
52.0
69.5
49.9
55.0
54.6
57.5
Video-3D LLM 25
51.1
72.4
55.5
69.8
51.3
56.0
58.6
Video-Input Models
Qwen2.5-VL-3B [14
34.8
52.1
39.8
52.7
45.6
47.0
43.4
45.9
Qwen2.5-VL-7B [14]
39.7
56.6
41.1
55.9
47.6
47.2
46.5
49.8
Qwen2.5-VL-72B
[4
41.7
56.3
41.5
55.6
44.5
48.0
47.0
50.9
LLaVA-Video-7B [12
42.7
56.3
47.5
55.3
50.1
47.2
48.5
Spatial-MLLM-44B
45.9
71.6
55.1
69.5
52.0
53.0
55.9
58.7
27

---

Video
Question
"How many chair(s) are this room?"
Spatial-MLLM
"<think>
Let me think about this carefully. In the images provided, need to identify any chairs present in the room. First, l'll scan the entire room to ensure I'm not missing anything. The room appears be living space with a kitchen area, a couch, a dining area, and other furniture. Now, let's break down: In the kitchen area, there is small black stool or cushioned seat near the cabinets. This doesn't seem like chair.
There's couch in the living room area, but that's sofa, not chair.
3. the dining area, there is small table with chairs visible. They are wooden and have simple designs. So, after thorough review of the images, can confirm that there are two chairs in the room. Oh, The question was asking for the number of chairs, and now I'm confident that there are two.
</think>
<answer>2 </answer>"
Figure 7: Qualitative example on VSI-Bench [18].
Video
Question
"How many sofa(s) are in this room?"
Spatial-MLLM
"<think>
Let me think about this for moment. The images show a living room with various pieces of furniture. As scan through them, notice few clearly identifiable sofas.
First, in the initial image, there's a partially visible sofa near the window. It appears to be traditional design, likely made of fabric and leather. Next, in the following image, there's another sofa positioned near the fireplace. This one seems to have a more modern design, possibly made leather or similar material. Then, in the third image, there's third sofa visible in the background, near the bookshelf. It looks like it's a different style from the first two, possibly sectional sofa. In the fourth image, there's a fourth sofa visible on the left side of the room. It appears to be single-seater sofa, different from others. Let's break down:
Traditional sofa: 1 Modern sofa: Sectional sofa: 1 Single-seater sofa:
Oh, see. There four distinct sofas visible in these images. should double-check make I'm not missing any, but after reviewing again, confirm there are no additional sofas hidden or partially obscured. Final count: 4</think>
<answer>4 </answer>"
Figure 8: Qualitative example on VSI-Bench (18)
28

---

Video
Question
"If am standing by the sofa and facing the stove, is the tv to my left, right, or back?|nAn object is to my back would have to at least 135 degrees in order face Options: A. back B. right C. left"
Spatial-MLLM
"<think>>
Let me think about this for moment. I'm standing by the sofa and facing the stove, need to consider the layout of the room. The stove is in the kitchen area, which appears to be separated from the living room positioned on the wall near the kitchen area, which means it's not directly in front of me but rather my side. Wait, the question asks if the TV is to my left, right, back. Let's break it down: if I'm facing stove,
by counter and some cabinets. Now, where is the TV? The TV is
positioned such way that not in front of me, must either to the left or the right. Given the layout of room, the TV Is on the opposite side of the kitchen area, which would be the right of the stove I'm facing Hmm, let's verify this. turn around from the sofa and face the kitchen area, the TV is indeed to the right of the kitchen appliances. This means that I'm facing the stove from the sofa, the would be my right. Oh, TV is to my right when I'm facing the stove from the sofa. </thinko-ansser>BB </answer>"
The
Figure 9: Qualitative example on VSI-Bench [8]
Video
Question
"What will be the first-time appearance order of the following categories in the video: basket, pillow, toilet, tv? Options:InA. basket, pillow, toilet, tv B. pillow, tv, toilet, basket C. toilet, pillow, basket D. toilet, basket, pillow, tv"
Spatial-MLLM
"ethink-Let me think about this carefully. The question asks for the first appearance order certain objects in video, but need to focus the video that shows the bathroom. the video, see various objects appear different sequences. Let's break down:
Bathroom sequence: The video starts with the bathroom, showing toilet, sink, shower curtain, and some shelves. There's also a basket visible one part the bathroom. Transition living room: The video then transitions to living room with a bed, which is different space entirely. pillow and some decorative items. The third part shows the living room with a couch, TV, and other decorations.
The living room shows a
Now, let's consider the appearance order. The toilet in the bathroom, the basket appears the bathroom, the pillow is in the bedroom, and the TV is in the living room. Let's verify this
time: Bathroom toilet basket >> Bedroom Yes pillows, other bedroom items >> Living Room >> TV."
Therefore, the correct answer D. Toilet, basket, pillow, tw.//thinko<cnnswerrD </answer>"
Figure 10: Qualitative example VSI-Bench 18


Reviews:
[{"review_id": "qv95Mn9adb", "reviewer": "Reviewer_Ti44", "summary": "The paper introduces a novel framework, Spatial-MLLM, aimed at enhancing the spatial intelligence of existing Multimodal Large Language Models (MLLMs) using purely 2D visual inputs. The paper proposes a dual-encoder architecture that integrates a 2D visual encoder for semantic feature extraction and a spatial encoder for 3D structure feature extraction. Additionally, the paper presents a space-aware frame sampling strategy to select the most spatially informative frames under input constraints.", "questions": "Please refer to the weakness.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n1. The paper presents a technically sound approach with a dual-encoder architecture and space-aware frame sampling strategy. The results are supported by extensive experiments.\n2. The paper is well-organized and clearly written, with detailed explanations and diagrams that aid understanding.\n\n\n**Weaknesses**\n1. The paper lacks a detailed comparison of the computational efficiency of Spatial-MLLM versus 3D/2.5D-Input Models like Video-3D LLM and Video-Input Models. \n2. Testing on other benchmarks like MMScan QA and OpenEQA used in LLaVA-3D would better demonstrate its generalizability.\n3. The space-aware frame sampling is applied after uniform sampling. Testing it directly on original video frames could show its robustness and effectiveness under different conditions.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and for the time spent reviewing our paper.  We address the weaknesses (W) raised by the reviewer as follows:\n\n------\n\n> **W1:** The paper lacks a detailed comparison of the computational efficiency of Spatial-MLLM versus 3D/2.5D-Input Models like Video-3D LLM and Video-Input Models.\n\n- **R1: Computational efficiency comparison between Spatial‑MLLM and baseline Models.**  In Table 1, we presents the average inference time and peak memory usage on the ScanQA for Spatial‑MLLM, Video‑3D‑MLLM (3D/2.5D-Input Model), and Qwen‑2.5‑VL‑3B (Video Input Model). Due to the global attention used in the VGGT, 3D encoding takes relatively large time for our model. In spatial reasoning benchmark like VSI-Bench, existing RL-based models tend to achieve stronger performance due to the long-CoT reasoning process, in which model typically generates a large number of tokens for each question. In such scenarios, we observe that the autoregressive token generation takes much more time than 3D spatial encoding, which only needs to be performed once at the beginning of the reasoning process. We show the inference time of each model component when generating sequences of different lengths in Table 2.\n\n  **Table 1.** *Average inference time and peak memory usage on ScanQA (without CoT).*\n\n|      Model       | 2D Encoding Time | 3D Spatial Encoding Time | Language Backbone Time | Avg. Time / Question | Avg. Peak Memory Usage |\n|:----------------:|:----------------:|:------------------------:|:----------------------:|:--------------------:|:------------------------:|\n| Spatial-MLLM     |      0.34s       |          1.66s           |         0.25s          |        2.27s         |        13.06 GB          |\n| Video-3D-LLM     |      0.55s       |           N/A            |         1.30s          |        1.87s         |        21.34 GB          |\n| Qwen2.5-VL-3B    |      0.34s       |           N/A            |         0.25s          |        0.61s         |         8.28 GB          |\n\n\n  **Table 2.** *Inference time of each model component when generating sequences of different lengths. As shown, the overhead introduced by the 3D Spatial Encoder becomes relatively small when generated token lengths increase.*\n\n| Generated Token Length | Language Backbone | 3D Spatial Encoder | 2D Encoder | Connector | 3D Encoding / Total Time |\n|:----------------------:|:-----------------:|:-------------------:|:----------:|:---------:|:------------------------:|\n|          64            |       1.98s       |       1.66s         |   0.34s    | 0.031s    |          41.4%           |\n|         128            |       3.97s       |       1.66s         |   0.34s    | 0.031s    |          27.7%           |\n|         256            |       7.94s       |       1.66s         |   0.34s    | 0.031s    |          16.6%           |\n|         512            |      15.87s       |       1.66s         |   0.34s    | 0.031s    |           9.3%           |\n|        1024            |      31.74s       |       1.66s         |   0.34s    | 0.031s    |           4.9%           |\n\n\n------\n> **W2:** Testing on other benchmarks like MMScan QA and OpenEQA used in LLaVA-3D would better demonstrate its generalizability.\n\n- **R2: Additional benchmarks.** We appreciate the reviewer’s suggestion to evaluate our model on additional benchmarks such as MMScanQA and OpenEQA. MMScanQA primarily assesses a model’s 3D scene understanding capability through a series of QA tasks. Notably, many of its subtasks (*e.g.*, ST-Space, OO-Space) require the model to receive ground-truth 3D spatial coordinates and perform reasoning based on them. This evaluation setup differs from our target scenario, where the model receives only RGB videos as input without access to ground-truth metric 3D coordinates. Due to time constraints during the rebuttal period, we are unable to adapt the MMScanQA evaluation protocol to align with our setting and therefore don't include it in our experiments. We will explore the possibility of adapting MMScanQA to our setting in the revised version of the paper. For OpenEQA, we present the evaluation results of our model and the baseline models in Table 3. As shown, Spatial‑MLLM outperforms Qwen‑2.5‑VL‑3B and Gemini‑Pro, particularly in the spatial understanding task. While LLaVA‑3D achieves the highest overall score, it heavily relies on depth maps, per-frame camera poses, and camera intrinsics in evaluation, which are not applicable in many real-world scenarios. Instead, Spatial‑MLLM only requires videos as input, making it more applicable to real-world scenarios.\n\n**Table 3.** *Performance comparison on the OpenEQA benchmark.*\n\n| Model         | Require Video Input | Require 3D Ground Truth Input | Spatial Understanding Performance | Overall Performance |\n|---------------|:-------------------:|:-----------------------------:|:---------------------------------:|:-------------------:|\n| Gemini-Pro    |        Yes          |              No              |              37.6                 |        44.9         |\n| Qwen2.5-VL-3B |        Yes          |              No              |              36.7                 |        47.2         |\n| Spatial-MLLM  |        Yes          |              No              |              40.6                 |        48.3         |\n| LLaVA-3D      |        Yes          |             Yes              |                /                  |        53.2         |\n\n------\n> **W3:** The space-aware frame sampling is applied after uniform sampling. Testing it directly on original video frames could show its robustness and effectiveness under different conditions.\n\n- **R3: Directly use space-aware frame sampling on original video.**  In space-aware frame sampling, we first use VGGT to extract 3D features from the video, and then decode depth maps and camera parameters. For short videos (*e.g.*, 64 or 128 frames), we can directly apply space-aware sampling on original video frames. However, as videos in our evaluation benchmarks typically contain 1,000 to 2,000 frames, it is hard to feed all frames into VGGT due to VARM's constraints (as shown in Table 4). Therefore, in practice, we first perform uniform sampling to reduce the number of frames to 128, and then apply the space-aware sampling strategy to select spatially informative frames. As shown in Table 3 of the main paper, this approach outperforms directly uniformly sample 16 frames. To further evaluate the effectiveness of space-aware sampling on original frames, we construct an additional test set based on ScanNet, consisting of 100 short videos. Each video contains 128 frames captured at 5 fps. We applied both space-aware and uniform sampling directly to the original videos and reported the coverage ratio in Table 5. As shown, space-aware sampling achieves higher coverage than uniform sampling, demonstrating its effectiveness in selecting spatially informative frames.\n\n  **Table 4.** Peak memory usage of VGGT under different numbers of input frames (input resolution: 644×476).\n\n  | Number of Frames | Peak Memory Usage |\n  |:----------------:|:-----------------:|\n  |       16         |     7.75 GB       |\n  |       32         |    17.72 GB       |\n  |       64         |    33.07 GB       |\n  |      128         |    63.79 GB       |\n  |      256         |      OOM          |\n\n  **Table 5.** Coverage ratio of space-aware sampling and uniform sampling on original video frames.\n\n  | **Sampling Strategy**           | **Coverage Ratio** |\n  |:-------------------------------|:------------------:|\n  | Space-aware Frame Sampling     | 63.6%              |\n  | Uniform Frame Sampling         | 54.7%              |"}, {"review_id": "kz4WNtZ5tF", "reviewer": "Reviewer_s9Ab", "summary": "This paper presents Spatial-MLLM, a framework designed to improve the spatial reasoning capabilities of Multimodal Large Language Models using only 2D video inputs. The core contribution is a dual-encoder architecture that combines a standard 2D visual encoder for semantic understanding with a novel spatial encoder, initialized from a visual geometry foundation model, to extract 3D structural information. These complementary features are fused by a lightweight connector before being processed by the LLM backbone. Experiments show that this approach achieves state-of-the-art performance on several visual-based spatial reasoning benchmarks, outperforming existing open-source and proprietary models.", "questions": "See the weakness section above.", "limitations": "See the weakness section above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper tackles the important and challenging problem of improving spatial intelligence in MLLMs from purely 2D video inputs, a key limitation of current models that are often optimized for semantic rather than structural understanding.\n- The authors introduce a new large-scale dataset, Spatial-MLLM-120K, which is a valuable asset for the community. \n\nWeaknesses\n- The approach relies heavily on combining existing, powerful pretrained models (Qwen2.5-VL and VGGT) with a very simple fusion mechanism (MLP-based addition). While effective, the novelty lies in the combination rather than a fundamental new technique, which might be seen as an incremental contribution.\n- While the paper avoids the need for 3D data inputs, comprehensively understanding a 3D object from 2D data requires many frames. The proposed sampling strategy highlights this by pre-processing a large number of frames (128) to select a small subset (16). The paper is missing a critical analysis of the total computational load and efficiency of this approach compared to models that use 3D data directly.", "was_revised": true, "final_justification": "I thank the authors for their rebuttal, which has successfully addressed my primary concerns regarding novelty and efficiency. They clarified their contribution in creating a novel pipeline for integrating 3D priors from video-only inputs, and their analysis convincingly showed the computational overhead is manageable for complex reasoning tasks. With these issues resolved, I have raised my score to support acceptance.", "rebuttal": "We thank the reviewer for the comments and for the time spent reviewing our paper. We address the weaknesses (W) raised by the reviewer as follows:\n\n------\n\n> **W1:** The approach relies heavily on combining existing, powerful pretrained models (Qwen2.5-VL and VGGT) with a very simple fusion mechanism (MLP-based addition). While effective, the novelty lies in the combination rather than a fundamental new technique, which might be seen as an incremental contribution.\n\n- **R1: Discussion on the contribution.** Our motivation is to enhance the fine-grained spatial understanding and reasoning capabilities of existing video MLLMs by introducing 3D geometric priors. Existing works either employ a single video encoder which fails to capture rich geometric information due to the CLIP-style training paradigm, or require ground-truth 3D inputs which are not applicable in many real-world scenarios. In this work, to our best knowledge, we are the first to address this problem by designing a pipeline that exploits the complementary strengths of both a 2D visual encoder from a pretrained MLLM and a 3D spatial encoder from a visual geometric foundation model, which effectively extracts 3D information with only video inputs. The main challenges lie in efficiently extracting 3D information from videos and effectively fusing semantic and geometric representations. To tackle this, we introduce a space-aware frame sampling strategy that selects frames based on voxel coverage to preserve essential 3D structure. We also find that a simple MLP-based fusion module can effectively integrate semantic and geometric features to support spatial reasoning. We believe our work could motivate more explorations on MLLM's spatial understanding and reasoning capabilities in the future.\n\n\n> **W2:** While the paper avoids the need for 3D data inputs, comprehensively understanding a 3D object from 2D data requires many frames. The proposed sampling strategy highlights this by pre-processing a large number of frames (128) to select a small subset (16). The paper is missing a critical analysis of the total computational load and efficiency of this approach compared to models that use 3D data directly.\n\n- **R2: Analysis of computational load and efficiency.** In Table 1, we presents the average inference time and peak memory usage on the ScanQA for Spatial‑MLLM, Video‑3D‑MLLM (3D/2.5D-Input Model), and Qwen‑2.5‑VL‑3B (Video Input Model). Due to the global attention used in the VGGT, 3D encoding takes relatively large time for our model. In spatial reasoning benchmark like VSI-Bench, existing RL-based models tend to achieve stronger performance due to the long-CoT reasoning process, in which model typically generates a large number of tokens for each question. In such scenarios, we observe that the autoregressive token generation takes much more time than 3D spatial encoding, which only needs to be performed once at the beginning of the reasoning process. We show the inference time of each model component when generating sequences of different lengths in Table 2.\n\n  **Table 1.** *Average inference time and peak memory usage on ScanQA (without CoT).*\n\n  |      Model       | 2D Encoding Time | 3D Spatial Encoding Time | Language Backbone Time | Avg. Time / Question | Avg. Peak Memory Usage |\n  |:----------------:|:----------------:|:------------------------:|:----------------------:|:--------------------:|:------------------:|\n  | Spatial-MLLM     |       0.34s      |          1.66s           |          0.25s         |        2.27s         |      13.06 GB      |\n  | Video-3D-LLM     |       0.55s      |          N/A             |          1.30s         |        1.87s         |      21.34 GB      |\n  | Qwen2.5-VL-3B    |       0.34s      |          N/A             |          0.25s         |        0.61s         |      8.28 GB       |\n\n  **Table 2.** *Inference time of each model component when generating sequences of different lengths. As shown, the overhead introduced by the 3D Spatial Encoder becomes relatively small when generated token lengths increase.*\n\n  | Generated Token Length | Language Backbone | 3D Spatial Encoder | 2D Encoder | Connector | 3D Encoding / Total Time |\n  |:----------------------:|:-----------------:|:-------------------:|:----------:|:---------:|:------------------------:|\n  |          64            |       1.98s       |       1.66s         |   0.34s    | 0.031s    |          41.4%           |\n  |         128            |       3.97s       |       1.66s         |   0.34s    | 0.031s    |          27.7%           |\n  |         256            |       7.94s       |       1.66s         |   0.34s    | 0.031s    |          16.6%           |\n  |         512            |      15.87s       |       1.66s         |   0.34s    | 0.031s    |           9.3%           |\n  |        1024            |      31.74s       |       1.66s         |   0.34s    | 0.031s    |           4.9%           |"}, {"review_id": "RR6tOnujCH", "reviewer": "Reviewer_hZ9X", "summary": "This paper presents the Spatial-MLLM framework for visual-based spatial reasoning using purely 2D observations, aiming to enhance spatial intelligence. The authors propose a dual-encoder architecture to separately encode 2D semantic features and 3D structural features. During inference, a space-aware frame sampling strategy is employed to select spatially informative frames, improving the efficiency and accuracy of spatial understanding.", "questions": "1. The MLP connector is introduced into the pretrained Qwen-VL model; however, it is unclear whether the 120K dataset is sufficient to effectively train this module. Additionally, the trainable parameters in the overall model is not clearly reported.\n\n2. It is not explicitly stated which variant of Qwen2.5-VL the Spatial-MLLM is built upon. The parameter size of the backbone model is also missing from the tables. Providing this information would improve the transparency and reproducibility of the work.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThis is a well-executed study with impressive performance results. The experiments convincingly demonstrate the effectiveness of the Spatial-MLLM-120K dataset, the proposed architecture, and the GRPO method across multiple spatial reasoning tasks.\n\nWeaknesses:\nThe proposed dual-encoder architecture increases the complexity of the multi-modal large language model. Given that the vision encoder in Qwen2.5-VL already supports 3D patches, incorporating additional spatial information through VGGT further increases the computational cost, which may limit scalability or real-time applicability.", "was_revised": true, "final_justification": "Thanks for your response. It has solved some of my concerns. I will maintain the score.", "rebuttal": "We thank the reviewer for the comments and for the time spent reviewing our paper. We address the weaknesses (W) and questions (Q) raised by the reviewer as follows:\n\n------\n\n> **W1:** *The proposed dual-encoder architecture increases the complexity of the multi-modal large language model. Given that the vision encoder in Qwen2.5-VL already supports 3D patches, incorporating additional spatial information through VGGT further increases the computational cost, which may limit scalability or real-time applicability.*\n\n- **R1: Clarification on 3D patches used in Qwen2.5-VL**. The \"3D patches\" in the Qwen2.5-VL model refer to height (h), width (w), and time (t) dimensions of video, modeling 2D image space with time rather than 3D spatial information (*i.e.*, x, y, z coordinates), which 3D spatial encoder provides. As demonstrated by the experimental results in Tables 1-3 of the main paper, such 3D spatial information significantly enhances the model's spatial understanding and reasoning capabilities.\n\n- **R2: Discussion on the computational cost.** We agree that integrating VGGT as 3D spatial encoder increases computational cost, and we report inference time of each model component when generating sequences of different lengths in Table 1. In spatial reasoning benchmark like VSI-Bench, existing RL-based models tend to achieve stronger performance due to the long-CoT reasoning process, in which model typically generates a large number of tokens for each question. In such scenarios, we observe that the autoregressive token generation takes much more time than 3D spatial encoding, which only needs to be performed once at the beginning of the reasoning process.\n\n  **Table 1.** *Inference time of each model component when generating sequences of different lengths. As shown, the overhead introduced by the 3D Spatial Encoder becomes relatively small when generated token lengths increase.*\n\n  | Generated Token Length | Language Backbone | 3D Spatial Encoder | 2D Encoder | Connector | 3D Encoding / Total Time |\n  |:----------------------:|:-----------------:|:-------------------:|:----------:|:---------:|:------------------------:|\n  |          64            |       1.98s       |       1.66s         |   0.34s    | 0.031s    |          41.4%           |\n  |         128            |       3.97s       |       1.66s         |   0.34s    | 0.031s    |          27.7%           |\n  |         256            |       7.94s       |       1.66s         |   0.34s    | 0.031s    |          16.6%           |\n  |         512            |      15.87s       |       1.66s         |   0.34s    | 0.031s    |           9.3%           |\n  |        1024            |      31.74s       |       1.66s         |   0.34s    | 0.031s    |           4.9%           |\n\n\n\n------\n> **Q1:** The MLP connector is introduced into the pretrained Qwen-VL model; however, it is unclear whether the 120K dataset is sufficient to effectively train this module.\n\n- **R3: Dataset size for the connector.**  Following standard practice, we designed the connector module to be relatively lightweight, serving primarily to project features from different modalities into a shared embedding space. To evaluate whether the 120K dataset is sufficient to train the connector, we conducte an ablation using different training data scales. As shown in Table 2, the model's performance improves rapidly when the dataset size increases from 0 to 90K and gradually slows down when approaching 120K. This suggests 120K dataset is a reasonable scale for training the connector. Nevertheless, we will continue to explore increasing the dataset size while also improving data quality and diversity to further enhance the model's performance in future work.\n\n\n    **Table 2.** *Model performance across different dataset sizes*\n\n  | SFT Data Scale | Overall Score | Numerical | Multiple Choice |\n  |------------|:-------------:|:---------:|:----------------:|\n  | 30K        |     43.0      |   47.5    |       38.3       |\n  | 60K        |     44.0      |   48.3    |       39.4       |\n  | 90K        |     45.7      |   50.8    |       40.2       |\n  | 120K       |     46.1      |   51.5    |       40.4       |\n\n------\n> **Q1:** Additionally, the trainable parameters in the overall model is not clearly reported.\n\n> **Q2:** It is not explicitly stated which variant of Qwen2.5-VL the Spatial-MLLM is built upon. The parameter size of the backbone model is also missing from the tables. Providing this information would improve the transparency and reproducibility of the work.\n\n- **R4: Backbone, model size, and trainable parameters.**  Thank you for pointing this out. Spatial-MLLM is built upon Qwen2.5-VL-3B. We report the parameter counts of different modules for our model in Table 3. During the training of Spatial-MLLM, we freeze the two encoders, allowing only the LLM backbone and the connector to remain trainable. We will provide a more detailed explanation in the revised paper.\n\n**Table 3.** *Parameter statistics for Spatial-MLLM.*\n\n| Model        | Total | Base Model (Qwen2.5) | 2D Encoder | 3D Spatial Encoder | Connector | LM Head |\n| :----------: | :---: | :------------------: | :--------: | :----------------: | :-------: | :-----: |\n| Spatial-MLLM | 4.96B |        3.01B         |    602M    |        867M        |   331M    |  297M   |\n\n------"}, {"review_id": "26CwMM0aBe", "reviewer": "Reviewer_MyVc", "summary": "The paper addresses a question/answering task in video which requires 3D spatial understanding to be able to solve these questions. Their main innovation is to combine a 3D visual encoder (VGGT) with the common 2D visual encoder (Qwen2.5-VL-3B) and demonstrate that adding 3D information significantly improves performance.\nMore specifically, this paper first samples frames from a video uniformly and encodes these using a frozen VGGT encoder to get 3D features. Then they create a subselection of these frames such that the maximum number of voxels of the 3D space is covered by these features (Sec. 3.2). Given this subset, they keep corresponding 3D VGGT features and extract Gwen2.5 features from these frames. On top of this they train a 'connector' which fuses these features and they also finetune an existing LLM (text tower) on the training set for answering these questions. Additionally, they use the GRPO RL-strategy from DeepSeek to boost performance. Finally, they annotate extra data closely following the protocol of VSI-Bench [18].\nResults on VSI-Bench, ScanQA, and SQA3D convincingly demonstrate the benefits of using 3D information. Their system outperforms open-source baselines fine-tuned on the same data (e.g. Qwen2.5). They also outperform proprietary models like Gemini 1.5 Pro and GPT4o on VSI-Bench", "questions": "See the two main weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n* Using 3D information makes sense and this is a convincing implementation.\n* Very good results on 3 different datasets.\n* Well-written\n* There is a reasonable ablation.\n\nMain weaknesses\n* Video-3D-LLM outperforms the current method when a ground-truth input map is fed to the model. What happens when this model is fed an estimate depth map from, say, DepthAnything V2?\n* Since the authors annotate extra data, I would expect a small ablation of the influence of the training data on the performance.\n\nWeaknesses (minor)\n* IMHO Spatial-MLLM-120k is rather misleading to present as a single dataset. In fact, it combines ScanQA, SQA3D, and new data closely following the VSI-Bench data curation pipeline [18]. Why not present it as three datasets and write you train on a mixture? That sets expectations more fairly.\n* The additional annotated data is for training only (which is fine, since is is also not explicitly mentioned as a contributions, but it should be more clearly communicated when it is introduced to set the expectations of the reader).\n* To me it was unclear that 'spatial intelligence' really means 3D. Many papers explicitly consider 2D spatial relations so I would advice to just call it 3D spatial intelligence to avoid any ambiguity. Or at the very minimum, 'spatial intelligence' should be defined better in the second sentence of the abstract.\n* Fig 2: please consider adding freeze icons to the 2D and 3D spatial encoder. And consider calling the 'Spatial Encoder' the '3D Spatial Encoder'.\n* The connector was a bit unclear to me. If I understand correctly the method only works if the 2D encoder and the 3D encoder extract exactly the same number of tokens from (almost) the same locations within the video. Then the combination happens on a per-token basis by projecting each token to the same number of dimensions and adding them. Please improve clarity on this section.\n* VSI-Bench also uses ScanNet. Therefore it is unclear where VSI-Bench ends and the extra data of this paper begins. This should be clarified in a final version of this manuscript.\n* The experiments on Gemini Pro result in many more frames per video to be analysed. But even though Gemini is advertised to be long-context, the extra number of frames may actually hurt performance. The authors could consider testing that.", "was_revised": true, "final_justification": "Solid and original paper as also acknowledged by the other reviews. I keep my rating and would advise the AC to consider this paper as an oral or highlight.", "rebuttal": "We thank the reviewer for the comments and for the time spent reviewing our paper. We address the main weaknesses and minor weaknesses raised by the reviewer as follows:\n\n---\n**Main weaknesses:**\n\n> **W1:** Video-3D-LLM outperforms the current method when a ground-truth input map is fed to the model. What happens when this model is fed an estimate depth map from, say, DepthAnything V2?\n\n* **R1: Video-3D-LLM with estimated depth map.** We thank the reviewer for the insightful question. We agree that it is necessary to evaluate the model’s performance when the input map is estimated by a pre-trained model rather than obtained from ground truth. Originally, Video-3D-LLM uses ground-truth depth and ground-truth camera poses to compute 3D positional encoding. In our rebuttal, we conducte an additional experiment using depth maps estimated by DepthAnything V2 (metric version), along with ground-truth camera poses as input. As shown in Table 1, Video-3D-LLM with estimated depth maps performs worse than both the original Video-3D-LLM and Spatial-MLLM. There are two possible reasons for this. First, the estimated depth maps is not consistent across frames, leading to inaccurate 3D position encoding. Second, although using metric version, there are scale misalignment between depth maps and camera poses, which may also affect the model's performance. In contrast, Spatial-MLLM directly uses a 3D spatial encoder to extract 3D information from videos and does not require ground-truth 3D inputs (including depth maps and camera poses), making it more applicable to real-world scenarios. We will clarify this in the revised paper.\n\n    **Table 1:** *Performance on ScanQA Benchmark.*\n\n    | **Method**                          | **GT Depth Input** | **GT Camera Input** | **BLEU-1** | **BLEU-4** | **METEOR** | **ROUGE-L** | **CIDEr** |\n    |------------------------------------|:------------------:|:-------------------:|:----------:|:----------:|:----------:|:-----------:|:--------:|\n    | Video-3D-LLM                       | yes                | yes                 | 47.1       | 16.2       | 19.8       | 49.0        | 102.1    |\n    | Video-3D-LLM (Estimated depth map) | no                 | yes                 | 42.2       | 14.6       | 17.4       | 43.6        | 87.6     |\n    | Spatial-MLLM                       | no                 | no                  | 44.4       | 14.8       | 18.4       | 45.0        | 91.8     |\n\n\n\n---\n\n> **W2:** Since the authors annotate extra data, I would expect a small ablation of the influence of the training data on the performance.\n\n- **R2: Ablation of training data.** We ablate the influence of training data and report the results in Table 2. The rows 1 to 4 correspond to the ablation of data scale. As shown, the model's performance improves rapidly when the dataset size increases from 0 to 90K and gradually slows down when approaching 120K. The row 4 and row 5 ablate the effectiveness of our newly annotated data. The results show that incorporating these additional data significantly enhances the model’s spatial reasoning ability on VSI-Bench, particularly in multiple-choice questions (which including relative distance/direction, route planning and appearance order questions).\n\n    **Table 2.** *Ablation of training data (evaluated on VSI-Bench).*\n\n    | Data Configuration                                  | Overall Score | Numerical | Multiple Choice |\n    |-----------------------------------------------------|:-------------:|:---------:|:----------------:|\n    | 25% of Full Data              |     43.0      |   47.5    |       38.3       |\n    | 50% of Full Data              |     44.0      |   48.3    |       39.4       |\n    | 75% of Full Data              |     45.7      |   50.8    |       40.2       |\n    | ScanQA + SQA3D + Newly annotated data (Full Data)   |     46.1      |   51.5    |       40.4       |\n    | ScanQA + SQA3D                                      |     36.7      |   45.2    |       27.7       |\n\n\n---\n**Minor weaknesses:**\n> **W3:** IMHO Spatial-MLLM-120k is rather misleading to present as a single dataset. In fact, it combines ScanQA, SQA3D, and new data closely following the VSI-Bench data curation pipeline [18]. Why not present it as three datasets and write you train on a mixture? That sets expectations more fairly.\n\n> **W4:** The additional annotated data is for training only (which is fine, since is is also not explicitly mentioned as a contributions, but it should be more clearly communicated when it is introduced to set the expectations of the reader).\n\n- **R3 R4: Clarification on Spatial-MLLM-120k.** Thanks for the suggestion. In the submitted version of the paper, we referred to the training data as Spatial-MLLM-120k for brevity, and we agree that this may be misleading. We will assign a separate name to the newly introduced data and clearly state that our model is trained on a mixture of three datasets in the revised paper. We will also clarify that the newly annotated data is used for training only, as VSI-Bench already provides test data targeting similar question types and model capabilities.\n\n---\n> **W5:** To me it was unclear that 'spatial intelligence' really means 3D. Many papers explicitly consider 2D spatial relations so I would advice to just call it 3D spatial intelligence to avoid any ambiguity. Or at the very minimum, 'spatial intelligence' should be defined better in the second sentence of the abstract.\n\n- **R5: Definition of 'spatial intelligence'.** Thanks for the suggestion. We agree that the term “spatial intelligence” may be ambiguous, and to avoid confusion with 2D spatial reasoning tasks, we will call it as 3D spatial intelligence in the revised paper.\n\n---\n> **W6:** Fig 2: please consider adding freeze icons to the 2D and 3D spatial encoder. And consider calling the 'Spatial Encoder' the '3D Spatial Encoder'.\n\n- **R6: Fig. icon and encoder name.** Thanks. We will add freeze icons and rename the “Spatial Encoder” to “3D Spatial Encoder” in the revised paper.\n\n---\n> **W7:** The connector was a bit unclear to me. If I understand correctly the method only works if the 2D encoder and the 3D encoder extract exactly the same number of tokens from (almost) the same locations within the video. Then the combination happens on a per-token basis by projecting each token to the same number of dimensions and adding them. Please improve clarity on this section.\n\n- **R7: Details of connector.** Both the 2D and 3D encoders use a spatial patch size of 14. The 2D encoder further reduces the token sequence length by merging tokens spatially (2×2 adjacent tokens) and temporally (every 2 consecutive tokens). As a result, the 2D encoder outputs exactly one-eighth the number of tokens compared to the 3D encoder (we excludes register and camera tokens). To align these tokens, we first rearrange the 3D encoder's tokens from the shape (seq, dim) to (t, h, w, dim) and apply the same spatial-temporal merging strategy as used in the 2D encoder. After merging, we rearrange the tokens back into shape (seq/8, dim), ensuring the two sets of tokens are precisely aligned in both position and number. Then we project both tokens into language model’s hidden dimension and fused them by element-wise addition. We will clarify this process in the revised paper. \n\n---\n> **W8:** VSI-Bench also uses ScanNet. Therefore it is unclear where VSI-Bench ends and the extra data of this paper begins. This should be clarified in a final version of this manuscript.\n\n- **R8: ScanNet scene usage.** To avoid data leakage, we filtered out all ScanNet scenes that are used in VSI-Bench during the generation of our additional training datasets. We will further clarify this in the revised paper.\n\n---\n> **W9:** The experiments on Gemini Pro result in many more frames per video to be analysed. But even though Gemini is advertised to be long-context, the extra number of frames may actually hurt performance. The authors could consider testing that.\n\n- **R9: Gemini 1.5 pro’s performance on different frame sampling rates.** Thank you for the suggestion. We evaluated Gemini 1.5 Pro's performance with different frame sampling rates (fps: 1, 0.5, 0.25 and 0.1) and report the results in Table 3. The results indicate that overall performance improves with more input frames, particularly when the initial number of frames is small. When the input frame number is large, further increasing the input frame number does not significantly enhance the model's performance (*e.g.*, when fps=1, the overall score only increases by 0.1 compared to fps=0.5).\n\n    **Table 3.** *Gemini 1.5 Pro's performance on VSI-Bench under different frame sampling rates (FPS).*\n\n    | FPS | Avg. Frames per Video | Overall Score |\n    |:---:|:---------------------:|:-------------:|\n    | 1.0 |          85           |     45.4      |\n    | 0.5 |          43           |     45.3      |\n    | 0.25|          22           |     43.5      |\n    | 0.1 |          12           |     39.5      |"}]

Author rebuttal:

Meta-review: The paper addresses video understanding that requires 3D spatial understanding, an area which current VLMs currently struggle in. The authors do so by combining a standard 2D vision encoder with a 3D vision encoder (VGG-T), and show how adding this 3D information significantly improves performance. Concretely, the authors fuse frozen VGG-T encoder features (sampled in a "space-aware" manner with those from Qwen-2.5-VL and train a ligthweight-connector to feed this into the LLM. In addition, the authors propose a new dataset (Spatial-MLLM-120K) and show how this dataset improves performance from both SFT and RL.

Reviewers appreciated the strong results of the proposed method on 3 datasets, the overall motivation of the approach to leverage 3D informaton from video, and the that the paper is well-written with convincing ablations. Moreover, the dataset proposed by the authors will be valuable to the community, particularly as the authors have demonstrated its utility. The authors also addressed reviewer comments well during the rebuttal phase.

For these reasons, the decision is to accept the paper with a spotlight. Please update the final camera-ready version of the paper in light of the rebuttal.
Decision: Accept (spotlight)