SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models
Authors: Pingyi Chen
Yujing Lou
Shen Cao
Jinhui Guo
Lubin Fan
Yue Wu
Lin Yang
Lizhuang Ma
Jieping Ye
Venue: NeurIPS 2025 poster ()
Keywords: MLLM
CV
Spatial Understanding
Primary area: deep_learning

Abstract: While vision language models (VLMs) excel in 2D semantic visual understanding, their ability to quantitatively reason about 3D spatial relationships remains underexplored due to the deficiency of spatial representation ability of 2D images. In this paper, we analyze the problem hindering VLMs’ spatial understanding abilities and propose SD-VLM, a novel framework that significantly enhances fundamental spatial perception abilities of VLMs through two key contributions: (1) propose Massive Spatial Measuring and Understanding (MSMU) dataset with precise spatial annotations, and (2) introduce a simple depth positional encoding method strengthening VLMs’ spatial awareness. MSMU dataset includes massive quantitative spatial tasks with 700K QA pairs, 2.5M physical numerical annotations, and 10K chain-of-thought augmented samples. We have trained SD-VLM, a strong generalist VLM which shows superior quantitative spatial measuring and understanding capability. SD-VLM not only achieves state-of-the-art performance on our proposed MSMU-Bench, but also shows spatial generalization abilities on other spatial understanding benchmarks including Q-Spatial and SpatialRGPTBench. Extensive experiments demonstrate that SD-VLM outperforms GPT-4o and Intern-VL3-78B by 26.91% and 25.56% respectively on MSMU-Bench. Code and models are released at https://github.com/cpystan/SD-VLM.

SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models
Pingyi Chen 12,," Yujing Lou Shen Cao Jinhui Guo', Lubin Fan
Yue Wu', Lin Yang', Lizhuang Ma Jieping Ye²
'Zhejiang University, Westlake University, Allibaba Cloud Computing, *Shanghai Jiao Tong University
Abstract
While vision language models (VLMs) excel in 2D semantic visual understanding, their ability to quantitatively reason about 3D spatial relationships remains under- explored due to the deficiency of spatial representation ability of 2D images. In this paper, we analyze the problem hindering VLMs' spatial understanding abilities and propose SD-VLM, a novel framework that significantly enhances fundamental spatial perception abilities of VLMs through two key contributions: (1) propose Massive Spatial Measuring and Understanding (MSMU) dataset with precise spatial annotations, and (2) introduce a simple depth positional encoding method strengthening VLMs' spatial awareness. MSMU dataset includes massive quantitative spatial tasks with 700K QA pairs, 2.5M physical numerical annotations, and 10K ofain-of-thoughh augmented samples. We have trained SD-VLM, a strong generalist VLM which shows superior quantitative spatial measuring and understanding capability. SD-VLM not only achieves statc-of-the- the-arr- performance on our proposed ASMU-Bennh but also shows spatial generalization abilities other spatial understanding benchmarks including Q-Spatial and SpatialRGPT- Bench. Extensive experiments demonstrate that SD-VLM outperforms GPT-44 and Intern- VL3-78B by 26.91% and 25.56% respectively on MSMU-Bench -enchh. Code and models are released at http://www....ooopttasni/////..e
Introduction
Vision-language models (VLMs) [1, 2, 3, 4, 5, 6, 7, 8, 9] have revolutionized how machines interpret and reason about visual content, achieving human-level performance on tasks like visual question answering (VQA). However, these models exhibit significant limitations understanding 3D spatial concepts, particularly regarding quantitative spatial reasoning, e.g., absolute distances and physical dimensions. Even state-of-the- models struggle with queries like "What is the size of the table in the image?", which reveals the gap between 2D perception and 3D quantitative spatial understanding. This limitation is particularly concerning given the increasing demand for models that can be operated effectively in real-world environments, such as robotics [10, 11], autonomous vehicles [12, 13), and augmented reality [14, 15].
Humans perceive spatial relationships based on building 3D cognitive map of scene in their mind. However, images, as the vision input of VLMs, are merely projections from the 3D scene to planes, losing much of the original 3D structural information. Although some works [16, 17, 18, 19, 20] explicitly use 3D scene representations as input to provide sufficient spatial information, we
"Work done during an internship at Alibaba Cloud Computing Equual contributions *Corresponiing author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

avoid introducing 3D structure data into current VLM architectures. One reason is that accurate point cloud data is hard to acquire in daily life. Besides, current 3D datasets suffer from uneven distribution and incomplete scene coverage compared with large-scale image datasets, limiting the training effectiveness and scalability with existing VIM frameworks. Recent studies [21, 22, 23] have incorporated depth maps as additional inputs to enhance the spatial comprehension of VLMs. However, depth maps alone are not enough to fully transform an image into a 3D structure due to the lack of camera intrinsics. The camera intrinsics are typically calibrated by geometric priors, such as using a checkerboard- calibration board. Inspired by this, an intriguing idea is that we provide not only depth map but also sufficient and accurate physical measurements, VLMs could implicitly learn the "camera intrinsics", having a better understanding of 2D to 3D mapping.
Therefore, we assume that the unsatisfactory spatial ability of existing VLMs stems from two fundamental factors: (1) the scarcity of datasets featuring precise spatial quantitative annotations, and (2) the deficiency of the image inputs, preventing VLMMs from fully understanding spatial context and conducting spatial reasoning.
Several studies have explored how to improve the spatial ability of VLMs by constructing spatial understanding datasets [24, 25, 26, 27, 28]. These datasets predominantly focus on basic qualitative spatial concepts, such as relative relations, which can be effectively addressed using only 2D visual cues. As a result, they fail to be competent in complex spatial reasoning. Datasets proposed in Spa- tialVLM [29] and SpatialRGPT [21] include both basic qualitative and quantitative spatial reasoning data. But their data construction pipeline relies on specific models, e.g., detection, segmentation. metric depth estimation, and camera calibration, instead of accurate physical annotations. Such a model-driven property may introduce systematic errors into the quantitative labels within the dataset.
Instead, we utilize 3D scene data with weal-world scales to provide comprehensive metrically accurate annotations for 2D images. Hence, we propose MSMU dataset, namely Massive Spatial Measuring and Understanding dataset shown in Figure I, a large-scale quantitative spatial reasoning dataset comprising about 25K images and 700K QA pairs (including 10K chain-of-thouglee samples) from 2K real 3D scenes, with 2.5M numerical annotations.
essential link bridging 2D visual perception and 3D scene understanding [21, 22, 23). We comprehensively compare various ways to integrate depth information into VLMs and introduce a simple but effective approach, depth positional encoding (DPE). Building upon the success of positional embeddings in Transformer architectures [30, 31], DPE introduces the information along the third dimension (z-axis) orthogonal to the input image plane. It effectively upgrades the model's ' spatial awareness from 2D to 3D space by simply adding the depth positional embeddings on image features. Equipped with the depth positional encoding, we have trained a spatial generalist, SD-VLM, with our proposed MSMU data. Extensive experiments show that SD-VLM has a remarkable advantage on spatial tasks over image-only models or other VLMs with depth priors. Our contributions can be summarized as follows:
Furthermore, depth priors serve
MSMU dataset, a large-scale dataset is proposed for quantitative spatial reasoning, with 700K QA pairs with 2.5M numerical annotations, generated from real 3D scenes. novel benchmark. MSMU-Bench, is also introduced to fully evaluate quantitative spatial reasoning capabilities of VLMs.
We analyze different ways to integrate depth into VLMs and design a simple but effective depth positional encoding module, that equips VLMs with explicit spatial priors, bridging the gap between 2D perception and spatial understanding.
Comprehensive experiments demonstrate that SD-VLM outperforms both previous VLMs and depth-encodee models achieving state-of-the-ar performance on quantitative spatial reasoning tasks, validating the effectiveness of MSMU dataset and depth positional encoding.
2 Related Work
Datasets Related to Spatial Analysis. The boom of multi-modal datasets facilitates the rapid development of advanced VLMs [32, 33, 34, 35]. Many studies work on 2D spatial relationships and comprehensively evaluate the 2D spatial ability of VLMs [24, 25, 26, 27, 28, 36, 37, 38] Although spatial concepts are explicitly or implicitly included these datasets, they are still insufficient for advanced and precise spatial understanding. Some works rely on complete 3D scans, which pose high

---

CCole Catrssion rriget
AINS and NN ANN
The done
CHlece of ATO
tight
AaRENC valty
laward
The of told
with deallyy
deviel chart
072 and he knia
on t4
it thouge zaup
Die wale
the black tnrket then
cruptlo hurt the with
win raconano
This the untread selval around
and neyelng bi
woydlng br
Figure 1: Demonstration of VQA pairs in MSMU. Our proposed dataset covers a range of quantitative spatial tasks involving multiple objects in the scene.
demands for integrating specific modules (e.g., point cloud encoder) that can effectively capture 3D information [40, 41, 42, 43, 44]. Q-spatial [45] has been proposed to benchmark the quantitative spatial ability of VLMs. Although it obtains accurate manual labels, the limited data volume restricts its applicability for model training. Several datasets have tried to tackle this issue by gathering abundant images from the web. However, these datasets often rely on estimation models and lack accurate 3D annotations [29, 21].
3D Spatial Understanding with VLMs. Recent studies have sought extend capabilities to 3D spatial understanding by integrating 3D information as inputs. On one hand, some works that concentrate scene-level 3D understanding, such as scene-level captioning and scenc-level visual question answering, which enable VLMs to directly process the entire scene by providing complex 3D representations like videos or point clouds [16, 17, 18, 19, 20]. On the other hand, several works discard explicit 3D inputs and use standard VLM backbones to achieve 3D spatial understanding by introducing depth information. For example, SpatialRGPT [21] incorporates the depth maps as auxiliary inputs. These depth maps are encoded and concatenated with RGB embeddings to provide additional spatial context. SpatialBot [22] is trained to adaptively call APIs to convert depth information into textual descriptions, which are then fed to the VLM as the prompt.
3 Problem Analysis
We review techniques for recovering 3D structures from an image. Given an image I, there exists mapping F transforming the image coordinates p [u, v, 1]" on the image, the corresponding depth value d and the camera intrinsics K are required for mapping p to the 3D point P IX z
3D points P, i.e. P P=F(I). For any homogeneous pixel
P=d-K
Therefore, an image can be mapped to its corresponding 3D points, if the depth map and camera intrinsics are available. The depth map alone is insufficient to derive a 3D point from a pixel due to missing intrinsics. The camera intrinsics could be calibrated when sufficient constraints based on the geometric priors, like using a checkerboard calibration board [46]. Theoretically, at least four line segment lengths (i.e., physical distances of corresponding 3D point pairs) in an image can be used as constraints to calibrate the intrinsics. We provide proofs in Section A.
Although 3D point clouds can be recovered from estimated depth maps and camera intrinsics [47, 48, 49, 50], we avoid explicitly integrating 3D modality into current VLMs considering the training effectiveness and scalability with existing VLM frameworks. To further avoid explicitly integrating the camera intrinsic as the input of VLMs, we tend to provide enough accurate physical measurements, enabling VLMs to implicitly learn a better mapping from 2D to 3D spatial concepts.
3

---

Scene Graph
3D to 2D Mapping
QA Generation
3D Mesh
Reaterize
Template
Paidence
Counting
Scate
Distaroe
Groun dinnnigg
Posidon
RGB Irstance arnobatkens
Desti Mapa
Detailed Re abeliing
Informaiio Agprogate
Oijici
Image-oti Parr
aptior
Ellcing fleoorm ng Pait with Relerene Dijjcde
(tcy)) "Laal: The yolkow sake
ia desply lagas)
Loglol
'Maee: Mile "hhr'' (Moo)
"Capt"t "Capat:1dyeii "Seetoon"
Scares insgue Deyid Mapa. Olgads,
Figure 2: Overview of the data generation pipeline of MSMU. It consists of scene graph construction, 3D to 2D mapping, and QA generation.
We observe that current LLMs and MLLMs [51, 52, 53, 2, 9] demonstrate an insufficient grasp of spatial concepts like the physical sizes of objects. Their training data is not equipped with abundant spatial concepts or lacks precise numerical annotations [29, 21, 45], which motivates to propose massive and precise spatial data.
In Section 4, introduce the MSMU dataset with massive spatial measuring data. In Section we delve into multiple approaches to fuse depth into VLMs and design a simple but effective depth- encoding strategy to integrate depth maps into VLM inputs. In Section 6, conduct comprehensive experiments to investigate this problem.
4 MSMU Dataset
We elaborately design series of spatial questions, establish data generation pipeline acquire spatial VQA pairs with metrically accurate annotations, and leverage L.M collaboration to generate chain-of-thought (CoT) augmented pairs.
4.1 Task Categories and Definitions
In contrast previous spatial datasets [21] which mainly focus the basic spatial understanding tasks, MSMU is designed to introduce more complex quantitative spatial tasks requiring comprehen- sive and precise spatial perception ability of VL.Ms. MSMU contains fundamental quantitative spatial tasks, including scale estimation (e.g., height, width, or size), object grounding (2d coordinates). also encompasses more sophisticated quantitative spatial tasks involving multiple objects or numerical outcomes, such as relative position (c.g.. before/after, left/right, stand higher/lower), absolute distance measurement between two objects, scale comparison (e.g.. bigger/smaller, biggest/smallest), and reference object estimation (given one scale/distance, predict the scales/distances of other objects). Furthermore, the counting task is included assess the model's ability discern the quantity of objects present. In addition, to eliminate the hallucination problem in VLMs, we introduce existence tasks in MSMU. In these tasks, a nonexistent object is deliberately chosen for question-answerine construction. The VLM must accurately detect the absence of the object and refrain from providing misleading information. Several examples of QA pairs are shown in Figure
4.2 Data Generation
As demonstrated in Figure 2, starting from 3D scene point clouds, we first collect the spatial information (e.g., locations, sizes, relative distances) of objects in the scene to construct scene graph. Next, we rasterize 3D instances onto images and establish a 3D-to-2D mapping, which enables transferring spatial annotations to images. We also perform filtering on both images and objects to ensure the quality of the QA pairs. Finally, we design human-verified QA templates and employ LLM collaboration generate a rich set of QA pairs.

---

Nunerial Prodsko
Parchh Comparizen
Ditact
Sourcz
Soal
Giceoing
Disasue
Cargoeire
Couning
Relonae Resaxiig
Spatia [23]
BGBD insges
X
x
MIMLL Beack
Sparia MM [2]]
Imsgie
x
x
X
RuleSpurial (24)
3D
x
x
X
SpeistRrPP
Spricil
Imggs
MSMU
Trsk Corpothesseeeees
Figure 3: Comparison of different spatial datasets and benchmarks.
Building Scene Graph. Given 3D point cloud of a scene, we first construct scene graph (stored as a ISON file) to systematically organize all annotations and metadata. This graph includes the object categorization and corresponding 3D spatial localization data which provides bounding boxes for each object, defined by centroid coordinates and dimensional parameters (width, length, height).
Rasterize 3D instances to 2D images. We rasterize 3D instances onto images masks using official tools [54]. This process bridge an object in the 3D scene and 2D image plane, making transferring spatial annotations from 3D scene graph to each image feasible.
Image Filtering and Object Selection. We first sparsely sample the RGB images to reduce redun- dancy. After that, we carefully select objects in each image, which is guided by three principal criteria: (1) Prevalence and functionality. We focus on objects demonstrating clear functional purposes which commonly encountered in indoor environments. Architectural components (c.g., walls, ceilings) are excluded due to their limited interactive potential. (2) Instance visibility. Objects that are partially occluded (e.g., chair mostly hidden behind table), truncated by image borders (e.g.. only corner of a table is visible), or too small to annotate reliably (e.g., distant objects occupying fewer than 50 pixels) excluded from our dataset. (3) Semantic disambiguation. Addressing linguistic ambiguity is important before generating annotations. For example, tables which exist in one image may vary in color texture but are all labeled "table", which brings noisy correspondence and finally misleads VLMs. To mitigate this issue, we resort to Qwen2.5-VL. [8] to re-label these objects with more detailed descriptions, such as "the white table" or "the wooden table". Finally, filter non-informative images that have no valid objects.
Template-based Generation. We carefully design set of templates based on the task definitions which include various placeholders, denoted as [.]. For instance, one template for measuring the size of single target object is structured as follows: "Q: What is the size of [object AJ. A: The size For each image, we enumerate the selected objects
of [object A] is [Length] x WWdthh][Heighee and replace these placeholders with the corresponding object labels spatial annotations. In tasks involving two or more target objects, we also meticulously craft instructions that incorporate all relevant object labels and spatial information. More template details are in Section B.
Eliciting Reasoning Path with LLM collaboration. Inspired by SpatialPrompt [45] which signifi- cantly improves the quantitative spatial ability of VLMMs by eliciting reasoning paths with reference objects, we augment the QA pairs with CoT reasoning rationale via LLM collaboration. Specifically, we randomly select one object as the reference object and combine its spatial annotations along with the image as inputs to the advanced VLM, Qwen2.5-VL. The VLM is then prompted to construct reasoning path that leverages the reference object to infer the spatial properties of another object within the image. Subsequently, we utilize a large language model, DeepSeck- -3,, to assess and filter the CoT pairs by evaluating the factual consistency and logical coherence. Related prompts provided in Section B.
Dataset statistics. We employ this data generation pipeline to construct VQA pairs from ScanNet [55] and ScanNet++ [54]. The resulting MSMU dataset contains 2K scenes, 25K images, 75K objects, 700K QA pairs, and 2.5M numerical values, covering wide range of quantitative spatial tasks, as shown in Figure 3 (left), Besides, the CoT augmented group, named MSMU-CoT, consists of 10K quantitative spatial reasoning QA pairs.
4.3 MSMU-Bench
Existing spatial datasets struggle with annotations that lack precision, limited data volume, or insufficient task types [29, 22, 28, 45J. To address this issue, we have meticulously developed MSMU-Bench, held-out benchmark from MSMU, designed to rigorously assess the advanced spatial reasoning capabilities of VLMs. As shown in Figure 3 (right), USMMU-Bencl contains more
5

---

LLM
LIM
LLM
VIT
VIT
VIT
seline depalic
VIT
VIT
qleyy
(R) Depch as image
prompt
Dopth token
Figure 4: Illustrations of different approaches for integrating depth information.
quantitative QAs compared to other spatial benchmarks. Comprising about IK spatial VQA pairs, this benchmark features samples from unseen scans.
We leverage GPT-4 assess the responses generated by different models MSMU-Bench. In the case of qualitative questions, GPT-4 assigns a score of the model's answer on scale from 0 to 1. For quantitative queries, GPT-4 first extracts all numerical items in responses. Then, we compute the success rate by setting a threshold. For a estimated distance d and its ground truth value d", this ratio d d" 5
calculated as d max included in Section C.
The prompts used for scoring by GPT-4 and other details
Integrating Depth into VLMs
In this section, we explore various methods to integrate depth information into VLMs, and introduce simple but effective depth encoding method.
5.1 Approaches for Integrating Depth
Previous research has tried various techniques to utilize depth maps for VLMs, as shown in Figure 4. The first technique, "depth as image", was initially introduced in SpatialRGPT [21]. This method treats the depth map regular image and leverages a vision encoder to convert depth maps into embeddings. Besides, a learnable depth connector is required to align the depth embeddings with image embeddings. The second technique, "depth as prompt", draws inspiration from [22], of which depth values are retrieved by an API and subsequently textualized as prompts. The third technique, "depth token", directly concatenates the depth embeddings with image embeddings.
The first method relies on the vision encoder to extract features from depth maps and introduces additional modules for training. In the second case, it unavoidable teach the model to utilize depth APIs. The third approach has to extend the sequence length and finally compromise the efficiency of training. Therefore, depth data into VLMs, requiring minimal structural changes and training cost, while still being able to enhance spatial capabilities.
aim to discover straightforward integration method for incorporating
5.2 Depth Positional Encoding
We introduce the depth positional encoding (DPH), which encode the depth maps into depth positional embeddings, allowing for straightforward combination through addition.
An input depth map is represented D #xx1,, Suppose the image feature map output from CLIP is Eimage E R xw' xd We first divide the depth map into small patches P(i,j), matching the number of image patches. We then use adaptive mean pooling to calculate the mean depth values for each patch, and get pooled depth map D' E "R xW'x1 The compact nature of these patches ensures that the average depth values provide adequately detailed depth positional information. Alternative ways of pooling depth maps are also explored the ablation study.
Following [30], we utilize sine and cosine functions of varying frequencies to generate the depth positional embeddings Edepth HB xW' xd. which can be formulated as follows:

---

Large Language Model
Token
Add
"What the vize of the yellow table?"
Input
Vision Encoder
Depth Encoding
Depth Estimation
Bdi Image
Dopth Map
Figure 5: The architecture of SD-VLM designed to effectively integrate spatial information into vision-language models. We incorporate an additional depth estimation module, which is particularly useful when the ground-truth depth map is unavailable.
Edepth (i.j.tt) = sin (D((..)/1000002//) Edepth (i.j.
(dl +)) cos ( (D(00.)//00000///)
(2)
where D'(i,.) denotes the depth value, (i,j) represents the patch index, and / = 0,-... Each dimension of the depth positional encoding corresponds sinusoidal wave.
- ,
Finally, we obtain the final vision embedding Evision by integrating depth positional embeddings into image embeddings by adding, expressed as:
Evision
E gimage
Edepth
Depth-empowered vision embeddings are flattened and sent the LLM as the final input.
The architecture of our proposed model is visualized Figure 5. The model consists of a vision encoder to encode image features, depth encoding module to incorporate depth information, and a large language model to process sequences of tokens. When depth maps are unavailable during inference, employ external depth estimation model to generate the depth map. This allows model to adapt to various datasets and scenarios effectively.
Experiments
6.1
Inplementation Details
SD- VD-V is built upon pretrained LLaVA-1.5-7B The model fine-tuned with LoRA [56] on MSMU for epoch. The model is trained on 8 V100 GPUs, with the batch size of per GPU, using 32 GPU hours. The vision encoder is CLIP-ViT/14. The external depth estimation model is Depth-Anything-W V2 [48]. In the training phase, the vision encoder remains frozen. The learning rates for LLM and the projector are 2e-4 and 2e-5, respectively. The threshold for GPT-4 evaluation in MSMU-Bench 1.25.
6.2 Results on MSMU-Bench
We evaluate the spatial reasoning ability of the most advanced models on MSMU-Bench. The tested models encompass variety of proprictary VLMs, including GPT-4o [2] and Gemini-2 [3], well as open-source alternatives that vary model scale, such as Qwen2.5-VL [57]. Intern-VL3 [9], and LLaVA-1.5-7D [4]. Additionally, we include models that incorporate depth information like SpatiaIRGPT [21] and SpatialBot [22]. Furthermore, we broaden our assessment to encompass LLMs, such GPT-4-Turbo [53], Qwen2.5 [51], and DeepSeek- V3 [52]. The purpose of this evaluation is to measure these models' capacity to infer accurate responses based on common knowledge without visual inputs. The results shown in Table 1.
Among all the baseline models, our SD-VLMM stands out with the highest success rate 56.31%. excels not only in basic spatial tasks such as scale estimation, where it achieves a 51.35% success rate,
7

---

Exis- Ience
Object Counting
Scale Estimation
Relative Position
Absolute Distance
Scale
Ref. Object Estimation
Model
Grounding
Comparison
Large Language Models (LLMs)) Only Text Input
GPT-4-Turbo[33]
12.76
5.21
13.51
12.64
24.84
7.50
36.79
12.04
15.66
Qwen2.5[51]
0.00
0.78
13.79
0.00
16.04
1.77
DeepSeck-V3[S2]
0.00
5.24
1.54
6.90
10.56
0.00
25.47
5.24
7.39
Vision Language Models (VLMs): Image
Input
GPT-44022]
44.68
41.67
3.86
27.59
67.08
20.00
54.72
2.09
32.28
Gemini-2[3]
38.30
43.75
23.94
19.54
54.66
12.50
69.81
18.85
35.17
Qwen2.5-VL-722BB3
59.57
35.42
13.79
57.76
2.50
9.95
Qwen2.5-VL-32BB3
29.79
41.67
10.81
18.39
60.25
2.50
10.99
27.59
Qwen2.5-V1.7PIj
42.71
26.32
13.33
VL3-8B[91
4.63
18.39
2.50
LLaVA 1.5-7B[6]
36.46
20.69
5.00
Epyth-E Vision- Esion-Language Models
Image + Depth Map + Text as Input
Sp
10.64
46.88
15.83
28.74
66.46
5.00
50.94
8.90
29.17
SpatialRGPT[21
10.64
36.46
20.08
17.24
60.25
15.00
62.26
9.95
28.98
Ours
87.23
47.92
$1.35
42.53
75.16
40.00
55.66
46.07
56.31
Ours w/ MSMU-C6T
87.23
42.71
$1.74
49.43
73.29
50.00
69.81
49.32
$9.19
Table Overall results various models on ASMU-Bench. and depth-encoded VLMs as a comprehensive comparison.
We report the results of LLMs, VLMs,
Questia::
Question:
pipc?
width wallshstro?
mallstehho
GPT dec The stoc of pipe Zens.
GPI-4e: Iis usable dekmmiiac cusl wakhol slxtros.
LLaVa--l. Aol. The cruit.
klaVA-1.5 h-1.; The wall shelve an 12 inchecidde.
SystisRRCPT: sine pape
SpetiaRRCPP: The wishh dedve sackos
pipe
Ours:
Quesiion:
detersine
shuuld
The beizhe the cable in 77.0 cnn, can you micacue che beight frame?
appiees eigttficastt
comparing propotions in imago,
fruse
GFI-4s: snabli
cast beight frams: fna
aquastemanl tume times kright sdde. Thenfom, maattphynng the teight by diew gives extirated beigle door frone: 77.8 en cn. Canceting Bibs racken, the bight of doer france sppresinsoo 2:31 metre.
the imags
LiaVV.d.:: Nia
SpstialRCPT: boight freac naters
Figure 6: Examples of response from various models.
but also demonstrates significant prowess in complex spatial reasoning tasks like reference object estimation, reaching 40% success rate compared to the second-best model's 20%. Additionally, our model's performance in the existence task is noteworthy, with the highest success rate of 87 23%, indicating its robust ability to identify the presence or absence of objects within images, with less hallucination. We also illustrate various models' responses in Figure
Within the group text-only Large Language Models, Qwen2.5, DeepSeek- V3, and GPT-4-Turbo show average success rates of 4.63%, 7.39%, and 15.66%. These results reflect the complexity of MSMU-Bench, where questions demand more than just common knowledge and benefit greatly from visual information integration. It shows the potential of our proposed MSMU dataset for improving the general spatial ability of VLMs.
MSMU-COT can strengthen the model's quantitative spatial ability. We observe that employing additional CoT pairs during training can improve the spatial reasoning abilities of VLMs, particularly in complex tasks such scale comparison, with improvement in success rates from 55.66% to 19. The average success rate also increases from 56.31% to 59.19%.
Results on Other Spatial Benchmarks
We have also evaluated our model on other spatial datasets including Q-Spatial++ [45] and SpatialRGPT-Bench [21]. is important to note that SpatialRGPT-Benee refers to objects using bounding boxes masks, which are not directly compatible with language-driven models. To address this, we have refined the benchmark by using Qwen-2.5-VL to re-annotate the objects and select the quantitative (object scales and distances) and qualitative tasks (relative positions and scale compar-

---

SRGPT-Bench
Model
O-Spatial++
Quan.
Qual.
Model
MSMU-Bench
Baseline
46.73
GPT-4o[2]
52.0
13.0
60.5
Gemini-2[33
51.0
23.0
57.6
depth as image
22.64
(wen2.5-VL-728[88
43.6
16.3
61.4
depth prompt
48.78
depth token
35.72
Intem91
53.6
23.5
62,2
I.LaV--.5-78[55]
16.2
26.3
DPE w/ estimated depth
55.35
SpatialBot[22]
33.7
13.2
55.9
DPE-leurnable
56.18
Spatial
43.5
28.7
57.8
DPF-sincos
56.31
Ours
56.2
33.3
65.5
Table 2: Left: Results other spatial datasets (Q-Spatial++ and SpatialRGPT-Bench) Right: Comparison of models with various approaches of incorporating depth priors.
Model
MSMU:-Rend Rench
Q-Spatial++
SpatialRGPT Bench (Quan.)
Sparia/RGPT: Bench (Qual.)
LLaVA-1.5-78
17.3
11.2
10.3
26.3
LLAVA-1.5-7
DPE
18.8
23.0
184
27.5
Table 3: Overall results of models which are fine-tuned with LLaVA-I.5-mix6Gf and tested on spatial datasets.
isons) for evaluation in SpatialRGPT-Bench. We follow the official setting when evaluating these benchmarks. As shown in Table 2 (left), SD-VLM achieves 56.2% success on Q-Spatial++, surpassing all the other baselines. SD-VLM also achieves the state-of-the-art performance on the quantitative (33.3% success rate) and qualitative tasks (65.5% success rate) of SpatialRGPT-Bent which contains many outdoor scenes, revealing the strong generalization capabilities of our model. More details are included in Section
6.4 Comparison between Different Depth Integrations
We have examined various methods for incorporating depth information into models, shown in Table 2 (right). The result of "depth as image" suggests that treating depth maps as images may not be effective way to integrate depth data. The subpar performance of "depth as token" suggests that adding extra tokens not only increases training costs but also makes it more challenging for VLMs to learn the interactions between depth maps and image embeddings. The approach of "depth prompt" achieves a slight improvement from 46. 73% to 77... Our proposed depth positional encoding demonstrates marked advantage, achieving success rate of 56. 58%.
An alternative way of encoding depth maps involves utilizing learnable layers that adaptively condense the depth map and convert into depth positional embeddings. As shown in the bottom of Table (right), the success rate of 56.18% is comparable to the performance of sinusoidal depth positional encoding.
We have trained our model with estimated depth maps from Depth-Anything V2. Although its performance is not as good as the one with ground-truth depth maps due to the relatively noisy depth, still shows significant superiority over other depth integration methods. More experimental results on depth are included in Section E and F.
Further investigation of depth encoding. We have conducted further investigations into the efficacy of our proposed depth positional encoding technique. We resort to general visual instruction datasets, LLaVA-I.5-mixhe [5]. to fine-tune LLa with DPE in its instruction following training stage. This dataset, which is not explicitly equipped with spatial knowledge, allows us to examine the extent to which depth positional embeddings can enhance VLM's 's spatial reasoning ability implicitly without massive spatial data. The results are demonstrated in Table 3. Our observations indicate that while models trained on general datasets may not excel on spatial tasks, incorporating depth encoding can still enhance their performance across all three spatial benchmarks, with notable 25% relative improvement in model accuracy, which reflects the potential of depth positional encoding in eliciting the model's spatial reasoning ability.

---

Depth Estimator
MSMU-Bench
Q-Spatial++
SpatialRGPT-Becch
Average
DepthAnything UniDepth
56.3
56.2
33,3
48.6
56.2
54.7
32.0
47.6
Table 4: Overall results of different depth estimators.
No Noise
= 0.1
8 = 0.3
8 = 0.5
6 = 0.7
No Depth
56.3
55.1(-1.2)
54.0(-2.3)
53.3(-300
51.4(-4.)
46.7(-9.6)
Table 5: Model performance with various depth noise.
6.5 Robustness of DPE
Variations of depth estimators. We have conducted an ablation study on the depth estimation backbone by replacing DepthAnything with another powerful depth estimator, UniDepth [49]. As shown in Table 4, our model maintains competitive performance across spatial benchmarks. On MSMU-Bench, the scores are nearly identical (56.3% vs. 56.2%). The average performance decreases slightly from 48.6% to 47.6% when switching to UniDepth. The results indicates that our model has learned generalizable depth priors rather than overfitting any specific depth-estimation architecture.
Depth noise. We have conducted further ablation studies to evaluate the robustness of our model against depth estimation noise on the MSMU-Bench as shown in Table 5. Specifically, we inject zero-mean Gaussian noise with different standard deviations into the normalized depth maps. As shown in Table 5, as the noise level 8 increases, the performance gradually declines from 56.3% under no noise to 51.4%. Despite this degradation, the model maintains competitive performance even under significant noise conditions, outperforming the setting without depth input by clear margin. These results indicate that our DPE exhibits strong robustness to depth perturbations, effectively leveraging depth priors.
6.6 Performance General Benchmarks
In this section, we evaluate whether incorporating spatial VQA data and depth information affects the model's performance general VQA benchmarks. To ensure fair comparison, incorporate the LaaVA-1.5-mix666 dataset into our training data. As shown in Table 6, our model still achieves superior performance MSMU-Bench (55.8% 19.5%). Regarding other benchmarks, shows comparable performance to the baseline and even exhibits a slight edge on Whatsup, GQA, and VQA-v2. The overall competitive results confirm that our approach can significantly enhance spatial understanding abilities of VLMs without compromising general capabilities.
model
MSMU-Bench
Whatsup|26]
GQA[58]
TEXIQQA591
VQA-v2(00)
Vizwi/61]
LLLVA-1.5.7
78.5
Ours
55,8
60.9
62.9
57.5
79.1
49.2
Table 6: Comparison of SD-VLM and base model performance on general benchmarks.
Conclusion
In this work, we identified a critical gap in the ability of Dision-Languee Models (VLMs) to perform quantitative spatial reasoning. To address this, we developed MSMU, a large-scale dataset comprising 700K QA pairs and 2.5M numerical physical annotations derived from real 3D scenes, designed to provide precise metric supervision for enhancing VLMs' spatial reasoning capabilities. We introduced simple but effective depth positional encoding module that integrates the third-dimension information into the VLM architectures, effectively upgrading the model's spatial awareness from 2D to 3D. This innovation was shown to significantly enhance spatial reasoning abilities, outperforming both RGB-only VLMs and depth-encoded VLMs. We anticipate that our contributions will pave the way for further advancements in VLMM* spatial reasoning capabilities, enabling more effective application in real-world environments.
10

---

Acknowledgement. Pingyi Chen was supported by Alibaba Research Intern Program. Yujing Lou was supported by Alibaba Postdoc Program. We thank Prof. Xiaoguang Han, director of the GAP Lab (Chinese University of Hong Kong, Shenzhen) and the lab members for their assistance with the dataset.
References
[1] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309. 02223099.722... 2023. 1
[2] Aaron Hurst. Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 1, 1,4,7. 9
[3] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Sean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023. 1, 1,7.8,9
[4] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NuerIPS, 2023. 7,7
[5] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. 2023. 1,9
[6] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprini arXiv:2408.033: 03326, 2024. 8
[7] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint 0rXiv:2308.12966 2023.
[8] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint WXXv:2550.113923 2025. 5, 8,
[9] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10477 2025. 1, 4, 7,
[10] Gao Jensen, Sarkar Bidipta, Xia Fei, Xiao Ted, Wu Jiajun, Ichter Brian, Majumdar Anirudha, and Sadigh Dorsa. Physically grounded vision-languag models for robotic manipulation. In ICRA, 2024.
[11] Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingnan Zhang, Tseng-Wei Edward Lee, Kuang-Huel Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn, Sergey Levine, and Brian Ichter. PIVOT: iterative visual prompting elicits actionable knowledge for vlms. In ICML, 2024.
[12] Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hong- sheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In CVPR, 2024.
[13] Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In ECCV, 2024.
[14] Mikhail Konenkov, Artem Lykov, Daria Trinitatova, and Dzmitry Tsetserukon. Vr-gpt: Visual language model for intelligent virtual reality applications. arXiv preprint arXiv:2405.00 15377, 2024.
11

---

[15] Junlong Chen, Jens Grubert, and Per Ola Kristensson. Analyzing multimodal interaction strategies for llm-assisted manipulation of 3d scenes. In 2025 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pages 206-216. IEEE, 2025. 1
[16] Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chur Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In ICML, 2024. 3,3
[17] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. NeurIPS, 2023. 1,
[18] Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In CVPR, 2024. 1,3
[19] Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with lsionnlanguage models. wxXiv22501.01425 2025. 1,3
[20] Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual reasoning. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision ( (WACV), 2025. 1,3
[21] An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision- laiiuula models. In NeurIPS, 2024. 2, 3, 4, 5, 6, 8,
[22] Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406. 13642, 2024. 2, 3, 5, 6, 7, 8,
[23] Erik Daxberger, Nina Wenzel, David Griffiths, Haiming Gang, Justin Lazarow, Gefen Kohavi, Kai Kang. Marcin Eichner, Yinfei Yang, Afshin Dehghan, et al. Mm-spatial: Exploring 3d spatial understanding in multimodal Ilms. arXiv preprini arXiv:2503.1311 13111, 2025. 2
[24] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635-651, 06 2023. 2
[25] Kaiyu Yang, Olga Russakovsky, and Jia Deng. Spatialsense: An adversarially crowdsourced benchmark for spatial relation recognition. In ICCV, 2019. 2
[26] Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's "up" with vision- -ssion-la models? investigating their struggle with spatial reasoning, In EMNLP, 2023. 2, 10
[27] Fatemeh Shiri, Xiao Yu Guo, Mona Far, Xin Yu, Reza Haf, and Yuan-Fang Li. An empirical analysis on spatial reasoning capabilities of large multimodal models. In EMNLP, 2024. 2, 5
[28] Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. RoboSpatial: Teaching spatial understanding to 2D and 3D vision-language models for robotics. In CVPR, 2025. 2,5
[29] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In CVPR, June 2024. 2, 3,4,
[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 30, 2017. 2, 6
[31] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[32] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In CVPR, 2024. 2
12

---

[33] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: massive multi- disciiiiie multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024. 2
[34] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, 2024.
[35] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang. Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can but not perceive. In ECCV, 2024.
[36] Georgios Pantazopoulos, Alessandro Suglia, Oliver Lemon, and Arash Eshghi. Lost in space: Probing fine- gr-ineraindd spatial understanding vision and language resamplers. NAACL, 2024.
2
[37] Zheyuan Zhang, Fengyuan Hu, Jayjun Lee, Freda Shi, Parisa Kordjamshidi. Joyce Chai, and Ziqiao Ma. Do vision- language models represent space and how? evaluating spatial frame of reference under ambiguities, 2024. 2
[38] Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Kraehenbuehl, and Vladlen Koltun. Does spatial cognition emerge in frontier models? arXiv preprint arXiv:2410.00 X1v:2410.006668, 2024.
[39] Julius Mayer, Mohamad Ballout, Serwan Jassim, Farbod Nosrat Nezami, and Elia Brumi. ivispar- interactive visual-spatial reasoning benchmark for vlms. arXiv preprint 250.v:250..03214,
2025.
[40] Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. In ICLR, 2023. 3
[41] Ruiyuan Lyu, Jingli Lin, Tai Wang, Xiaohan Mao, Yilun Chen, Runsen Xu, Haifeng Huang, Chenming Zhu, Dahua Lin, and Jiangmiao Pang. Mmscan: A multi-modal 3d scene dataset with hierarchical grounded language annotations. NeurlPS, 2024. 3
[42] Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In ECCV, 2020.
[43] Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d objects. In ICCV, 2023. 3
[44] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe, Scanqa: 3d question answering for spatial scene understanding. In CVPR, 2022.
[45] Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models. In EMNLP, 2024. 3, 4, 8,8
[46] Zhengyou Zhang. A flexible technique for camera calibration. IEEE Transactions pattern analysis and machine intelligence, 22 1330-1333, 2002.
[47] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10371-10381 2024.
3
[48] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao Depth anything: Unleashing the power of large-scale unlabeled data. In NeurIPS, 2024. 3,
[49] Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Lue Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In CVPR, 2024. 3, 10
[50] Shengjie Zhu, Abhinav Kumar, Masa Hu, and Xiaoming Liu. Tame a wild camera: in-the-wild monocular camera calibration. NeurIPS, 2023.
13

---

[51] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15115, 2024. 4,7,
[52] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 techmical report. arXiv preprint arXtv:2412.19437,, 2024. 4, 7,8
[53] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint vrxiv:2303.088744, 2023. 4, 7,
[54] Chandan Yeshwanth, Yuch-Cheng Liu, Matthias NieBner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In ICCV, 2023. 5
[55] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. CVPR, 2017.
[56] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhi Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2022.
[57] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024." 7
[58] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. CVPR, 2019. 10
[59] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019. 10
[60] Yash Goyal, Tejas Khot, Douglas Summers- Stay,, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. 10
[61] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018. 10
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitaions are discussed in the supplementary.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
Complete proofs are shown in the supplementary.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: It is shown in Section 6.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification:
Code and data will be open spur-seurced after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips..
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececccccccccct http://///iieeeeeeeeuuubiiiiiiii
/CodeSubmie ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: is included in Section6.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: No error bars are reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: is included in Section6.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: /ttp///eerrp..///ee ct.ps.......ooooo.... cctppuuic//thiccsodeeeelllle
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
The research is conducted in the paper conform.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts dicussed in the supplementary.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Non-commercial research and educational purposes of our source data are permitted.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
19

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: new dataset is introduced in this paper. The dataset details are discussed in Sec. 4. The dataset will be open-sourced after acceptence.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
20

---

Answer: [Yes]
Justification: is described in the Section 4.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: htmp::///neuree //teurips...e/eeeee /ooooorreeeee/2222 /LLM) for what should or should not be described.
21

---

A Proof of Mapping from 2D to 3D with Distance Constraints
In this section, we prove that given an image with its depth map and enough annotated physical lengths, the mapping from the image to its 3D structure can be established.
Mapping from 2D image 3D points. Given an image Z, there exists mapping FF transforming the image Z to 3D points P. i.c. p = =F) For any homogeneous pixel coordinates p [24, V, 1 on the image, the corresponding metric depth value and the camera intrinsics are required for mapping to the 3D point P [X, Y. 2
X
[u Cz) d. fg
P=
Y
= d
K-
K
p=
(U
Cy Tu
(4)
Z
K the camera intrinsic matrix with four unknown parameters,
fm
0
K
0
fy
Cy
(5)
0
0
Constraints Based on Metric Distances. Camera intrinsics necessary for mapping depth map to 3D structure. 3D point coordinates in camera coordinate system are hard to obtain daily scenarios, while measuring metric distances between two points is feasible.
Suppose two pixels p1 and p2 are the endpoints of line segment with physical length L in the image I. The corresponding depth values are dj and d2 and 3D points from the mapping are PI and Pa. Hence, the physical length calculated by
= Pi Pal
(6)
Explicitly, the constraint based the metric distance is
L
11
1/2 Cz
d2
)2
01 Cy
U2
Cy d2) + (d1 - d2)
fI
Sg
f
This is one nonlinear equation for four unknowns fax fy, Cz. Cy. Suppose there N line segments labeled with physical lengths in an image. We define
Ealfar Cay Cy) Pl Piall L2 i = NN.
The residual vector is
r r= [Ei, ".I]"
(9)
Hence, the intrinsic parameters are estimated by optimization algorithms with the objective:
min
||r||2
(10)
Solving above optimization problem needs at least four segments (N > 4) with ground truth length. Actually, the depth values are sometime noisy and the ground-truth depth map is difficult to acquire in daily scenarios. Besides, if the given depth map is relative, i.c. d dret + b. there exists two more parameters. Empirically, abundant constraints would produce a more robust estimation, which improves robustness to noise and improve stability.
Theoretically, given an image with its depth map, we can fully mapping pixels to corresponding point cloud when enough annotated physical lengths are provided. We believe that providing enough physical labeled lengths in images would facilitate the spatial understanding of images.
B Details for Data Generation
In this section, we provide more details for the data generation procedure.
22

---

B.1 Prompt Details
Prompts for semantic disambiguation. We crop instances from an image, which are fed to Qwen- 2.5-VL with the prompt as below:
Describe the object class the image and directly return a term. For example, the red car, the wooden table, the man in white.
Output:
Prompts for CoT data generation. To clicit reasoning paths with reference objects, we randomly select an object the reference object, combining its spatial annotations and the image as inputs to Qwen2.5-VL with the prompt as below:
Please help me rephrase the following VQA (Visual Question Answering) pairs improve their rationale. I will give you an image which shows an indoor environment and contains various objects. Based on the image. I will also give you question and answer, the question containing reference object. You need to propose robust step-by-step plan to answer the question by using the reference scales and the information from the image.
For example:
Q: The height of the chair is 0.7 m, can you measure the height of the wooden table?
A: Since the height of the chair is 0.7 m, I think the height of the wooden table is 1.4 m
Example Output:
To determine the height of the table. should think it step by step carefully. Firstly, the chair height can be used as reference, which is known as 0.7 m. The wooden table appears to be about double the height of the counter. So, the height of the wooden table 1.4 m.
Please process the following VQA pairs in the same way:
Q: [Q]
A: [A].
Output:
Prompts for CoT data quality assessment. We employ a large language model, DeepSeek-Vi V3, to evaluate and filter the CoT pairs based on their factual accuracy and logical coherence. The relevant prompts are provided below:
23

---

Please help me evaluate the factual consistency and logical coherence between the original VQA pairs and the generated ones. The goal is to ensure that the generated answers align with the original facts and maintain logical reasoning.
Task:
Compare the original VQA pair with the generated one.
Check for factual consistency: Ensure that the generated answer does not contradict the original facts.
Check for logical coherence: Ensure that the generated answer and its reasoning (if provided) are logically sound and aligned with the original context. Finally give score between and 10, where 0 indicates poor match and 10 indicates
perfect match.
Input:
Original VQA Pair:
Q: [Original Question]
A: [Original Answer]
Generated VQA Pair:
Q: [Generated Question]
A: [Generated Answer]
The output should follow the format:
Factual Consistency: [Yes/No]
Logical Coherence: [Yes/No]
Score: [Score]. An example of output is
Factual Consistency: Yes,
Logical Coherence: Yes, Score: 10.
Now return your output:
B.2 Statisties of MSMU
We categorize the spatial tasks in MSMU into types, the distribution of which is illustrated in Figure 7 (left). The QA distribution of MSMU-Bench is also shown in Figure which provides detailed breakdown of these eight categories.
neigght
scale estimation
relative position
Mand higgher Rowee
scolle
(209)
distacce
absolute
absolute distance
(80)
height
cnunt
comporre
Sunds
scale compare
postton
EUNOO
existence
retal ob)
Figure The left shows the QA distribution of the MSMU dataset. The right shows the QA distribution of MSMU-Bench and the specific numbers of each category.
24

---

B.3 QA Templates
We provide templates that are used to construct the spatial tasks. Since MSMU consists eight spatial tasks, the templates can also be grouped into eight types, which are demonstrated below:
Scale Estimation
Scale Estimation
size_ sazze_tmmllte..e. htwpllateequettoons.
"What is the size of [A]?", "How big is [A
]?", " "Can you provide the size measurement of [A] ?", saze_template_anseer
"The size of [A] is [Length] x [Width] x Height]. ", "[A] is with the length of [Length], width of [Width], and height of [Height]."]
"What is the height [A]?", "How tall is [A]?", "Can you measure the height of [A] ?", biight_templat..e answers "The height of [A] is [Height].". "[A] is with the height of [Height]. "[A] measures [Height] in height. "]
questions - [ "What is the width of [A]?", "Determine the width of [A].", , "Can you measure the width of [A] ?", width_ peathhtemllate____ tenpeate_snevers anewers "The width of [A] is [Width]. ", " "[A] is with the width of [Width]. " "[A]
[Width] in width. "]
Counting
Counting
pount_templat..e quesstonss "How many [A]s are there in the image ?", "what's the total number of [A]s in the image?" count_template,
@eewwers anewers "There are [X] [A]s.", "There are [x] [A] the image.", " "[x]."]
Grounding
Grounding
"What object is located at ([x], ? "What can you find at ([x), [y))?, "What object does the position [y]) belong to?", positioon pooitiinn_.emppaee__qeeee
teeitioon_temppateeooo quesstinns
"It [A].", "That [A].", > [A]
"What the coordinate [A] ?" ] "([x) [y])." "IIt is located at ([x)
Desatioo2_ttsplat___amae 1) in the image.") "J
Existence
Existence
zero_tes cero_template_.. questtinssss { "What is the size of [A]?", "How big is [A]?
"Can you provide the size measurement of [A] ?" "What is the height of [A]?", "How tall is [A]?", "Where is [A]?", there in the image [A]s in the image?"]
How many [A]s
"shat's the total number
zero @ero_template ansvers=[PThers is no [A] in the image', Can not find [A
1. No [A] in the scene. ']
Absolute Distance
Absolute Distance
[ "What is the distance between [A] and
[B]?", "How far away is [A] from [B]?", "Can you provide the
distance measurement between [A] and [B]?",
distance_templer answers
"[A] and [B] are [X] apart.",
distance of [X] exists between [A] and [B] ", "[A] and [B] are [x]
apart from each other. "The distance is [x]
25

---

Relative Position
Relative Position
[ "Is [A] to the left/right of [B] from the viewer's perapective?", "Does [A] appears on the left/right side of [B]?", "Can you confirm if [A] is positioned to the left/right of [B]?", Left_teeplate__nseee aanwerrss ["Yes, [A] is the left/right [B] Indeed, [A] is positioned on the left/right side of [B]."]
pesser_template_ussseeee questions=="Froo the vieuer's perspective, what is closer, [A] [B] ?"]
aeser__template__asverrss [X] is more closer.")
stands Standd_templateeeee SSmppaate_qusttooss(MMMa.
stands higher/lover in the image, [A
[B] ?*]
stande_template_ anseers answers ["[X] stands higher/lower. .)
Scale Comparison
Scale Comparison
-tr.oooeeeeeeaaeeeuuiii-ooaaaaaa wwu.stiionnn("Bteeeeee [A] and [B], which one is taller/ taller_templle
1ower?", "Which one is taller/lowes [A] [B]? "]
terr_eeeppaee_assersssssse answers-("Thee height of [A] is [Height Al and [B] is taller_ taller__emplle
Height B], so [X] taller/lover.")
tallest_ beees__templat__qeeeteeee and [c]?"]
= ["What is tallest/lovest among [A], [B],
talleat_ setleetttemmlateeee answers ["The height of [A] is [Height Al height
[B] [Height B], and height of [C] [Height cl, so the tallest is [X]."]
[A] and [B], which is larger/ smaller?", "Which is larger/smaller pnggertemplate answers=(*Thh size of [A] is [Length A] [Width Al wtweeaaaeeasseeeeeeeeeeeeeeeeeee [Height A] and [B] is [Length B] x [Width B] x [Height BI, so [x] is
[A] [B]? "]
Reference Object Estimation
two objects
"The height of [A] is [Height Al, can you measure the height [B]?"] referi_templas @emplate_anavera answers ["Since the height of [A] is [Height A],
referi_template questions
think [B] is [Height B] in height."
teffer_template_que questions ["The width of [A] is [Width Al, can you measure the width of [B]?"] tempr_tteplatee..m answers ["Since the width of [A] is [Width A], i think the width of [B] is [Width "]"]
@efter__temllate__qqe measure the size of [B]?"]
["The height of [A] is [Height A], can you
1eerr_templlae answers = ["Since the height of [A] is [Height A], think the size of [B] is [Length B] [Width B] [Height B].
26

---

# three objects teeerr__tree_teteeee paapplaae_questtoss-""111 height of [A] is [Height A] what is the height [B] and [c] ?"]
beeee__chreeeeellateeee. a (innce the height [A] [Height Al
i think the height of [B] is [Height B] and the height of [C] is
[Height c]."]
Seferr_three__e wew.rreeeempltee.uuutitoooo
distance between [A] and [B]
[dis A2B], what is the distance between [B] and [c] ?"] eeferb_three_tee peo..teellatte.ssveeeeee
the distance between [A] and [B]
is [dis A2B], i think the distance between [B] and [c] is [dis
B2C]
C Evaluation Details
C.1 GPT-4 Evaluation for MSMU-Bench
We resort to LLMs (i.e. GPT-4-Turbo) to evaluate the results. For quantitative queries, GPT-4 extracts numerical values from the responses, and we calculate the success rate using predefined threshold. The prompt used to extract numerical values is shown as below:
You should help me evaluate the response given the question and the correct answer. You need to convert the measurement of the correct answer and response to meters. The conversion factors are as follows: 1 inch 0.0254 meters. 1 foot 0.3048 meters. centimeter (cm) = 0.01 meters. You should output two floats in meters, one for the answer, and one for the response. the answer or response contains more than one number for prediction, you should output the List that contains the numbers. The output should be in JSON format.
Example
Question: How tall is the long brown table opposite the crossed table?
Answer: The height of the long brown table opposite the crossed table is 1.02 m.
Response: It is 2.17 meters wide.
"answer in meters": 1.02, "response in meters"; 2.17
Example 2:
Question: what's the total number of chairs the image?
Answer: 2.
Response: There are 2 chairs.
"answer_in_ :eters":: 2, "response_iie in meters": 2
Example 3:
Question: What is the size of the dark pillow?
Answer: The dark pillow is with the size of 0.8 0.63 0.55 m
Response: It is 35.9 inches wide.
"answer in meters": [0.78,0.63,0.55], "response in meters": 0.91
Example 4:
Question: The height the bed is 0.81 m, what is the height of the table and nightstand?
Answer: Since the height of the bed is 0.81 1 m, i think the height of the table is 1.02 meters and the height of the nightstand 0.93 meters.
Response: Since the height of the bed is 0.81 m, think the height of the table is 1.36 meters and the height of the nightstand is 0.77 meters.
"answer_in_metenss [1.02,0.93). "response_in [nn_meters":(133//0
Your Turn:
Question: [Question]
Answer: [Answer]
Response: [Pred]
27

---

For qualitative questions, GPT-4 scores the model's answers between 0 and 1. The prompt is shown below:
You should help me to evaluate the response given the question and the correct answer. To mark a response, you should output a single integer between 0 and 1. 1 means that the response perfectly matches the answer. means that the response is completely different from the answer. The output should be in JSON format.
Example 1:
Question: Is the blue bed to the left of the curtain from the viewer's perspective?
Answer: Indeed, the bed is to the left of the curtain.
Response: Yes, the blue bed is positioned the left side of the curtain.
"your_marks
Example
Question: Between the wooden table and the black chair, which on is taller?
Answer: The wooden table taller. Response: The chair. "your_ _our__ark":: 0
Example 3:
Question: What is the tallest among the table, the chair, and the curtain?
Answer: The tallest is the curtain. Response: The curtain.
"your mark":
Your Turn:
Question: [Question]
Answer: [Answer]
Response: [Response]
C.2 Q-Spatial
Following the official setting, the evaluation threshold for Q-Spatial is 2.0. And the system prompt used is shown below:
You will be provided with a question and 2D image.
The question involves measuring the precise distance in 3D space through 2D image. You will answer the question by providing a numerical answer.
For example:
Question: What is the distance between the two chairs?
Answer: The minimum distance between the two speckled pattern stool chairs is 1 meter.
C.3 SpatiaIRGPT
Following the official setting, the evaluation threshold for SpatialRGPT's is 1.25. The prompts used to evaluate the qualitative and quantitative questions are the same as those used in MSMU-Bench.
D Detailed Results on SpatialRGPT-Bench
More detailed results are shown in Table 7 and Table 8. Our SD-VLM shows the best performance on quantitative tasks such as Height, Vertical Distance, Horizontal Distance, and Direct Distance and qualitative tasks such as Big/Small, Behind/Front, Left/Right, and Tall/Short.
28

---

SpatialRCPT- Bench
Model
Height
Vertical Distance
Distance
GPT-4o
0.76
9.07
18.0
149/0.64
Gemini-2
42.2 1.63
2.52/
1..3/0668
25.07 3.89
9.4/4.75
Qwen2.5-VL-77
VLLB228
31.8 1.38
2..8/
8.5
0.84
9.470.70
InlemnVL-3-788
23.7
L.aVA--1.5-BB
31.0/1.3 1.33
24.6/0.55
7.5 70.73
10.9/0.8
7.9/0.73
SpatialBot
28.4/2.16
20.5/0.72
6,670.70
8.0/
2.4/0.87
Spatia/RGPT
41.3/0.48
44.2 (0.51
2..5/0.58
13.0/0.64
2.55
Ours
42.2/0.55
26.2/0050
3.,8/0050
7..0/0.45
25.2/
Table 7: Results on quantitative tasks in SpatialRGPT-Bench.. We report the success rate and absolute relative error for SpatiaRGGPT-Bench.
SpatialRGPT-Ran
Model
Big/Small
Bchind/Froat
Left/Right
Tall/Short
Wide/Thin
Below/Above
GPT-4o
63.6
56.3
55.6
Gemini-2
54.1
44.6
59.1
67.7
55.6
64.2
Qwen2.5-V7 -V--225
60.2
54.3
67.0
66.7
56.7
63.3
IntemVL-33788
60.2
55.4
64.8
66.7
60.0
65.8
LLaVA-1.5-7B
16.3
40.2
23.9
32.3
17.8
27.5
SpatialBot
SpariaRRPP
59.2
56.5
39.8
65.6
55.6
70.0
Ours
68.2
58.9
67.5
Table 8: Performance of various baselines on the qualitative spatial tasks in SpatialRGPT-Benchh
E
Ablation Study on Estimated Depth
We have conducted a further investigation about the sources of depth maps. From the data presented in the Table 9, it is evident that the use of ground-truth depth maps during both training and inference phases leads to the best performance (57.71%) on the MSMU-Bench dataset. This suggests that the accuracy of depth information is crucial for the model's ability process and interpret spatial data effectively. If the ground-trute depth is not provided, the overall success rate of the model with estimated depth maps still competitive. It is noteworthy that significant disadvantage can be observed when the model is not equipped with any depth information, revealing the importance of incorporating depth priors into the VLM framework.
Ablation Study on Normalization
To bridge the gap between different sources of depth maps, we conduct normalization in the depth map before depth positional encoding, which can be formulated as:
depth depthmin depthmaz - depthmon
depthnorm:
az
(11)
where represents the normalization coefficient, which restricts the maximum value of the depth map.
As shown in Table 10, the highest success rate in MSMU-Bench is achieved when a is 100.
G Limitations
MSMU concentrates on indoor settings, featuring objects typical of domestic environments, reflecting our source datasets' composition. It narrows the model's applicability to social or dynamic interaction contexts. However, our model still exhibits strong adaptability, as evidenced by its solid performance on benchmarks like SpatialRGPT-Bench which contains abundant outdoor scenes. In the future, we will explore larger base models and alternative architectures, such as Qwen- VL., further investigate proposed framework.
29

---

Training Setting
Inference Setting
MSMU-Bench
w/ GT depth
w/ GT depth
57.71
w/ GT depth
w/ estimated depth
56.31
w/ estimated depth
depth
54.17
w/ estimated depth
w/ estimated depth
55.35
w/o any depth
w/o any depth
46.73
Table 9: Ablation study on the sources depth maps.
MSMU-Bench
50
49.50
100
56.31
200
53.67
500
52.98
Table 10: Ablation study on the normalization coefficient.
H
Broader Impact
Our model enhances its role as robust multi-modal generalist by demonstrating superior precision in spatial understanding. This capability is particularly valuable in embodied Al applications, where it aids robots in perceiving their surroundings with greater accuracy and performing precise manipulations. In addition, model, which is based on large language models, may encounter issues with hallucination, posing significant challenges when deploying the model real-world environments.
I More Result Comparisons on MSMU-Bench
Figure 8 and illustrate more result comparisons on MSMU-Bendh tasks. Our model shows consistent advantage spatial measuring and understanding. is worth noting that our model is able to reason about complex spatial tasks with the chain-of of-thought while other models fail to answer or return an incorrect answer.
Question (Reference Object Estimation):
The height of the right shelf is 1.65 meters, can you measure the height the desk with laptop water bottle?
Answer:
GT: Since the height of the right shelf is 1.65 meters, think the desk
with laptop and water bottle is 0.71 meters in height
GPT-4o: I'm unable directly measure distances heights in
possibly estimate the desk height by comparing to the 1.65-meter tall shelf if both are on the same plane and at a similar distance from the camera.
images. However,
LLaVA-1.5: N/A.
SpatialRGPT: The height of the desk with laptop and water bottle is 0.55 meters.
bottle, should think through step by step carefully. Firstly, the right shelf's height given 1.65 meters, which Observing the image, desk appears be slightly shorter than right shelf. By estimating the proportion, desk be around 50% of the shelf's height. Therefore, multiplying 1.65 meters 0.50 gives approximate height 0.825 meters for desk. the height of the desk with the laptop and bottle estimated to be 0.825 meters.
Ours:
To determine height desk with laptop
used referencc scale.
Figure 8: An example of the reference object estimation question in MSMU-Bench.
30

---

Question (Existence):
Can you provide the measurement of washing machines?
Answer:
GT: Can not
notfind find washing machines.
GGTT-4: I'm unable provide size measurements there are
washing machines visible the image.
LLLVV A-1.5: N/A.
SpatialRGPT: The size washing machines can vary depending on the model and brand.
Ours: There is washing machines in the image.
Question (Counting):
How many chairs there in the image??
Answer:
GT:
GPT-4o: There chairs image.
LLaVA-1.5; There two chairs in the image.
SpatialRGPT: There chuirs in the image.
Ours: There chairs.
Question (Grounding):
What object is located at (0.53,0.59y²
Answer:
GT: shower curtain.
GPT-4o: object located (0.53, 0.59) showerhead.
LLaVA-115: The shower curtain is located at (0.53,0.59).
SpatialRGPT: object located at (0.53,0.59) is shower.
Ours: shower curicain.
Question (Seale Estimation):
What the height of the office chair?
Answer:
GT: The office chuir meusures 1. meters in height.
GPT-4o: The height of the office chair is 0.8 meters.
LLaVA-1.5: The office chair of average height.
SpatialRGPT: The height the office chair inches.
Ours: height the office chair
Question (Relative Position):
Can you confirm if the coffee machine positioned the left of the dishwusher?
Answer:
GT: No, the coffee machine to the right of the dishwasher.
GPT-4o: Yes, the coffer machine positioned to the left of the
dishwasher.
LLaVA-1.55: NIA.
SpatiaIRGPT: Yes, the coffee machine is located to the left
dishwusher.
Ours: No, the coffee machine is right of the dishwasher.
Question (Scale Comparison):
Between the white toilet and the shower curtain, which one is larger?
Answer:
GT: The size of the white toilet is 0.69 x 0.64 m x 0.7 71 m and the
shower curtain is 41.29 em 30.74 cm 183.82 em, the white
toilet is larger.
GPT-4o: In terms of overall size, the shower curtain is larger than the toilet.
LLaVA-1.5;
SpatialRGPT: The shower curtain is larger than the toilet.
Ours: The of the white toilet 0.74 0.49 0.77 and the shower curtain 0.44 meters x 0.31 meters x 1.89 meters, so white toilet is larger.
Figure 9: More results ISMU-Bench.
31


Reviews:
[{"review_id": "eCjozbdYpy", "reviewer": "Reviewer_NnRS", "summary": "The authors propose SD-VLM, a framework to enhance the spatial perception abilities. It includes two parts. A dataset/benchmark MSMU with spatial annotations. and a depth-aware positional encoding method. Experiments show that this pipeline could successfully boost the spatial reasoning ability of VLMs.", "questions": "1. How robust is the DPE to domain shift and depth estimation noise?\nThe model uses estimated depth from Depth-Anything-V2. Have the authors evaluated DPE’s performance when using other depth estimators or when applying the model to out-of-domain scenarios (e.g., outdoor scenes or synthetic data)?\n2. Does the addition of DPE affect the model’s performance on non-spatial vision-language tasks?\nGiven that the model is fine-tuned for spatial understanding, it would be important to know whether performance degrades on general VQA benchmarks like GQA or TextVQA that include a broader range of reasoning skills and some other spatial reasoning benchmarks like WhatsUp.\n3. I noticed that DPE-sincos has similar performance with DPE-learnable on MSMU-bench. Has this conclusion been evaluated on wider benchmarks? Intuitively, DPE-sincos should be better generalization ability than DPE-learnable.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The model attains strong performance on MSMU and external benchmarks such as Q-Spatial++ and SpatialRGPT-Bench, showing clear gains over prior work.\n2. The depth-aware positional embedding is well-motivated and insightful; it is likely to be useful method to boost future VLM performance.\n3. I believe the released dataset and benchmark, MSMU, are valuable contributions to the community for advancing spatial reasoning in VLMs and will be beneficial for future research.\n\n**Weaknesses**\n\nAll training data are based on ScanNet, as are most of the evaluation settings (on MSMU benchmark in Table 1). Therefore, results on other spatial or general VQA benchmarks—such as WhatsUp, TextVQA, and GQA—would better demonstrate the broader applicability of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedbacks. We’re glad to hear that you found our work valuable and insightful. Further explanations and additional experimental results are added to address your concerns.\n\n**Q1: Robustness to depth estimation noise**\n\nWe take more ablation studies on the depth estimation noise on MSMU-Bench. To be specific, we inject zero&#8209;mean Gaussian noise with standard deviations $\\delta \\in$  {0.1, 0.3, 0.5, 0.7} into the normalized estimated depth maps. As δ increases, the downstream metrics naturally degrade, yet the overall performance remains competitive. The results demonstrate that our DPE is robust to depth perturbations. \n\n| No noise | $\\delta$ = 0.1 | $\\delta$ = 0.3 | $\\delta$ = 0.5 | $\\delta$ = 0.7 | w/o depth|\n| --- | --- | --- | --- | --- | --- |\n| 56.3 | 55.1 (-1.2) | 54.0 (-2.3) | 53.3 (-3.0) | 51.4 (-4.9) | 46.7 (-9.6)  |\n\n**Q2: Using other depth estimators**\n\nWe conduct an additional ablation study on the depth estimation backbone by replacing DepthAnything with another powerful depth estimator, UniDepth [1]. As summarized in the table below, our model maintains comparable and competitive performance across various spatial benchmarks. This robustness indicates that it has learned generalizable depth priors rather than overfitting to any specific depth-estimation architecture.\n\n|  | MSMU-Bench | Q-spatial++ | SpatialRGPT-Bench | Average |\n| --- | --- | --- | --- | --- |\n| SpatialRGPT | 29.0 | 43.5 | 28.7 | 33.7 |\n| Ours (UniDepth) | 56.2 | 54.7 | 32.0 | 47.6 |\n| Ours (DepthAnything) | 56.3 | 56.2 | 33.3 | 48.6 |\n\n**Q3: Performance in out-of-domain scenarios**\n\nIn terms of the **domain shift** of  **out&#8209;of&#8209;domain scenarios**,  we have conducted evaluation on SpatialRGPT&#8209;Bench which covers a variety of scene data, including nuScenes ( outdoor urban scenes ), KITTI ( outdoor urban scenes ), and Hypersim ( synthetic scenes ). We have achieved the best performance on this benchmark, as shown in Table 2 of the main paper, revealing the generalization ability of our proposed SD-VLM.\n\n\n**Q4: Results on other spatial or general VQA benchmarks**\n\nWe conduct experiments on other tasks, including Whatsup, TextVQA, and GQA. Related results are shown in the following table. Our variants perform similarly to the baselines on these benchmarks. It means that the spatial capabilities of VLMs can be enhanced without compromising general VQA performance.\n\n| Model |  Whatsup | TextVQA | GQA |\n| --- |  --- | --- | --- |\n| LLaVA-1.5-7B | 58.3 | 58.2 | 61.9 |\n| Ours  | 60.5 | 55.8 | 59.1 |\n\n**Q5: DPE-sincos vs DPE-learnable on wider benchmarks**\n\nThank you for your insightful suggestions. To further validate the generalization ability of depth&#8209;encoding strategies, we expanded our ablation to compare sincos and learnable variants.  The table shows that sincos consistently generalizes better. We attribute this to two factors: \n\n(1) learnable embeddings can overfit the specific domain of the training data, limiting robustness when test distributions shift;\n\n(2) sincos encoding is theoretically fixed and frequency&#8209;based, providing a stable, domain&#8209;agnostic prior that transfers reliably to unseen scenes.\n\n| Model | MSMU-Bench | Q-spatial++ | SpatialRGPT-Bench | Whatsup | TextVQA | GQA |\n| --- | --- | --- | --- | --- | --- | --- |\n| DPE-learnable | 56.2 | 50.7 | 25.7 | 56.4 | 51.4 | 56.4 |\n| DPE-sincos | 56.3 | 56.2 | 33.3 | 60.5 | 55.8 | 59.1 |\n\n**References**\n\n[1] Piccinelli et al., UniDepth: Universal Monocular Metric Depth Estimation. CVPR 2024."}, {"review_id": "Jvw7xSx7Zc", "reviewer": "Reviewer_1Nia", "summary": "This paper studies the 3D spatial relationship problem. The author proposes a new dataset called Massive Spatial Measuring and Understanding (MSMU) with precise spatial annotations. The MSMU datasets cover massive quantitative spatial tasks with 700K QA pairs and 2.5M physical numerical annotations. The author benchmarked tens of state-of-the-art multi-modality large language models including GPT model family, Gemini, Qwen, LLaVA, SpatialRGPT, etc. The author also identifies the key issue of the existing vision-language model for 3D spatial understanding and proposes a simple depth positional encoding method to strengthen the vision-language model's spatial awareness. Results show that the proposed method, i.e., SD-VLM, significantly outperforms existing SOTA with a large margin in both the new dataset and existing spatial understanding benchmarks.", "questions": "All the questions are fully discussed in the Strengths and Weaknesses section. Please refer to those sections for all the details.", "limitations": "Yes. The author has discussed the limitations of their work in their supplementary materials.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. Overall, the paper is very well-written and well-motivated. The introduction precisely summarizes the existing practices and challenges of 3D reasoning with vision-language models, and the shortcomings of the existing datasets proposed in SpatialVLM and SpatialRGPT. This provides a strong motivation for proposing a new dataset to push the boundary of the 3D spatial understanding capability of the multi-modal large language model (MLLM), as it is a critical capability for MLLM.\n\n2. The proposed MSMU dataset is extensive and comprehensive. It contains 700K QA pairs with 2.5M numerical annotations, generated from real 3D scenes. The author provides a clear description of the data generation pipeline in Figure 2 and the comparison of the proposed dataset with existing datasets in Figure 3, making it clear to evaluate the significance of the proposed dataset. The MSMU has a large task comprehensiveness compared to existing datasets, which is beneficial to evaluate the MLLM performance on 3D spatial understanding under different aspects. Overall, the reviewer believes that such a dataset will have a large contribution to the community.\n\n3. The idea of depth positional encoding is simple yet very effective. The reviewer appreciates that the author provides a clear comparison of the existing practice of integrating depth information into the MLLM, as shown in Figure 4. This will help the reviewers understand the high-level idea of how the depth information can be integrated into MLLM. The experiments in Section 6 further show the effectiveness of the depth positional encoding idea, where it outperforms tens of state-of-the-art MLLMs in different benchmarks.\n\n4. The experiment section is very well-designed. The author compared tens of existing SOTA MLLMs under different 3D spatial measuring and understanding tasks, which provide many insights and takeaways for the community to further improve the VLMs in the future. Moreover, the results on other existing spatial benchmarks further verify the effectiveness of the proposed SD-VLM as it consistently outperforms SOTA with a large margin.\n\n&nbsp;\n\n**Weaknesses**:\n\n**Technical novelty of the depth positional encoding**: As the positional encoding is well-known in the literature and commonly used in practice, the technical novelty is limited. It would be beneficial if the author could provide more insights and analysis to explain why this design is fundamentally superior to other designs, either theoretically or empirically. This can help further improve the contribution of the present submission.\n\n**Open-source of the dataset and code**: One concern is that the author does not indicate whether the dataset and the code will be publicly available in the future. This will influence the significance of the submission.\n\n&nbsp;\n\nReference:", "was_revised": true, "final_justification": "The rebuttal has addressed my concern of the novelty of the depth positional encoding. Overall this is a good paper and I maintain my recommendation for Acceptance", "rebuttal": "We sincerely appreciate your insightful feedback. We’re pleased to hear that you found our work well&#8209;motivated and Depth Positional Encoding (DPE) simple yet very effective. We provide more analysis to explain the superiority of DPE.\n\n**Q1: The superiority of DPE**\n\nTheoretically, depth positional embeddings are injected into the query–key score of each attention head, so every image token pair (i,j) receives a distance-aware bias. Specifically, self-attention computes relevance via the dot-product of query ($\\mathbf Q$) and key ($\\mathbf K$). If depth position encodings are fused additively: \n\n$\\mathbf Q=(\\mathbf E^{image}+\\mathbf E^{depth})\\ \\cdot \\mathbf W\\_{Q},\\qquad \\mathbf K=(\\mathbf E^{image}+\\mathbf E^{depth})\\ \\cdot \\mathbf W\\_{K},$\n\nthe dot&#8209;product $QK^T$ naturally decomposes into **image&#8209;image**, **image&#8209;depth**, and **depth&#8209;depth** interaction terms. The model can directly learn the relative importance on depth of each image token.\n\nIn contrast, existing depth integration methods can all be categorized as token concatenation. Putting “depth” as a text prompt or a side token only changes the context embeddings in the sequence. This introduces an unavoidable burden that \"depth\" tokens have to attend to all other \"image\" or \"text\" tokens, disturbing the relation between an image patch with its corresponding depth information. Therefore, the problem of **Redundancy of Concatenation** is inevitable. \n\nEmpirically, more comparative results in SpatialRGPT&#8209;Bench and Q&#8209;Spatial++ among different depth integration methods are shown below. Our DPE demonstrates a consistent advantage over other methods.\n\n| Model | MSMU-Bench | Q-Spatial++ | SpatialRGPT-Bench |\n| --- | --- | --- | --- |\n| depth as image | 22.6 | 46.2 | 25.5 |\n| depth as prompt | 48.8 | 54.4 | 30.7 |\n| depth as token | 35.7 | 40.9 | 23.6 |\n| DPE | 56.2 | 56.2 | 33.3 |\n\n**Q2: Open-source dataset and code**\n\nAs stated in the Abstract, both the dataset and the source code will be released publicly in the future."}, {"review_id": "vhEk5c3DQZ", "reviewer": "Reviewer_vpgs", "summary": "The authors propose to mitigate the challenges of VLM’s with spatial understanding, by developing the dataset, with precise spatial annotations with 2.5 mil numerical annotations, generated from real 3D scenes. They analyze different ways to incorporate depth information, fine-tune LlaVa-7.5B model and carry out comprehensive experiments improving the performance on the main spatial benchmarks.", "questions": "It would be help to make more quantitative statements comparing the methods with previous models that try to generate datasets for spatial information and also comment on possible shortcomings of their approach. What are commonly encountered errors of 3D scene graph construction ?  (in comparison with methods that use detectors and segmentation)  \n\nMore precise definition of the tasks evaluated in Table 1, what are the templates of VQA's ? \n\nThe improvement in the grounding performance is quite significant. What is the insight behind this ?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n- new spatial quantitative annotation and  new large scale dataset with 2.5 mil numerical annotations, generated from real 3D scenes\n- examination how to incorporate the depth information and ablation on different choices\n- improved performance on the spatial benchmarks and other tasks\n\nWeaknesses: \n\nBetter positioning of the architecture wrt other models (e.g. SpatialVLM's that have slightly different architectures). \nBetter discussion of the insights that go beyond improvement of performance and depth representation,  as \nnot surprising that fine-tuning with higher quality dataset yields better results.", "was_revised": true, "final_justification": "After reading the rebuttal, other reviews and discussion with the authors, I have upgraded my rating to borderline accept.", "rebuttal": "We thank you for your constructive reviews. We give more explanations on our insights and comparisons with previous methods to address your concerns as follows.\n\n**Q1: Insights of facilitating the spatial capability of VLMs**\n\nOur work provides novel insights into the spatial ability of VLMs from the viewpoint of 3D reconstruction. We assume that the spatial reasoning capability of VLMs is based on its understanding of the 3D spatial structure of an image. As discussed in Section 3 of the main paper, we review the procedure recovering 3D structure from an image. The precise physical measurements and depth are both necessary to reconstruct 3D geometry. \n\nWe first construct the dataset with precise physical numerical annotations. However, merely having high&#8209;quality data is not enough, as z&#8209;axis depth information is essential for constructing a comprehensive 3D scene. Then, we introduce Depth Positional Encoding (DPE) to incorporate depth information into image features. This information allows the model to implicitly learn 3D spatial concepts.  As shown in the following table, our model without depth, merely trained on MSMU dataset, falls behind our model equipped with DPE, even behind GPT-4o on Q-Spatial++.\n\n| Model | MSMU-Bench | Q-Spatial++ | \n| --- | --- | --- | \n| GPT-4o | 32.28 | 52.0 | \n| Ours w/o depth | 46.7 | 51.1 | \n| Ours w/ depth | 56.3 | 56.2 | \n\nFurthermore, we explore different ways of integrating depth and finally propose DPE, which can fully unleash the potential of numerical spatial data. Detailed analysis and insights can be referred to **Q1** in response to Reviewer 1Nia.\n\nTo further explain the advantage of DPE, we provide more empirical results on different datasets in the following table. Results on three benchmarks show that DPE is a more effective method to unleash the potential of 3D spatial awareness.\n\n| Model | MSMU-Bench | Q-Spatial++ | SpatialRGPT-Bench |\n| --- | --- | --- | --- |\n| Ours w/o depth | 46.7 | 51.1 | 30.1 |\n| depth as image | 22.6 | 46.2 | 25.5 |\n| depth as prompt | 48.8 | 54.4 | 30.7 |\n| depth as token | 35.7 | 40.9 | 23.6 |\n| DPE | 56.3 | 56.2 | 33.3 |\n\n**Q2: Better positioning of the architecture wrt other models**\n\nSpatialVLM inherits the structure of PaLM-E [1] without architecture modification or depth integration. SpatialRGPT inherits the VILA [2]  with independent image/depth linear connectors. Besides, SpatialRGPT uses Region&#8209;feature Extractor (from RegionGPT [3]) to extract features in given bboxes. SpatialBot directly inherits traditional VLM architecture and trains the model to recognize depth values. \n\nOur model uses LLaVA as our backbone. We further integrate depth via a simple yet effective method, DPE, which is also mentioned by reviewer 1Nia. Figure 4 of the main paper illustrates the differences between depth integration methods. DPE is free of introducing extra modules of VLMs like SpatialRGPT or employing a specialized depth api like SpatialBot, and gets better spatial performances at a small cost.\n\n\n**Q3: More comparisons with previous spatial information generating methods**\n\nCompared with previous spatial information generating methods, our data source (ScanNet, ScanNet++) is from reconstructed 3D meshes with physical sizes and manually annotated semantic labels. In ScanNet++, each scene is captured with a high&#8209;end laser scanner at sub&#8209;millimeter resolution, with an average distance of 0.9mm between points in a scan.  Therefore, the numerical measurements to construct the 3D scene graph in our dataset are from high-fidelity 3D geometry. \n\nPrevious detector/segmentation-based pipelines may encounter errors in two aspects:\n\n(1) **Recognition errors caused by segmentation/detection models**\nSpatialRGPT uses GroundingDINO [4] to localize objects. The zero&#8209;shot performance of GroundingDINO is only 46.7 AP. SpatialVLM uses ShapeMask[5] whose instance segmentation accuracy is only 40.0 AP. Although these two detectors are currently superior deep learning models, noisy masks will lead to inaccurate object shapes (occluded or including background). Furthermore, the object sizes (length/width/height/) will be noisy.\n\n(2) **Noise from camera calibration and depth estimation models**\nCompared to the millimeter&#8209;level errors of laser scanner, current powerful monocular depth estimation models struggle to achieve centimeter&#8209;level accuracy. SpatialVLM uses ZeoDepth [6] to get the depth map. It's zero&#8209;shot transfer error is at least 5%. Camera calibration models also introduce noise. These noise will lead to inaccurate reconstruction from an image to 3D geometry.\n\nThe noise from various deep learning models could be accumulated and influence the precision of numerical annotations if they are combined to reconstruct the 3D scene graph.\n\n**Q4: Definitions of MSMU tasks**\n\nWe have given the task categories and definitions in Section 4.1 of the main paper and examples of different tasks in Figure 1 of the main paper. The template details used to generate samples were provided in the Section B.3 in the supplementary.\n\n**Q5: The insight behind grounding performance improvement**\n\nAs mentioned in **Q1**, our methods enable the VLM to implicitly understand the corresponding 3D scene of an image. VLMs can better understand the volume occupation and relative distribution of objects in the space. Hence, VLMs can distinguish an object and the background clearly in an image, so that the grounding performance is improved.\n\n\n\n**References**\n\n\\[1\\] Driess et al. Palm-e: An embodied multimodal language model. ICML, 2023.\n  \n\\[2\\] Lin et al. VILA: On Pre-training for Visual Language Models. CVPR 2024.\n\n\\[3\\] Guo et al. RegionGPT: Towards Region Understanding Vision Language Model. CVPR 2024.\n\n\\[4\\] Liu et al. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. ECCV 2024.\n\n\\[5\\] Kuo et al. ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors. ICCV 2019.\n\n\\[6\\] Bhat et al. ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth."}, {"review_id": "uWpFxUWJx4", "reviewer": "Reviewer_hh1L", "summary": "In this paper, the authors propose a new dataset with precise spatial annotations and introduce depth positional encoding method which can strengthen VLM's spatial awareness. Their experiments and ablations show their method not only achieve SOTA performance on their proposed dataset but also on other spatial understanding bechmarks.", "questions": "1. When using LLMs or VLMs, do the authors only prompt once or several times and pick the best answers?\n\n2. When filtering image and selecting objects, how can you define the objects are partially occluded? To be specific, to what extent, the objects are excluded from your dataset. And in semantic disambiguation, do authors try other VLM instead of Qwen2.5-VL to relabel the objects? How can the authors make the Qwen2.5-VL label the object correctly? \n\n3. Can authors provide statistics of your dataset, for example, for each QA pairs, how many object will be involved and their relative sizes.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The topic is interesting.\n2. The paper is well-structured and easy to follow.\n3. The experiments and ablations are well-designed and comprehensive.\n4. Datasets is also well-designed.\n\nWeaknesses:\n1. Line 107, $K$ is not defined, if i am not wrong, it should be camera intrinsics.\n2. Some details are missing, I will put these in Question Section.", "was_revised": true, "final_justification": "Most of my main concerns are resolved by the authors' explanations. I will keep my rating (borderline accept).", "rebuttal": "Thank you for your constructive reviews and finding our work interesting. We have added further explanations and statistics to address your concern regarding the LLM/VLM usages and our dataset.\n\n**Q1: The definition of the** $\\mathbf{K}$\n\nYes, $\\mathbf{K}$ is the camera intrinsics. Thanks for pointing this out. We will add it to the main paper in the revised version.\n\n**Q2: When using LLMs or VLMs, do the authors only prompt once or several times and pick the best answers?**\n\nWe utilize different LLMs or VLMs in some parts of data generation pipeline. The main purpose of using LLMs or VLMs is to enrich the text including the detailed object names and chain of thought. The numerical values are directly extracted from 3D scene data so that the accuracy of numerical annotaions can be guaranteed. The final samples are generated by pre-defined templates.\n\nFor generating MSMU&#8209;CoT samples, considering the large number of samples, we are unable to manually pick the outputs of large models. Instead, to ensure the data quality, \nwe resort to another LLM (DeepSeek V3) to verify the generated reasoning path. We call it \"LLM collaboration\".\n\nIn terms of GPT-4 evaluation, we prompt it once for each question to make a fair comparison.\n\n**Q3: The definition of occluded objects during filtering images**\n\nWe calculate the visible size of the mask that are projected to the 2D plane. To be specific, the projected area size of the object can be denoted as $V\\_o$and the area size that is visualized in the image is denoted as $V\\_{visible}$. We compute the ratio of $V\\_{visible} / V\\_o$. If the ratio is below the threshold,  the object will be excluded from the image pair.\n\n**Q4: How can the authors make the Qwen2.5-VL label the object correctly?**\n\nWe use Qwen2.5&#8209;VL since it is an open-sourced and superior VLM model at the time we constructed the dataset. To make Qwen2.5&#8209;VL label the object correctly, we provide not only the object label in the prompt, but also the cropped object image patch. The attributes (color, texture, position, etc.) of objects are different. The Qwen2.5&#8209;VL is capable to distinguish them and enrich the details of object descriptions. So we did not try other VLMs.\n\n**Q4: More statistics of MSMU dataset**\n\nThank you for your suggestions. We have already provided some statistics about our dataset in the Figure 2 of main paper, and Figure 1 of supplementary. \nHere are more statistics as follows. For each QA pair, there are about 1.8 objects. As shown in the table below, we can observe that, among all objects: the average width=0.51m with std=0.44m; average height=0.88m with std=0.63m; average distance=1.73m with std=1.18m. \n\n|       |  Mean  | Std |\n|---------------|-----------------|-------------------|\n| Width   | 0.51  | 0.44     | \n| Height   |  0.88  | 0.63    |\n| Distance   |   1.73 | 1.18    |"}]

Author rebuttal:

Meta-review: Summary: 
This paper introduces SD-VLM, a lightweight depth-encoded VLM trained on the new MSMU dataset for quantitative spatial understanding. It adds DPE to image tokens and reports clear gains on MSMU-Bench, Q-Spatial++, and SpatialRGPT-Bench.

Strengths:
Practical, plug-and-play design (depth→PE added to visual tokens) with consistent improvements across spatial benchmarks; MSMU is a sizable, useful resource. The authors' rebuttal adds robustness evidence (noise injection, swapping depth estimators) and cross-domain results supporting generalization.

Weaknesses:
Reviewers pointed out that the novelty is limited (essentially dataset + standard SFT with simple DPE), and some reviewers question broader generalization beyond ScanNet-style data; mixing with general VQA data shows small trade-offs.

Overall, this is a solid, deployment-friendly work, with clear empirical value. The ACs agree on acceptance.
Decision: Accept (poster)