SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models
Authors: An-Chieh Cheng
Hongxu Yin
Yang Fu
Qiushan Guo
Ruihan Yang
Jan Kautz
Xiaolong Wang
Sifei Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision-Language Models
Spatial Reasoning
Primary area: generative_models
TL;DR: A powerful region-level VLM adept at 3D spatial reasoning.

Abstract: Vision Language Models (VLMs) have demonstrated remarkable performance in 2D vision and language tasks. However, their ability to reason about spatial arrangements remains limited. In this work, we introduce Spatial Region GPT (SpatialRGPT) to enhance VLMs’ spatial perception and reasoning capabilities. SpatialRGPT advances VLMs’ spatial understanding through two key innovations: (i) a data curation pipeline that enables effective learning of regional representation from 3D scene graphs, and (ii) a flexible ``plugin'' module for integrating depth information into the visual encoder of existing VLMs. During inference, when provided with user-specified region proposals, SpatialRGPT can accurately perceive their relative directions and distances. Additionally, we propose SpatialRGBT-Bench, a benchmark with ground-truth 3D annotations encompassing indoor, outdoor, and simulated environments, for evaluating 3D spatial cognition in Vision-Language Models (VLMs). Our results demonstrate that SpatialRGPT significantly enhances performance in spatial reasoning tasks, both with and without local region prompts. The model also exhibits strong generalization capabilities, effectively reasoning about complex spatial relations and functioning as a region-aware dense reward annotator for robotic tasks. Code, dataset, and benchmark are released at https://www.anjiecheng.me/SpatialRGPT.

SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models
An-Chieh Cheng', Hongxu Yin', Yang Fu', Qiushan Guo Ruíhan Yang',
Jan Kautz?, Xiaolong Wang Sifei Liu² UC San Diego, 2NVIDIA
Relative Relation
Spatial Reasoning
Can you confirm has greater
Can you try estimate how many floors are Think step by step. The height of 204.54 feet, As- suming each floor is about 10 feet high, the total number would be 20.454, Since you can't have a fraction of a floor, the total number of floors would be approxi- mately 20.
width compared
In fact, might be narrower than Which of these four, tallest?
floors
is the tallest.
From the viewer's 9 perspective, is the position of closer than that of
I'm planning to build bridge from to is possible?
No. closer.
The distance between and is 14.82 meters, build bridge between them.
B
Metric Measurement
possible
If you are at where will you find
you are riding motorcycle with inches wide, do you think you can pass through the area between and
at I o'clock from
What is the height of !
1.38 meters tall.
The distance between and is 38.95 inches, yes, you pass through the between and since the motoreycle is narrower than the distance between them.
How far apart are and hori- zontally?
They are 15.33 meters apart horizontally.
Abstract
Vision Language Models (VLMs) have demonstrated remarkable performance in 2D vision and language tasks. However, their ability to reason about spa- tial arrangements remains limited. In this work, we introduce Spatial Region GPT (SpatialRGPT) to enhance VLMs' spatial perception and reasoning capa- bilities. SpatialRGPT advances VLMS* spatial understanding through two key innovations: (i) data curation pipeline that enables effective learning of regional representation from 3D scene graphs, and (ii) a flexible "plugin" module for integrating depth information into the visual encoder of existing VLMs. Dur- ing inference, when provided with user-specified region proposals, SpatialRGPT can accurately perceive their relative directions and distances. Additionally, we propose SpatialR BaanllRBB---eeree Bench, a benchmark with ground-trui 3D annotations en- compassing indoor, outdoor, and simulated environments, for evaluating 3D spatial cognition in VLMs. Our results demonstrate that SpatialRGPT significantly en- hances performance in spatial reasoning tasks, both with and without local region prompts. The model also exhibits strong generalization capabilities, effectively reasoning about complex spatial relations and functioning as region-aware dense reward annotator for robotic tasks. Code, dataset, and benchmark are released at https: /www an www.://njjech.............. iecheng...... me htt//ppaaaalaaee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Understanding spatial arrangements in both 2D [1, 2] and 3D [3] spaces is crucial for accurately interpreting complex visual environments. Despite the impressive advancements in Vision Language Models (VLMs) across a variety of tasks such as image classification [4], captioning [5], object detection [6], video understanding [7]. and document parsing [8], etc., these models still face significant challenges with spatial reasoning. This includes difficulties [9, 10, 11] in distinguishing simple spatial concepts like "left" and "right," "above" and "below," as well as more complex relationships such as "behind" and "in front," "inside" and "outside." and "near" and "far." The ability to comprehend and reason about these spatial relationships is fundamental not only for visual understanding, but also for enabling practical applications in fields like robotics [12, 13] and augmented reality [14], where precise spatial awareness is crucial for tasks such as navigation [15], manipulation [12], and interaction with real-world environments [16].
Recently, several works [11, 17, 18] has advanced VLMss spatial reasoning capabilities by introduc- ing a comprehensive data generation pipeline that enables large-scale training with spatially- visual question answering (VQA) tasks. This approach based on the hypothesis that the limited spatial reasoning capabiliitiss of current VLMs are due to a lack of 3D/2D spatial knowledge in their training data. However, two critical challenges remain. First, effective spatial reasoning requires VLMs to accurately parse regional information, particularly the regions of object instances, whereas most existing VLMS are primarily designed to understand the global context of an image. When an image contains numerous instances, it becomes challenging to prompt the model to reason about the spatial relations between specific instances. This is because most VLMs function as global image parsers and do not support specifying regions for which users want understand spatial relationships. Second, accurately perceiving spatial relations such as direction and distance cannot rely solely on RGB pixel data. Thus, the architecture needs to incorporate 3D inputs, such as depth information.
In this work, we propose SpatialRGPT. leveraging a data curation pipeline, along with a region and 3D-aware visual encoder architecture to improve the spatial reasoning capability of VLMs.
Our data pipeline automatically generates 3D, region-aware annotations from 2D images at scale by constructing 3D scene graph for each image, where nodes represent object instances and edges denote spatial relationships. This is achieved through three scalable components: (i) open-vocabulary detection and segmentation for instance extraction, (ii) metric depth estimation, and (iii) camera calibration for projecting objects into 3D space. These scene graphs are subsequently transformed into region-aware spatial QA tasks using both template-based and large language model (LLM)) approaches. This dual approach provides region- VLMs with the necessary spatial knowledge and advanced reasoning capabilities to interpret complex environments. We use the collected data to train SpatialRGPT. While SpatiaIRGPT is designed to support region prompts, it effectively avoids the ambiguity issues found in Spatial VLM. In Spatial multiple similar objects in an image can confuse caption labels. In contrast, our pipeline naturally handles these scenarios without requiring carefully crafted rules or extensive post-processing.
Similar to RGPT [19]. SpatiaIRGPT introduces a region representation module that allows region proposals to be included as additional inputs alongside the image. This approach enables the LLM to leverage both regional and global contexts, allowing the model to reason about relationships between local regions while maintaining an understanding of the overall scene. In addition, we propose novel architecture that features a flexible "plugin" module for integrating relative-depth information into the visual encoder of existing VLMs. This design allows pre-trained visual encoder to optionally learn additional depth representation while still functioning effectively when depth inputs are absent. Our experiments demonstrate that this design can substantially improve the spatial reasoning capabilitics compared to VLMs that only use RGB images as input. Furthermore, we highlight practical applications enabled by SpatialRGPT, such serving region-await dense reward annotator and a stand- complex spatial reasoner. Our work has four main contributions:
1. We present SpatialRGPT. framework that enhances region-level spatial reasoning in VLMs by enabling effective representation of regional information and acquisition of spatial knowledge. Our novel architecture also integrates depth information flexibly, significantly improving 3D perception and analysis.
2. To facilitate model training, we introduce scalable data pipeline that constructs region- aware spatial reasoning QAs from existing datasets. With the pipeline, we create the Open Spatial Dataset (OSD), encompassing 8.7M spatial concepts grounded in 5M unique regions.
2

---

3. To address the absence of a benchmark for evaluating spatial cognition in VLMs, we present SpatialRGPT-Bencee -eeehh, a comprehensive benchmark based on ground-truth 3D annotations that span indoor, outdoor, and simulated environments.
4. We demonstrate downstream applications of SpatialRGPT. Leveraging SpatialRGPT's region capabilities, we develop region-aware dense reward annotator for robotics. Additionally, we show that SpatialRGPT can function as a stand-alone complex spatial reasoner, as well as its capacity to perform multi-hop reasoning.
2 Related work
Spatial Reasoning via Large Language Models. Recently, there has been significant push to obtain spatial reasoning capabilities using LLMs. Initiatives [20. 21] have focused on reconstructing scenes from multi-view images, such point clouds neural fields, and enhancing these represen- tations with dense semantic features. The resulting 3D representation and dense features are then integrated into an LLM. However, multi- view images not always available, and constructing a scene explicitly with dense semantic features is resource-intensii between 3D representations and language often results in decreased performance. ConceptGraph [22] avoids directly incorporating 3D representations into L.MMs. Instead, it constructs a scene graph and integrates this with the L.L.. Yet, recent studies [10] indicate that L.LMs struggle utilize coordinate information effectively when presented in text, which can undermine their ability to understand and reason about spatial relationships. Our research is most aligned with SpatialVLM [17], which uses 2D VLMs to understand spatial relationships and metric distances. Unlike the above approaches, the spatial understanding is encoded implicitly. The VLM directly handles the spatial relationship problem without an explicit 3D representation or scene graph. However, Spatial V relies on language descriptions of objects as input, while LLMs can already resolve some spatial queries even without visual data [23]. The responses can be inferred directly from the questions or derived from the world knowledge embedded in L.MMs. This reliance on textual cues suggests that the training may not effectively teach VLMs to learn spatial reasoning from visual data. Additionally, Spatial VL.M lacks the capability to specify regions precisely. This is particularly problematic in real-world scenarios where describing ambiguous locations or objects in language can be challenging.
Additionally, the modal gap
Region-level Visual Language Models. KOSMOS-2 [24], Shikra [25]. MiniGPT-2 [26],
CogVLM [27], SPHINX [28], and LLaVA [29] have enabled MLLMs to achieve region- based image understanding. However, these methods provide region information in textual form, such as bounding box coordinates. This method heavily depends on the language decoder to understand the position. In contrast, VisionL.MM [30], GPT4Rol [31], [32], and Ferret [33, 34], along with
GLaMM [35], use spatial boxes with ROI-aligned features to map region-level features into the L.M word embedding space. However, bounding boxes include unwanted background features, leading to inaccurate alignment between region descriptions and text, which complicates spatial reasoning. Recently, RegionGPT [19] and Osprey [36] have introduced visual spatial-aware modules that can directly extract pixel-level features. These models support using input masks that can accommodate regions of any shape. Despite these advancements, none of these approaches specifically focus on enhancing spatial reasoning at the region level in VLMs. Our framework is based on RegionGPT's ability process pixel-level inputs, with the aim of deepening spatial reasoning within region VLMs.
3 Method
SpatialRGPT is a powerful multimodal language model adept at understanding both 2D and 3D spatial arrangements. It can process any region proposal, such as boxes or masks, and provide answers to spatial reasoning questions. While effective training dataset is the key to learn spatial-aware region representation, we introduce: (i) how to build 3D scene Graph from a single image, in Sec. 3.1, and (ii) how to facilitate visual representation learning from these scene graphs in Sec. 3.2. We propose novel SpTt visual encoder architecture that flexibly leveraging monocular depth information into existing 2D VLM, in Sec. 3.3, with training detail explained in Sec. 3.1.
3.1 3D Scene Graph from Single 2D Images
Our scene graph construction pipeline (Figurel) begins with filtering process to remove any unsuitable images FII. Using open- vocabulayy models, we identify and ground candidate objects, followed by lifting them into 3D space using metric depth estimation and camera calibration. We then process the point clouds (Appx. F.3) to construct the final 3D scene graph.
Open-Vocabulare Vocabulary Detection & Segmentation. Segmenting objects the initial stage of building scene graph. Our models must satisfy two criteria: (i) object descriptions, e.g., class labels, should
3

---

Image Collections
RGB Image
Reasoning QA
Template QA
LLM
Filtering
Voen-Vocabulary Detection & Seg.
Metric Depth Estimation
Camera Calibration
Open-Vocab. 3D Scene Graph
Point Cloud Processing
Region Masks
Metric Depth
Pitch, Roll, Intrinsics
Figure I: 3D scene graph construction via automatic data curation pipeline.
adhere open-world setting for better generalization; (ii) mask proposals need to be highly accurate, ensuring precise contour outlines. This precision is crucial, as even small deviations can lead to significant inaccuracies in the resulting 3D bounding boxes. To this end, we first employ an open-vocabulary image tagging model [37] to identify all the object classes present in the image. Next, we use GroundingDino [38], open- vocabubayy 2D detector to determine the corresponding object bounding boxes. Finally, we apply segmentation models [39] to refine these bounding boxes into precise masks. We do not use existing dataset annotations since they either fall short due to vocabulary limitations, or use polygon annotations [40] or compressed masks [41] for segmentation.
Metric Depth Estimation. Several studies have explored the recovery of metric depth from single image. The main challenge is to address the scale ambiguity, and one common approach [42, 43] to use relative depth along with metric heads fine-tuned specific metric datasets. However, these methods may tend to overfit the depth scale for particular datasets such as KITTI [44] or NYU [45], which makes them less robust for in-the-wild images. Recently, Metric3Dvv [46] takes focal length as input and is trained end-to-end to predict metric depth and surface normals. The model is trained jointly on diverse indoor and outdoor scenes, making it less prone overfitting to the depth distribution of specific datasets. We adopt Metric3Dv2 as our metric depth estimator and found that Metric3Dv2 together with WildCamera [47]'s s camera intrinsic, is robust for images taken in real-world settings. Additionally, thanks to the joint depth-normal optimization training in Metric3Dv2, the recovered geometry is improved particularly around object edges.
Camera Calibration. Camera calibration includes (i) intrinsic estimation to back back-project depth maps to 3D point clouds, and (ii) scene canonicalization to ensure that scene relations described in a shared space. To estimate the camera intrinsic, we use the WildCamera model [47], which estimates four DoF intrinsic parameters (focal point and focal length in two dimensions). This model excels in real-world scenarios due to its acale-awareness and ability to detect image cropping. To convert the camera coordinates of the point cloud into a canonicalized geodetic coordinate system for each scene, we leverage PerspectiveFields [48], which provides per-pixel up-vectors and latitude values that can be transformed into camera extrinsics, such as pitch and roll. Using these, we derive a rotation matrix to convert the point cloud from camera coordinates to geodetic coordinates. We note that while SpatialVLM [17] uses surface segmentation (e.g.. "floor," "tabletop")) to identify a horizontal plane and then uses the normal axis of this plane to align the point cloud to the horizontal plane, this approach limited by the presence of specific classes, such as floors or tables. Additionally, the plane segmentation may fail if there are not enough points for RANSAC.
Constructing 3D Scene Graph. The 3D scene graph collection of tuples where the nodes represent specific 3D object instances, and the edges represent the spatial relationships between the nodes. Each node defined by the object's class, width, and height in metric seale. To create the node, we start by using the instance mask to deproject the object points from the depth map. Then, we perform canonicalization and denoising, and build 3D axis-aligned bounding boxes for each object. With the 3D bounding box. calculate the width and height of the objects in

---

How wide is Region [1]?
Is Region [1] behind Region [4]7
Between Region [0] and Region [2], which one more height? Region [2]
The width of Region [1] is 7.73 feet.
No, it is in front of Region [4].
and two sculptures, in Region [0] and the other in Region you walk from sculpture the other, how far will you have walked?
You visitor
You helicupter pilot
flying the city and you
Region [1] and Region
Which one higher?
The tower in Region [1] higher than the skyseraper in Region (7].
You will have walked 4.85 meters.
Example data entries from our Open Spatial Dataset. The first row contains template-based QAs, and the second row shows LLM-based entries.
Figure 2:
real-world units. The edges represent the spatial relationships between the nodes within two types of relations: relative and metric. Relative relations contain 1 ftf right, above, bel ow, beh nd, front, wide, thin, tall, short, big, and small. Metric relations include di direct ion, direct distance, hori zontal distance, and vert ical distance between the two objects. We then traverse all the object nodes and use the point cloud centroids and bounding boxes calculate their spatial relationships.
3.2 Learning Spatial-aware VLMs from 3D Scene Graph
In this section, we discuss converting the constructed 3D scene graph into textual representations for VLM training. One simple approach is through template-based methods via predefined handcrafted instructions. However, this approach limits the diversity of instructions and hinder the model's reasoning capabilities. Thus, we employ additional complex QAs to enhance the model's reasoning ability. Our results in Figure 4 show that blending these two types of data lead to a generalized and complex spatial reasoning model.
Template-based Question Answering. These QAs serve the foundation for learning basic spatial knowledge. We extract information about node attributes such as width and height, as well relative and metric relations from the edge attributes. We create both qualitative and quantitative templates to generate questions and answers for each type of attribute, using entities in the form of Region [X]. This approach results in examples shown in the first row of Figure 2. We provide detailed templates for each attribute Appx. F.4.
LLM-based Complex Reasoning Question Answering. We employ Llama3-700 to generate complex spatial reasoning questions to enhance the model's spatial reasoning capabilities. One approach is to input the scene graph directly into the LLMs. However, LLMs struggle to utilize 3D coordinate information effectively [10], so we opt for an alternative approach. We first construct spatial descriptions in a language format. Similar to the template-based approach, we extract attributes spatial descriptions based on these attributes.
from the scene graph and then construct template-basse We combine the spatial descriptions and the region tags as inputs the LLM. The LLM is then tasked with creating a complex reasoning question and answer that is based on the description and matches the context. Examples of LLM- --geeeerteedd QAs are shown in the second row of Figure 2. Our LLM prompts for generating QAs are provided in Appx. F.5.
We use our automated annotation pipeline to annotate images from the OpenImages [49] dataset, which covers a wide range of subjects and of high resolution. The resulting Open Spatial Dataset (OSD) contains 1M unique images and 5M open-vocabulary regions, each associated with a bounding box and segmentation mask. Furthermore, the dataset includes 8M template-based QAs and 700K LLM-based QAs.
3.3 VLM Architecture
An overview of SpatialRGPT's VLM architecture is shown in Figure 3. SpatiaIRGPT consists of visual encoder (Appx. G. to encode vision features, region-feature extractor [19] to obtain region- level embeddings (Appx. G.2), linear connectors (Appx. G.3) to project multi-mode embeddings into
5

---

They around 45 centimeters apart.
Large Language Model
<rgb> <depth>
<rgh>
<depth>
Visual Backbone
RGB Connector
Depth Connector
Region Masks/Boxes
Masks/ Boxes
Region Feature Extractor
What is the distance between
and
?
Input RGB(D)
Figure 3: An architecture overview of Spatial RGPT.
denotes freczed/trainable parameters.
the word embedding space, and large language model using LLaMA2-7B for language processing. In this section, we will explain why and how we incorporate depth information into SpatialRGPT, as well as how SpatialRGPT handles tokenizations.
Plugin Module for Relative-depth Inputs. VLMs that learn solely from RGB pixels are ineffec- tive for 3D perception tasks. Direct learning from 3D data (e.g., point clouds), presents challenges due to issues with scale and diversity. To bridge this gap, we propose using relative depth, which can be obtained through off-the-shelf models [43], to provide additional 3D information alongside RGB as input to our network. Our goal is to elicit geometric reasoning capability through depth guidance. However, this goal is non-trivial. Most VLM's visual encoders are typically only trained with text and 2D images, and simply concatenating RGB and depth features may negatively impact performance. To address this, we introduce add-on module that seamlessly incorporates the depth information. We use the same image encoder to process the depth map and generate depth feature maps. Then, we employ additional hepth-to-language connector project the features into the language domain. The depth connector's weights are trained only on spatial-relate the 2D visual encoder leverage additional depth representation while still functioning when depth inputs are not presented, thus avoiding the need for a vast amount of training data.
QAs. This flexible design allows
Tokenization and Prompt Format. We generate multi-turn conversation data following [29, 19] for each image and make the image the initial input for the first instruction, providing contextual information. Specifically, we incorporate a prefix prompt: "<image>\(n". The <image> is a special token that acts as a placeholder, which would be replaced by the image-level embedding from the vision encoder. When specific mask regions are mentioned in the user input, we use special tokens <regi on> and <depth> as placeholders. Each region token will be substituted with the corresponding region RGB embedding and depth embedding. All image-level, region- level RGB/depth tokens and text tokens are interleaved and fed as the input to the LLM for an auto-regressive generation.
3.4 Training and Inference Paradigm
SpatialRGPT training includes three stages [50]: (i) Connector Feature Aligmment, (ii) Visual Language Pre-training, and (iii Visual Instruction-tuning. During the first stage, CC3M image- caption pairs are used to pretrain the RGB connector as [29, 51, 52]. In the second stage, the visual language corpus from MMC4 [53] and COYO [54], along with region understanding datasets from [19] and our OSD dataset, are used to pretrain the LLM and connectors (Figure 3). Finally, at stage three, we fine-tune all weights of the VLM on visual language instruction-following datasets, using combination of the instruction tuning dataset from [29], region-level instruction tuning data [19], and our OSD dataset. Detailed data blend of the visual instruction data is in Appx. H.1. For training region-level data and our OSD, we randomly sample from different modalities (c.g.. box, mask) for each sample to ensure the model is versatile to the input modality. At inference time, SpatialRGPT can take both boxes or masks as input. For the results shown the main paper, if the segmentation is available, we use the mask; if not, use the box provided and apply SAM to segment the corresponding mask.
Experiments
We evaluate the effectiveness of our proposed SpatiaIRGPT in three aspects: (1) spatial reasoning benchmarks (Section 4. 4. (2) standard vision-language benchmarks (Section 4.2), and (3) real-world applications (Section 4.3).

---

Below/ Above
LetV Right
Big/ Small
Tall/
Widel
Behind/ Front
Avg.
Short
[S544
64.16
42.85
42.85
61.60
61.60
49.09
57.83
GPT-4V [55]
63.34
46.67
64.15
60.71
68.26
45.45
58.14
LLaVA-v1.6-4B [56]
44.16
45.71
36.79
53.57
37.50
45.45
43.98
GPT-4V [55]4SoM [57]
75.00
55.23
42.45
54.46
49.03
47.27
54.33
LLaVA-v1.6-4BB [56]+SuM (57]
44.16
40.01
33,96
47.32
41.34
46.36
42.31
KOSMOS-2 [8]
28.33
15.23
4.71
26.78
12.50
12.72
17.04
RegionVILLA-7B
30.83
47.61
35.84
44.64
35.57
49.09
40.48
SpatiaRGG117B(rgg))
99.17
99.04
79.24
89.28
83.65
87.27
89.80
SpatiaRGPPT77B
99.17
99.04
80.19
91.96
87.50
91.81
91.78
SpaiilBBG@P--VILA------1
99.17
100.0
81.13
88.39
85.57
93.63
91.47
SpaialRGGPTVVILA-11.5
99.17
100.0
84.90
89.28
91.34
90.90
92.69
Direct Distance
Horizontal
Vertical Distance
Width
Height
Direction
Distance
GPT-4 [55]
1.29
11.5 2.08
33.0/
52.3/0.52
1.40
34.6/ 83.7"
GPT-4V [55]
25.4
33.0/
51.1 00.37
68.4 1.57
69.9"
LLaVA-v1.6-34B [56]
24.3 0.76
24.5 1.59
30.1
30.8 /0.40
42.8/ 1.66
33.6 78.2"
GPT-4V [55]+SoM [57]
25.7 1.02
22.1 / 2.36
3.69
45.8
62.47 1.08
54.2 55.5
LLaVA-v1.6-34B [56]+SoM [57]
12.8 1.15
20.4 1.79
1.93
9,0/0.91
7.5/3.11
12.8 33.3"
KOSMOS-2
4.10>110
4,91 /=10
/226
3,0/
1.8/
1.9/
RegionVILA-7B [19]
/
24.6/ 3.26
17.9
36.8/>100
49.6/ 1.61
35.5 79.8
SpatiaRGGPT-7B.rg//
3..1
0.20
53.8 0.27
$1.9
54.9/0.6
95.3 17.1"
SpatialRGP1-718
41.2/0.33
65.6 0.25
$1.9 0.27
0.36/0.31
0.69
95.3 15.4"
SpaiiARRGTT-VIL--L.--BB
44.6 0,30
63.1 /0.22
50.9 0.28
42.9 /0.33
63.2/0.60
93.5 10.4
SpatialRGGPT--VIAAA11.5.5
45.9 /.31
68.0 0.22
56.6/0.22
48.9 0.28
61.7/0.4
95.3 9..7
Table 1: SpatialRGPT-Bench results. O are Blind LLMs with Language Referral. O are VLMs with Language Referral. O are Region-aware VLMs. Numbers in the top table represent success rates (f), while the bottom table includes success rates (?) and absolute relative eπror (L).
vQAr2
GQA
SQA!
VQA
POPE
MME
MMB
VILA-LL5-3B
80.4
61.5
69.0
60.4
85.9
1442
63.4
SpatiaRGGPT-VIL--155888
81.1
62.3
71.0
61.7
85.5
1424
65.6
MMB-CN
SEED
SEED/
MMMUy
MMMUT
LLaVA D
MMVet
VILA-LL5-3B
52.7
60.9
67.9
33.3
30.8
75.9
35.4
SpatialRGPPEVVIL.LL....
53.6
61.8
69.0
33.0
31.3
71.5
38.2
Table 2: Comparison of SpatialRGPT and base model performance on general VLM benchmarks.
4.1 3D Spatial Reasoning Benchmarks
visual-language benchmarks that specifically focus on VLM's ability to understand 3D spatial concepts like metric distance or size differences between objects. Recently, SpatialVI.M created spatial reasoning VQA benchmark using human labelers to annotate spatial information on 2D images, but this benchmark not publicly available. To address this gap, we develop SpatialRGPT-Bench.
Currently, there
spatial reasoning VQA benchmark using data from both urban (nuScenes [58], KITTI (59]) and indoor (SUNRGBD [60], ARKitScenes [61]) environments, as well as simulated scenes (Hypersim [62]). These datasets cover various potential applications and include diverse object types, enhancing our benchmark's 's thoroughness. We use preprocessed 3D cuboids for each object from Omni3D [63], all positioned within a unified 3D camera coordinate system and categorized by object classes. With these 3D cuboid annotations, we developed conversational benchmark using our data generation pipeline. This benchmark comprises 657 qualitative and 749 quantitative VQA pairs, covering 88 distinct classes. All the samples come from the validation or test splits of the original datasets and are unseen by SpatiaIRGPT during the training phase. Please see Appx. E for statistics and examples of SpatialRGPTTBenchh

---

if you were the person at Region [I] consider Region [2], Region (3) and Region [4), which one standing at
Can you estimate the age of Region [1p Think step by step.
GPTT-V
your 100clock direction?
In Region [1], the individual appears to be a toddler, possibly around to 3 years old, judging from their size,
GPT+-V
the person Region [1, Region [3] is in the 10 alclock direction. This region to the upper left from their perspective, aligning with the general
posture, the style walking, which suggests early childhood.
Pegión
The height of Region [1] is 3.77 feet. Since average height
direction indicated by the 10 o'clock
position on clock.
3-year-old is around 3.5 feet, Region is likely 3-year-old.
The person in Region [1] person in Region at 10 clock.
the
How many people do you think can Region [1]P Explain why.
Can you estimate much time will for the person in Region [1] neach the loca-
Based on the spetial dimension of Region
[1]. what kind of vehicles would not fit in?
GPT4-V
Region [1] highlights a purple sota with vibrant pattern, capable of
tion where Region [2] now? Say if the
person is Usain Bolt.
GPT4-V
In the image you ve shown, Region [1] covers the garage area. To determine what type of vehicles might not fit in this garage based its spatial dimensions, we to estimate the size of the garago. From the image, the garage appears to typical residential garage. Standard single-car ga- rages usually have a width of about 10 to 12 feet and depth of about 20 to 24 feet.
GPT--V
Considering that the person is Usain Bolt, we would need some additional information
comfortably seating three people.
This estimation based
stan-
such as the distance between the two regions
dard dimensions sofas that usual-
and average speed Usain Bolt could
accommodate three adults when the size and arrangement resemble what is shown the image.
maintain that distance in running ting.
The distance between Region [1] and Region
The width of Region [1] is 1.2 meters. Since average human width around 0.5 meters, people can Region [T].
[2] is 10.84 meters. It Usain Bolt run speed of 10 meters per second, will take him 1.084 seconds to reach the location
The Region [1] is 2.7 meters in height, vehicles taller than 2.7 meters would not fit in.
where Region [2] is now.
Figure 4: SpatiaIRGPT is capable of complex spatial reasoning, addressing gaps that current leading vision language models, such as GPT-4V, struggle with.
We consider three categories of models as baselines:
Blind LLMs w/ Language Referral. The blind [IM model relies solely on text and generates answers using only the content of the question. To enhance this approach, we prepend the object class to each question. This method serves as baseline to gauge how much spatial reasoning can be derived from purely existing world knowledge. We choose GPT-4 to represent this baseline, as it is the most advanced model for encapsulating comprehensive world knowledge.
VLMs w/ Language Referral. The setup is similar to the blind LLMs but includes access to visual content, which could allow the model answer better than blind LLM. We employ current state-of-the-art VLMs, GPT-4V and LLLV 166-4B [56], as baselines for this category.
Region-aware VLMs. This category explores models with region-level capabilities similar to our method. The models do not receive any language captions or object class information related to the region of interest; they rely solely on their visual processing capabilities. We equip GPT-4V [55] and LLaVA-v1.6-8 166-34BBBBBBBB with Set of Marks (SoM) [57] to enable region-referrin capabilitics. Additionally, include KOSMOS-2 2 [24], a VLM capable of taking bounding box inputs to reference objects, and Region VILA (RegionGPT [19] with VILA [50] pre-training). Region VILA-7B also serves as an ablation baseline to method; it shares the same model architecture as our SpatialRGPT-7B variant but is trained without our specialized spatial VQA dataset.
We use GPT-4 to evaluate the response for each model; please see Appx. J for details. For qualitative QAs, GPT-4 scores the alignment between the model's response and the correct answer as 0 or 1. For quantitative QAs, GPT-4 standardizes numerical values across units into meters; we then calculate accuracy and error metrics. We present the results Table 1. The upper rows of the table show accuracy (correct incorrect or failed to answer) for qualitative QAs. The lower rows report
8

---

Acc. (%)
Model
mAP
Acc. (%)
Owen-VL-Max [67]
58,9
CLIP [64]
58.9
Gemini Pro [68]
50.0
RegionCLIP [65]
58.3
Claude 3 OPUS [69]
57.3
LLaVA-7B [29]
40.0
GPPe-V-preeviww [55]
Shikra-78 [25]
53.9
GPT-4V-Tirbo [55]
66.9
GPT4RoI-7B [31]
64.0
GPT-4 [55]
64.5
PVT1:78 [66]
64.5
InstructBLIP-13B [51]
50.0
ASM-78 [32]
69.3
Yi-VI-348 [70]
RegionGPT-7BB [19]
70.0
80.6
LLaVA-v1.5-13B-xuuee [71]
$4.0
SpatialR(GP1178
69.7
79.9
LL.VAAv1..6344B [56]
64.5
SpotaURGGPT-VILA------B
72.5
82.5
SpaiiaRKGPP.-VI.AA-15555
72.9
82.9
MiniCiPT-4-v--7
49.2
InstructBLIP-7B
50.8
Table 3: Region-level classification results. We
LLaVA-v1.5-75-xxueer [71]
50.8
follow the evaluation RegionCLIP [65] and Re-
CogVLM-7B [27]
50.8
gionGPT [19], report the results of object clas- sification with ground-truth box on COCO-2017 validation set.
I.L.VAvv..5-B8 [72]
51.6
I.LaVA-Interll. M2-78
52.4
SpatialRGPT-7B
82.3
SpatialRGPPVVII.......
87.9
Table 4: BLINK RelativeDepth results.
quantitative QAs, detailing their success rate (answers within +25% of the ground truth value) and the absolute relative error [43, 42]. We exclude answers that failed to produce numerical response from the relative error calculations. The results show that SpatialRGPT significantly outperforms baselines terms of success rate for qualitative QAs and maintains the lowest error rate for quantitative QAs. Interestingly, we found that blind LLMs and VLMs with language referrals achieved commendable success rates for quantitative QAs, especially for questions related to width and height. This suggests that LLMs can accurately answer specific spatial questions using their extensive world knowledge. Additionally, our SpatialRGPT-7B variant demonstrates improved performance over the SpatialRGPT- 7B rgb)) variant, especially in scenarios where relative depth information can be used resolve ambiguities, such as distinguishing between behind/front, wide/thin, and estimating distances.
4.2 Public Vision-language Benchmarks
General Benchmarks. In this section, we evaluate whether integrating spatial VQA data and depth information affects performance other VQA tasks. We compared our models with VILA-1 ILL--1.5-3B. which is trained on the same general VQA datasets. As shown in Table 2, our variants performed similarly to the baselines and slightly better on the VQA-v2 and MM datasets. These results align with findings from [17], indicating that VLMs generally underperform on spatial reasoning tasks but can improve with specific spatial VQA training without compromising general VQA performance.
Region & Spatial Benchmarks. We follow the evaluation protocol from RegionGPT [19] and report object classification results using ground-truft -ruth boxes on the COCO-2011 validation set. As shown in Table 3, SpatialRGPT outperforms the baselines, demonstrating its strong region cognition capabilities. We further evaluate SpatiaIRGPT on BLINK [9]'s Relative Depth Benchmark. This benchmark is particularly challenging it assesses point-level depths, while both the point-level region input and point-level questions were not specifically included in the training of SpatiaIRGPT. We use bounding boxes to mark the target points and evaluate the test set online with the EvalAI server. As shown in Table 4, SpatialRGPT significantly outperforms the state-of- oftthe-rrt achieving over 20% accuracy gain compared to GPT-4V-Turbo Our model demonstrated strong performance, highlighting its ability to generalize to new tasks without explicit training.
4.3 Real-world Applications
Complex Spatial Reasoning. In this application, we aim explore whether SpatialRGPT can function as a complex spatial reasoner on its own. Unlike the system mentioned in [17], which uses GPT-4 to handle reasoning tasks and employs VLM solely for answering basic spatial queries, SpatialRGPT directly integrates these capabilities. We provide examples in Figure 4, where we compare SpatialRGPTT responses those from GPT-4V using real-world samples. Our model demonstrates the ability to address complex spatial questions based on its own spatial knowledge. This suggests that SpatiaIRGPT has developed robust representation of spatial learning and that this knowledge has effectively generalized to enhance its intrinsic language reasoning abilities.

---

whatt the object in the basket to the right Region (o), and how
Owatt the object on the table to the right of Region and what dis-
lance to Region [M]?
PAcgion O2
Region G
The object in the basket the right of Region [o] is a peach, and is 10.42 inches away..
The object on the table the right Region [i] is ported plant, and it inches away from Region [0].
what the object on the table the right Region [o], and what is beight and width?
What the object to the left of Region [0], what is its height width?
Rcgion
objoct on the table to rightt Region [0] is coffee maker. is 19.1 inchos in height and 124 inclaes width.
The object to enndle, and it 10.49 inches height and 3,39 inches in width.
Region
Region
Figure 5: Examples of SpatialRGPT performing multi-hop reasoning.
Distance (inches)
SHORT Pape
3
Frame Index SpatialRGPT functions as region-aware reward annotator. The estimated distance decreased monotonically as the fingertip moves towards the target.
Figure 6:
Multi-hop Reasoning. In Figure 5, we show examples of Spat handling multi-hop rea- soning. In the upper left sample, the model first identifies what's to the right of Region [0] (a single apple), finds the basket there, determines what's inside the basket, and then provides spatial details about the object inside. Even though our training data doesn't it specifically include such multi-hop tasks, SpatialRGPT can still manage them effectively. This indicates that the model has developed a strong understanding of spatial relationships.
Region-aware Dense Reward Annotator. Recently, [17] has shown that VLMs can function as dense reward annotators for robotics tasks by specifying tasks in natural language and having the model annotate rewards for each frame in a trajectory. However, this approach can be constrained by the language's ambiguity, especially when multiple identical objects are present or when targeting small, specific region scene, which can be difficult to describe precisely with language alone. Given that GPT is equipped with region-aware capabilities, we can directly specify the regions of interest. To study this application, we conducted real robot experiment. Specifically, we defined two regions using bounding boxes (one for the fingertip and one for green cube) and tasked SpatialRGPT to annotate rewards using the distance between the two regions. The results, shown in Figure 6, indicate that the estimated distance between the fingertip and its target cube decreased monotonically as the fingertip moved towards its goal. Also, our depth variant performs slightly better than the RGB variant. This demonstrates Spa 's effectiveness region- aware dense reward annotator, offering a more precise and efficient alternative to language-only approaches. 5 Discussion
Conclusion. We introduce SpatiaIRGPT, novel framework designed to enhance the spatial rea- soning capabilities of Vision Language Models (VLMs). By integrating region representation module and flexible plugin for depth information, SpatialRGPT allows VLMMs to effectively perceive spatial arrangement at both local and global scopes. Our data curation pipeline f'aillitates the learning of 3D spatial knowledge from scene graphs, while SpatialRGPT- benchmark for evaluating spatial cognition across diverse environments. The results demonstrate significant improvements in spatial reasoning tasks while showcasing the model's 's ability to reason complex spatial relations and perform dense reward annotators for robotic applications.
provides a comprehensive
Limitations. One limitation of our work is the use of Axis- Aligned Bounding Boxes (AABBs), which can result in inaccuracies in label representation. A more accurate alternative is oriented bounding boxes (OBBs), but implementing them requires precise object pose estimation, which remains challenging due to the lack of open-world solutions. The most accurate approach would be human labeling [74], while this requires significant effort. We leave these for future work.
10

---

Acknowledgement.
This work was supported, in part, by the Qualcomm Innovation Fellowship.
References
[1] Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In CVPR, 2019. 2
[2] Long Chen, Hanwang Zhang, Jun Xiao, Xiangnan He, Shiliang Pu, and Shih-Fu Chang. Counterfactual critic multi-agent training for scene graph generation. In ICCV, 2019. 2
[3] Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In CVPR, 2020. 2
[4] Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In ICCV, 2023. 2
[5] Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen- Or.. Myvim: Personalizing vlms for user-specific queries, arXiv preprint arXiv:2403. 14599, 2024. 2
[6] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-vim: Open- vocabulary object detection upon frozen vision and language models. In ICLR, 2023. 2
[7] De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localizatio assistant. In ECCV, 2024.
[8] Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma. Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, et al. Kosmos-2.5: A multimodal literate model. arXiv preprint arXiv:2309. 1419, 2023. 2.7
[9] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint
2024. 2.9
[10] Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Sasha Sax, and Aravind Rajeswaran. Openeqa: Embodied question answering in the era of foundation models. In CVPR, 2024. 3, 8
[11) Jianrui Zhang, Mu Cai, Tengyang Xie, and Yong Jae Lee. Countercurate: Enhancing physical and semantic visio-linguistic compositional reasoning via counterfactual examples. In ACL, 2024.7 2
[12] Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-languaa models for robotic manipulation. In ICRA, 2024.
[13] Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07877 07877, 2024.
[14] Mikhail Konenkov, Artem Lykov, Daria Trinitatova, and Dzmitry Tsetserukou. Vr-gpt: Visual language model for intelligent virtual reality applications. arXiv preprint arXiv:2405. 2024.7 2
[15] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In ICRA, 2023. 2
[16] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. In ICMI., 2023.
[17] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas. and Fei Xia. Spatialvim: Endowing languaee models with spatial reasoning capabilities. In CVPR, 2024. 2, 3, 4, 9, 10. 19, 20, 23
[18] Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vasion-language model for spatial affordance prediction for robotics. In CoRL, 2024.

---

[19] Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision language model, In CVPR, 2024. 2, 3, 5, 6, 7, 9, 22
[20] Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan.
3d concept learning and reasoning from multi-view images. In CVPR, 2023. 3
[21]
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and
Chuang Gan. 3d-1Im: Injecting the 3d world into large language models. In NeurlPS, 2023. 3
[22] Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In ICRA, 2024.
3. 30
[23] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large rision-language models? arXiv preprint 2xXi::2403.20330 2024. 3
[24] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. In ICLR, 2024. 3,
[25] Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint
2023. 3, 9
[26] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint 10.0422310.09478, 2023. 3,9
[27] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXiv:231 1.30099, 2023. 3,9
[28] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights. tasks, and visual embeddings for multi-modal large language models. arXiv preprint a07555 2023.
[29] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. In NeurlPS, 2023. 3, 9,
[30] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. In NeurlPS, 2023. 3
[31] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region- of-oon----inn of-interest. arXiv preprint N.Xiv:2307.036000 2023. 3,
[32] Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. ICLR, 2024. 3,
[33] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang. Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In ICLR, 2024.
[34] Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu- Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint 2024. 3, 22
[35] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji. Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. CVPR, 2024.
[36] Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In CVPR, 2024. 3
12

---

[37] Youcai Zhang. Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo. Yaqian Li, Shilong Liu, et al. Recognize anything: strong image tagging model, arXiv preprini arXiv:2306.088: 035 14, 2023. 4
[38] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303..mm 05499, 2023. 4
[39] Lei Ke, Mingqiao Ye, Martin Danelljan, Yifan Liu, Yu-Win Wu-Wing Tai, Chi-Keung Tang, and Fisher
Yu. Segment anything in high quality. In NewrIPS, 2023. 4
[40] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Mícrosoft coco: Common objects in context. In ECCV, 2014. 4, 22
[41] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, 2023.
[42] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Mathhis Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. In CVPR, 2023. 4,
[43] Lihe Yang. Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024. 4, 6, 9
[44] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRR, 2013. 4
[45] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012. 4
[46] Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geomet- ric foundation model for zero-shot metric depth and surface normal estimation. arXiv preprint WwXiv:2404.55500 2024. 4
[47] Shengjie Zhu, Abhinav Kumar, Masa Hu, and Xiaoming Liu. Tame a wild camera: In-the-wild monocular camera calibration. In NeurIPS, 2023. 4
[48] Linyi Jin, Jianming Zhang, Yannick foof-Geoffrry, Oliver Wang, Kevin Matzen, Matthew Sticha, and David F. Fouhey. Perspective fields for single image camera calibration. In CVPR. 2023. 4
[49] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Tust-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJC,, 2020. 5, 19, 25
[50] Ji Lin, Hongxu Yin, Wei Ping. Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, 2024. 6, 8, 22, 23
[51] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurlPS, 2023. 6, 9
[52] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2 Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICMI., 2023. 6
[53] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. In NewrPPS, 2023. 6
[54] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset.
coyo- dayoodatsst, 2022.
[55] OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023. 8, 23
[56]
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal capabilities in the wild, 2024. 7, 8,
13

---

[57] Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint 1rX1422310.1141, 2023. 7,
[58] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
[59]
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012. 7
[60]
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A Agb-d scene understand- ing benchmark suite. In CVPR, 2015. 7
[61] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Yuri Feigin, Peter Fu, Thomas Gebauer, Daniel Kurz, Tal Dimry, Brandon Joffe, Arik Schwartz, and Elad Shulman. ARKitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data. In NeurIPS,
2021. 7
[62] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In ICCV, 2021.
[63] Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild. In CVPR, 2023. 7,
[64] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 9, 22
[65] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based language-image pretraining. In CVPR, 2022. 9
[66] Chi Chen, Ruoyu Qin, Fuwen Luo, Xiaoyue Mi, Peng Li, Maosong Sun, and Yang Liu, Position- enhanced visual instruction tuning for multimodal large language models. arXiv preprint arXiv::3008 13437, 2023. 9
[67] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint PrXii:2338.12966 2023. 9
[68] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- -aan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 /805, 2023.
[69]
Claude-3-family, hutps//www.anntroii.ccommmeee
claude-3-family. 9
[70] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangeheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv
preprint arXiv:2403. 04652, 2024. 9
[71] XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning Ilm.
com/ InternnM/ 2023. 9
[72] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024. 9
[73] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internim2 technical report. In arXiv preprint arXiv:2403. 172977 2024. 9
[74] Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models. arXiv preprint arXiv:2409.09788, 2024. 10, 24
[75] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion- 5b: An open scale dataset for training next generation image-text models. NewrIPS, 35,
2022. 19
14

---

[76] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint 2333::2303.13389 2023. 19
[77] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 22
[78] Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich. Discriminability objective for training descriptive captions. In CVPR, 2018. 22
[79] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016. 22
[80] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In ECCV, 2020. 22
[81] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 22
[82] Alane Suhr. Mike Lewis, James Yeh, and Yoav Artzi, A corpus of natural language for visual reasoning. In NAACI, 2017. 22
[83] Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehension on document images. In AAAI, 2021. 22
[84] Desmond Elliott, Stella Frank, Khalil Sima' an, and Lucia Specia. Multi30k: Multilingual english-germad -erhhnn image descriptions. In ACL, 2016. 22
[85] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, 2019.
[86] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. In CVPR, 2021. 22
[87] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 22
[88] Feng Liu, Tao Xiang, Timothy M Hospedales, Wankou Yang, and Changyin Sun. ivqa: Inverse visual question answering. In CVPR, 2018. 22
[89] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa:
Visual question answering by reading text in images. In ICDAR, 2019. 22
[90] Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, 2019.
22
[91] Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, Jose G. Moreno, and Jesús Lovón Melgarejo. Viquae, a dataset for knowledge-based visual question answering about named entities. In ACM SIGIR, 2022. 22
[92] Yash Goyal, Tejas Khot, Douglas Cummers-Stayy Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. 22
[93] Abhishek Das. Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yaday, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In CVPR, 2017. 22
[94] Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao. Yujic Zhou, Tong Wu, Bin Wang, Conghui Hc, and Dahua Lin. V3det: Vast vocabulary visual detection dataset. In ICCV, 2023. 22
[95] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance
segmentation. In CVPR, 2019. 22
[96] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017. 22
[97] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014. 22
[98] Unsplash. https//unsplash..om/ 25
15

---

Appendix Table of Contents
Ablation Study on Augmented SpatialRGPT-Bench
17
B Ablation Study on Metric-Scale Width and Height Data
17
Ablation Study on Bounding Box Types
17
D Ablation Study on Different Input Modalities
18
Statistics and Samples of SpatialRGPT-Bench
18
F Implementation Details for Data Pipeline
18
Implementation Details for SpatiaIRGPT Architecture
22
H Implementation Details for Training SpatialRGPT
22
Experimental Setting and Details
23
J Benchmark Evaluation Details
23
More Discussion on Limitations
24
L Broader Impacts
25
M Licenses
25
16

---

A Ablation Study on Augmented SpatialRGPT-Bench
We conduct additional experiments by augmenting and rephrasing both questions and answers in SpatialRGPT-Bence using GPT-4. The results are shown in Table 5. The results show that SpatiaIRGPT consistently outperforms the baseline models, even when the questions and answers are different from the training data.
Below/
Left/ Right
Big/
Tall/
Wide/
Behind/
Qualitative
Above
Thin
Average
GPT-4V-Turbo
66.7
47.6
66.0
64.2
71.1
47.2
60.5
SpaiiIRGPPT-78
95.8
99.0
77.4
92.9
82.7
90.9
90.0
Direct Distance
Horizontal Distance
Vertical Distance
Width
Height
Direction
GPT-4V-Turbo SputialRGPT-7B
30.4 0.87
26.2 2.66
33.9 0.51
48.8 0.35
69.1 /
40.1 / 70.0"
43.2 / 0.32
63,9 0..27
52.8 / 0.26
51.1 0.31
54.1 / 1.02
95.3 / 15.3"
Table Augmented SpatialRGPT- results. Numbers represent success rates (f) and absolute relative error (L).
B Ablation Study on Metric-Scale Width and Height Data
We conduct an ablation study to if adding width and height data affects other types of questions. As shown in Table 6, adding this data slightly improved the accuracy for questions about size (like big/small, tall/short, wide/thin) but slightly worsened the accuracy for questions about the distance between objects (horizontal and vertical). This suggests that information about object size helps with size-related questions but might make distance measurements less clear.
Below / Above
Left / Right
Big / Small
Tall / Short
Wide/ / Thin
Behind Front
Arg-
width & height 99.1
99.0
75.8
90.8
82.8
92.1
90.5
width & height
99.1
+0
99.0
+0
80.1
+4.3
91.9 +1.1
87.5
+4.7
91.8 -0.3
90.5
+1.2
Direct Distance
Horizontal Distance
Vertical Distance
Width
Height
Direction
widhh & height /41.2
69.3
54.8
22.8
21.2
95.1
width & height 41.2 40
68.6
-3.7
51.9
-2.9
49.6
+66.8
57.9 +36.
95.3 40.2
Table 6: Ablation study on the impact of width and height data on the performance of other categories. Numbers represent success rates (t).
C Ablation Study on Bounding Box Types
We conduct an ablation study to examine the effect of using axis-aligned bounding boxes (AABB) versus PCA-based oriented bounding boxes (OBB). For this study, we human-labeled OBBs from the Omni3D test set as the ground truth. We then compare the mean-square error of the width and height measurements for AABBs and PCA-based OBBs labeled by our 3D scene graph pipeline. The results are shown in Table 7. PCA-based OBB often lacks accuracy due to the incomplete and noisy nature of point clouds captured from a single view.
BBox Type
Width (L)
Height ())
Oriented BBox
17.09
4.83
Axis-aligned BBox
8.27
2.35
Table 7: Ablation study on axis axis-aligned vs. oriented bounding boxes. Numbers indicate MSE
comparing to Omni3D ground truth.
17

---

D Ablation Study on Different Input Modalities
As mentioned in Section 3.4, SpatialRGPT can take both boxes and masks as input during the inference phase. In this study, we aimed to test the impact of box and mask inputs on our SpatialRGPT--enn Bench.. We presented the results in Table 8, where we observed slight drop in performance when using boxes, but general, the performance was very close. This suggests that the random modality strategy used during training is effective.
Below/
Left/ Right
Big/ Small
Tall/ Short
Wide/ Thin
Behind/ Front
Avg.
Above
SpatialRGPT-7B-Mass
99.17
99.04
80.19
91.96
87.50
91.81 92.72
91.78
SpatialRGPT-7B-Bo.
99.17
98.09
83.01
91.96
82.69
91.47
Direct Distance
Horizontal
Vertical Distance
Width
Height
Direction
Distance
SpatialRGPT-7B-Mass
41.2/ 0.33
65.6/ 0.25
51.9/0.2
49.6 / 0.31
57.9 / 0.61
95.3 / 15.4
SpatiaRRGPT-7--Box
39.2 0.35
63.1 0.25
56.6 0.27
48.8 0.36
/ 1.06
94.3 / 10.2
Table 8: Ablation study effect of different input modalities Spatial RGPT. Numbers in the top table represent success rates (f), while the bottom table includes success rates (1) and absolute relative error (L).
E
Statistics and Samples of SpatialRGPT-Bench
Figure 7 presents key statistics from our SpatialRGPT-Bench including counts for QA categories, data sources, and objects. We categorize the QA data into 12 distinct types, evenly divided between relative relationships and metric measurements. Notably, some datasets, such as SUNRGBD, emphasize close- object scenarios. To reduce bias, we source our data from a diverse range of datasets following [63]. We also show six samples from our SpatialRGPT-Bendd in Figure 8.
ARKUSSones
110
105
KITTI
106
Blgg
Preee
Rudke
Talfy
112
Senat
Wide!
104
Exnction
Below/
Above
Direct
Distance
Disurr
Vertical
148
walt
Harismna
106
Height
133
133
Figure SpatialRGPT-Bench statistics. Left: Category count and source count. Right: Object count.
Implementation Details for Data Pipeline
In this section, we aim to provide a detailed implementation of our data annotation pipeline and intermediate results obtained through each component.
18

---

How from horizontally?
Could you tell the vertical size
How tall
in berms of height?
apart borizontally.
4.65 height.
Does
have a larger size compared
Does
have lesser width compared to
the pusition of less distant than
Conneet
larger than
No.
thinner than
No.
Figure 8: Samples in SpatialRGPT-Benn
F.1 Filtering.
Recent VLMs often benefit from the broad capabilities gained through training with large-scale 2D image datasets [75, 49]. However, many images in these datasets unsuitable for developing spatial reasoning QA. For instance, some images may be computer screenshots, paintings, collages, CLIP-based open-vocabulary
or simply piece of text. Similar to SpatialVLM [17],
classification model [76] to identify and exclude these unsuitable images. We follow the labeling used in SpatialVL.M but have made few adaptations better suit the data distribution of the Openlmage [49] dataset. We show the labels we use in Listing With this process, we filtered out 700K samples from the 1.7M OpenImage samples.
Listing CLIP labels used during filtering.
positive 1abels
DSLR photo an indoor scene",
"a DSLR of an outdoor scene",
"an iphone photo of an indoor scene",
"an iphone photo of outdoor scene",
negativeelabell
"a close up shot of single object", product displayed in front of white back ground" painting",
"a collage of images",
"a screenshot of graphics user interface",
"a piece of text"
F.2 Metric Depth Estimation
As stated in the main paper, we choose Metric3Dv2 as our metric depth estimator. We have observed that Metric3Dv2 and WildCamera's camera intrinsic perform well on images taken in natural environments. In this section, we present the predicted normal maps from the depth model on OpenImages. These normal maps can be viewed as proxy estimate the quality of the reconstructed geometry's 's edges.
19

---

Figure 9: Predicted normal maps using Metric3Dv2 and WildCamera.
F.3
Point Cloud Processing
Here, we detailed how we process the point clouds into scene graphs.
Canonicalization. Our canonicalization method is straightforward. After obtaining the pitch and roll through PerspectiveFielss we transform the point cloud into canonicalized space using the inverse of the rotation matrix. Figure 10 illustrates the successful alignment of the ground surface with the z-axis angle after canonicalization. This process ensures that the axis-aligned bounding box accurately represents the vertical information of the objects, such as height and vertical distance. Our simple yet effective approach liberates our method from surface segmentation and RANSAC. We have empirically found this procedure robust for most natural images taken by cameras in real-world conditions.
Before Canonicalization
After Canonicalization
Figure 10: Canonicalization Results.
Denoising and constructing axis-aligned bounding box. The point clouds obtained from single- view depth may contain noise. Following [17, 22], carry out several denoising steps based on the approach to filter out outliers and unwanted points, thereby improving the robustness and accuracy of the bounding box. Initially, we eliminate statistical outliers from the object points and then downsample the data to a lower resolution. Subsequently, we use DBSCAN further remove noise. If the points of an object are fewer than ten after DBSCAN clustering, exclude that object area. Finally, we employ Open3D to create axis-aligned bounding boxes for each object. The pseudocode for our denoising process is as in Listing 2.
Listing 2: Point cloud denoising steps.
def process cacess_pcppppe (ped)
scale = norm (ped) std 3.0 + 1e-6
[pcd, ped. remove_stat st i cat_out ler (nb_nei geborss--111, std_ratio -1.2)
ped pcd. pcd.voxel voxe1 lase__ddo___ampp (vexel_sie -e-maxx (0.01, scale/40)) )
pod pcd_ddendiee denoise dbscan (
pcd, eps=0.2, min points=10
return ped
20

---

F.4 Open Spatial Dataset QA Templates
We provide samples for each category QA in the templates that we use to generate QAs mentioned in Section 3.1.
Listing 3: Template for QA synthesis.
distance seaanncettmplateeeeeee tttpeaate_uuesttonn..e.eeeeee
"What is the distance between [A] and [B]?", "How far away is [A] from [B]?",
"Can you provide the distance measurement between [A] and [B]?",
di staance ate_ansers
"[A] and [8] are [x] apart.",
"A cistance of [X] exists between [A] and [B).", "IAl and [B] are [X] apart from each other.",
"Is [A] to the left of [B] from the viewer's perapective?",
"Does [A] appear on the left side of [B]?", "Can you confirm if [A] is positioned to the left of [B]?",
left cert__rue__tte srruerrssonses =
"Yes, [A] is to the left of [B].", "Indeed, [A] is positioned on the left side of (3).", "Correct, you'll find (AI to the left of [B].",
"No, [A] is not to the left of [B].", "In fact, [A] is to the right of [BI.", "Incorrect, [Ål is not the left side of [B].",
directionn setectiio_questions =
"If you are at [A], where will you find [B]?"
cirection
"[B] is roughly at [X] clock from [A].", [[A] will find [B] around the [X] of clock direction."
F.5
LLM Prompts for Complex QA
messages "sssttem"" "content p""" You helpful assistant tasked with generating spatial reasoning-basee questions and answers from provided descriptions of scenes. Always craft a question without directly revealing specific details from the description. Always generate questions related to the description using <regionX>, The description should always be used answer and not leak into the question. When mentioning the objects regions, <regionX> instead of the objects regions. Speak like you the observer's perspective. Always make sure all the description objects or regions are mentioned with <regionX> in the question. """]
for sample in fewshot fewshot_samples: lussages.appente (( rooee: "user", "content" ": :samlle "context' }1)
messages... append (( "role"" "assistant" "content samplel ('responsee]
messages append "role": "user", "content": Van join (query) F)
Table 9: Llama-3 prompts for complex QA synthesis.
21

---

G Implementation Details for SpatialRGPT Architecture
G.1 Visual Backbone.
For SpatialRGPT-7B. we adopt pre-trained OpenAl CLIP-L model [64] as the visual backbone. As for SeaaiaRRGP-VVLLA------BB-. more visual details for the model, which can help with vision language tasks that require fine-grained details [50] and are beneficial for region-level representations [34].
we use x84 image resolutions with SigLIP [77] to include
G.2 Region-feature Extractor.
We adopt the region feature extraction technique in [19]. To begin with, we use feature refinement module consisting of 2-layer deconvolution network designed upscale the original feature map. Then, we employ MaskPooling to extract and average the refined features from the masked area.
G.3 Multi-modal Connector
To bridge representations from various modalities (e.g., image language, depth to language), we employ a simple linear layer. Following the approach suggested in [50], using straightforward connector helps the LLM concentrate more on processing visual inputs, thereby enhancing generalization. We implement two separate connectors, one for image embeddings and another for depth embeddings, to ensure that each modality is handled distinctly. This separation prevents the mixing of modalities, which could otherwise compromise the effectiveness of the model. Note that for SpatialRGPT-VILA-L1555515
we follow VILA-115 and use a two-layer MLP our connector.
H Implementation Details for Training SpatialRGPT
H.1 Instruction Tuning Data
Here, we list the instruction tuning data we use addition to the OSD dataset. Includes general instruction tuning datasets from LLAVA-1.5 1.5 [29], LAN-style instructions from VILA [50] (listed in Table 10) and the region-level instruction tuning data from [19] (listed in Table 1 11) that we use in stage three of the training.
Categories
Datasets
Captioning
Image Paragraph Captioning [78], MSR-VTT [79], TextCaps [80]
Reasoning
CLEVR [81], NLVR [82], VisualMRC [83]
Translation
Multi30k [84]
VQA
ActivityNet QA [85], DocVQA [86]. GQA [87], iVQA [88], MSRVTT-QA [79],
MSVD-QA [79], EQQA [89]. ST VQA [90], ViQuAE [91]. VQAv2 [92].
Visual Dialog [93]
Table 10: The general SFT blend [50] we used.
Categories
Datasets
Classification
V3Det [94], COCO [40], LVIS [95] V3Det [94] VG [96], RefCOCO [97]
Caption
Relationship
VG [96]
REC
RefCOCO [97]
Table 11: The region-level SFT blend [19] used.
H.2 Hyperparameters
Please refer to VILA's paper on the implementation of the hyperparameters used in the first two stages. In the instruction fine-tuning stage, the maximum learning rate is reduced to 2e-5, and the batch size is adjusted to 16. All other hyperparameters remain the same in the pre-training stage.
22

---

I Experimental Setting and Details
I.1 Experiments Compute Resources
Open Spatial Dataset. Our Open Spatial Dataset uses images from OpenImages, which contains total of 1.7 million images. Our data preprocessing pipeline tested system with 8 GPUs. The filtering process for 1.7 million images takes 4 hours and results in million samples. The camera calibration and metric depth estimation each took around hours. Note that the depth estimation requires our estimated camera intrinsics as input, so these two processes cannot be parallelized. The open-vocabulary detection and segmentation process takes hours. As the process involves sequential operations, we did not specifically optimize it for parallelization. For LLM-based QA synthesis, we employ LLama3-700 using sglang backend, which takes 12 hours. In general, the total time required to convert OpenImages into 3D scene graphs is within a day, and constructing the QAs takes another half.
SpatialRGPT Training, The first two stages of Spatial RGPT are inherited from VILA [50], which trained on 16 A100 GPU nodes, with each node having GPUs. The training times for each stage of the 7B model are follows: connector initialization takes 4 hours, visual language pre-training takes hours. The depth connector is further pre-trained using 2 100 GPU nodes, taking hours. The final visual instruction-tuning is also experimented on 2 A 100 GPU nodes, taking 12 hours.
SpatialRGPT-Benchh The SpatialRGPT-Bench dataset is created from ground truth 3D cuboids and numan-annotaaee labels. Masks only need to be generated when bounding boxes are provided. We use SAM-HQ in our data pipeline to convert the bounding boxes into masks, which takes approximately 4 hours to process 10,000 samples. After this, we synthesize QA and randomly select 1,500 samples. Subsequently, we conduct human verification to filter out incorrect annotations, which takes day to complete.
J Benchmark Evaluation Details
Our benchmark poses challenge in evaluation due to the possibility of multiple correct answers in different units. Typically, human trials, like those used by [17], could handle this but are often too slow and costly, mainly as our benchmarks include employ GPT-4 [55] to assess correctness. The evaluation process involves providing a question, the correct answer, and the model's response the L.M.. For qualitative questions, GPT-4 determines if the model's response aligns with the correct answer by assigning a score of 0 or 1. For quantitative questions, GPT-4 extracts numerical values from both the correct answer and the model's 's response, converting them to the same unit (such as meters). We then measure the accuracy and error of the model's response based on this standardized unit. We provide prompts we use in Table 13 and Table 12.
thousand samples. As an alternative,
messages = ("role"; "systen", "content"; Y"""You are a helpful assistant designed output JSON.
You should help me to evaluate the response given the question and the correct answer. To mark response, you should output single integer hetween 0 and 1.
(1) means that the response perfectly matches the answer. (0) means that the response completely different from the answer. """!
for sample in Fewshot_sampl cesssggs..pppon (( ((role"; "user", "content" ssample 'context' }1)
messages. pessages.append ((" """ol"";"""""""""""""""""""""""" "assistant", "content "; samplel 'responss"] ]]
messages. append (1 "role": "user", "content": Ann join (query)
Table 12: GPT-4 prompts for SpatialRGPT-Bnnch qualitative evaluation.
23

---

messages [ "ro_e": "systen", "content"; ": You are helpful assistant designed output JSON.
You should help me to evaluate the response given the question and the correct answer.
You need to convert the distance of the correct answer and response to meters.
The conversion factors are as follows:
1 inch 0.0254 meters, foot 0.3048 meters. 1 centimeter (em) 0.01 meters. should output two floats meters, one for the answer, and one the response. """)
for sample in
lussages.appenee (( (("oole" "usser", "content" peeent"ssaappe ssamlle 'context' (1)
messages append (( ("zole"; "assistant", "content : samplle 'response' 1|
messages. append (1 Prole": "user", "content"; Ann join (query) ))
Table 13: GPT-4 prompts for SpatialRGPT-Bench quantitative evaluation.
K
More Discussion on Limitations
For the most accurate object detection, oriented bounding boxes (OBB) are preferred over axis-aligned bounding boxes (AABB). As illustrated in Figure 11, the dimensions obtained from AABBs can differ from those obtained with OBBs. There are methods to compute an OBB. A simple method involves calculating the OBB using Principal Component Analysis (PCA) of the object's convex hull, which provides an approximate minimal bounding box. However, this approximation often lacks accuracy due to the incomplete and noisy nature of point clouds captured from a single view. Furthermore, this method still cannot handle extreme cases when objects are partially elevated (see Appdx C). The most precise method involves determining the OBB based on the object's pose, which currently challenging due limitations obtaining accurate object poses. Future improvements could include integrating available pose estimation approaches. However, currently, there are no gpen-vocabulary solutions for object pose estimation, so this remains an area for future research. Another direction, explored in subsequent work (c.g., Q-Spatial Bench [74]), addresses this limitation by leveraging human labeling.
Oriented Bounding Box
Axis-aligned Bounding Box
Figure 11: Different types of bounding box.
24

---

L
Broader Impacts
GpatialRGPT serves general- purpose visual assistant, similar other VLMs. It offers potential benefits and risks due to its integration of LLMs. SpatialRGPT shares similar concerns with LLMs, such as output hallucinations, inherited biases from base models, and energy consumption during upscaling. Evaluating SpatialRGPTTs performance is also challenging, particularly in accurately measuring the spatial information. This is an area for future enhancement, especially the field of robotics, which values safety. Despite these challenges, releasing Spatia/RGPT to the research community would be beneficial, would foster further development and improvement of robotics applications.
M Licenses
1. The training data we use, OpenImages [49], is released under Apache Li cense 2.0.
2. Our paper contain images from Unsplash [98], which released under Unsplash Li cense, allowing use of photos for free, including for commercial purposes, without attributing the photographer or Unsplash.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction clearly state four main contributions made in the paper. Our experimental results also support our claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have a dedicated section to discuss the limitations of our work. Please refer to Appx.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [No]
Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have included detailed information data pipcline implementation (Appx. F, architecture design(Appx. G, and training details(Appx. H.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
27

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have provided instructions on data access and preparation, including how access the raw, preprocessed, intermediate, and generated data. The data pipeline, data, model weights, and benchmark will be publicly available upon paper publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: httppss//niip../t/eeeeeees cc/ public/ guides/ CoodeSummi ss cessonoollion for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: cttpccccppcccccccccccc / /nniss.. cc/ publi icc////// gu dees/ Codee Subm ss i onPol icy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ye]]
Justification:
We provide detailed experimental setups in Appx.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In Figure 6, we plot the std for five runs.
Guidelines: In Figure 6, we plot the standard deviation of our predicted distance using five runs. Note that did not include error bars in the main table as both the training and evaluation are costly.
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we report the type of computing resources, memory, and time required to reproduce our experiments Appx. I.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /nnuriiss.pp.
Answer: [Yes]
Justification:
Our research conforms in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have dedicated section to discuss the societal impacts of our work. Please refer to Appx. L.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
29

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research builds upon existing datasets rather than creating new ones from the internet. So our work does not pose such risks the best our knowledge.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets used in our work are commonly used datasets with open access. We have adhered to their licenses and provided citations to give them credit.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset.
30

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers swi thoode com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The data pipeline, data, model weights, and benchmark will be publicly available upon paper publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Our research does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "ckhy6u1Lur", "reviewer": "Reviewer_y3Gs", "summary": "The paper introduces SpatialRGPT, a framework designed to enhance region-level spatial reasoning in Visual Language Models (VLMs) by incorporating 3D and region-aware visual encoder architecture. The authors present a scalable data pipeline to generate region-aware spatial reasoning questions and answers from existing datasets, resulting in the creation of the Open Spatial Dataset (OSD). To evaluate the model's performance, they introduce SpatialRGPT-Bench, a comprehensive benchmark with ground-truth 3D annotations. The paper demonstrates practical applications of SpatialRGPT, such as serving as a region-aware dense reward annotator for robotics and a stand-alone complex spatial reasoner.", "questions": "Please answer the questions in weakness section.", "limitations": "Please refer to the weakness section.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses the important problem of enhancing the spatial perception capabilities of multimodal LLMs.\n\n- It creates a large-scale training dataset with millions of examples.\n\n- The paper is well-organized and easy to follow, clearly explaining the authors' motivations at each step.\n\n- The effectiveness of the approach is demonstrated not only in vision-language tasks but also in embodied tasks.", "weaknesses": "- The biggest weakness, in my opinion, is the evaluation. The evaluation using SpatialRGPT-Bench shares the same data creation pipeline as the training data. This means the good performance on SpatialRGPT-Bench might just reflect the model learning the language style of the training data. Using GPT-4 for evaluation further biases the assessment towards responses that include numerical language, as seen in the teaser example: “The height of..1....is 204.54 feet. Assuming each floor is about 10 feet high, the total number of floors would be 20.454. Since you can’t have a fraction of a floor, the total number of floors would be approximately 20.” This type of response, while technically correct, doesn’t align with normal logical thinking. Therefore, a proper evaluation should be conducted on benchmarks like BLINK, especially those related to 3D tasks like depth.\n\n- In Table 2, SpatialRGPT-Depth underperforms the original VILA in 6 out of 8 benchmarks. \n\n- What if we use an off-the-shelf pretrained 3D detector on SpatialRGPT-Bench and then use an LLM to answer questions based on the cuboids from the 3D detector? On other benchmarks like BLINK, is SpatialRGPT better, or is the data curation method of first extracting 3D scene graphs and then using an LLM to summarize better?\n\n- Does stage three of training (Visual Instruction-tuning) require updating all model parameters?\n\n- Does the training data need to overlay all region proposals on the original images, like in Fig. 2, similar to Set-of-Marks?\n\n- The model explanation is unclear, and Figure 3 is confusing. So, the input includes RGB, depth maps, and region proposals (masks or bounding boxes)? Then, a shared visual encoder extracts global features from the RGB image and depth map, and independent connectors project the global RGB/depth feature embeddings into the word embedding space? How many tokens represent RGB and depth, respectively? For region-level features, is each object represented by two tokens, one from the RGB feature and one from the depth feature? Does the Region Feature Extractor take features from the last layer of the visual encoder? Do the RGB and depth features share the same Region Feature Extractor? Figure 3 shows it as shared, but the appendix suggests they are independent.\n\n- Why does the model need to extract region-level tokens separately? Couldn’t the region proposal information be included in the image-level token sequence through visual prompting, like in Set-of-Marks (SOM)? If LLAVA+SOM were trained on OSD, it might also work.\n\n- The model size is not reported—Is it 7B?\n\n- Typos: Line 327 should refer to Fig. 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q:** *SpatialRGPT-Bench uses same data pipeline as the training data, so its good performance might just reflect the model learning the training data's language style.*\n\n**A:** Please see General Response (B), we conduct additional experiments on a GPT-4 augmented SpatialRGPT-Bench. The results show that SpatialRGPT consistently outperforms the baseline models, even with different questions and answers from the training data.\n\n---\n\n**Q:** *Using GPT-4 for evaluation biases the assessment towards responses that include numerical language, as seen in the teaser example: “...”...doesn’t align with normal logical thinking.* \n\n**A:** As mentioned in General Response (A), we employed in-context learning for baselines. With in-context learning, we found that GPT-4 is 100% and GPT-4V is 99% willing to provide answers consisting of numbers and units for all quantitative samples. Additionally, our quantitative benchmark only contains straightforward spatial questions, such as the width or height of an object. The teaser sample mentioned by the reviewer is not included.\n\n---\n\n**Q:** *Evaluation should be conducted on benchmarks like BLINK, especially the depth task.*\n\n**A:** In General Response (C), we show SpatialRGPT’s results on BLINK’s Relative Depth Benchmark. **SpatialRGPT outperforms current SOTA with over 20% accuracy.**\n\n---\n\n**Q:** *In Table 2, SpatialRGPT underperforms the original VILA.*\n\n**A:** In Table 2 of our submission, we demonstrate that SpatialRGPT maintains comparable performance on general VLM benchmarks, as adding new tasks often leads to a significant drop in the original model's performance. We further conduct a series of studies on SpatialRGPT with different model sizes from VILA-1.5 (3B and 8B), benchmarking it on general VLM, region understanding, and spatial benchmarks. The results in Table 2, Table 3, and Table 4 (response pdf) illustrate SpatialRGPT's ability to learn spatial capabilities without sacrificing performance on general and regional understanding benchmarks. **Notably, our 8B model shows consistent improvements compared to baseline, with more than a 2-point improvement on most benchmarks.**\n\n---\n\n**Q:** *Can we use a 3D detector on SpatialRGPT-Bench and then use LLM to answer questions based on the cuboids from the 3D detector?*\n\n**A:** Following the reviewer's suggestion, we employ an Omni3D pretrained 3D detector and use GPT-4 to answer questions based on the detected cuboids. As shown in Table 4 of our response, LLMs struggle to effectively use coordinate information when presented in the text. Similar findings are reported in OpenEQA, where GPT-4 equipped with 3D bounding box information performs no better than without it.\n\n---\n\n**Q:** *Compare SpatialRGPT vs LLM + data curation method on BLINK.*\n\n**A:** Our data curation method is object-centric, whereas BLINK requires point-level depth understanding. Therefore, our data pipeline cannot be directly applied to BLINK.\n\n---\n\n**Q:** *Does stage three of training require updating all model param.?*\n\n**A:** No. We freeze the vision encoder and update the rest.\n\n---\n\n**Q:** *Does the training data need to overlay all region proposals on the original images, like in Fig. 2, similar to SoM?*\n\n**A:** No. SpatialRGPT does not require overlayed region proposals on images. The overlay in Figure 2 (submission) is purely for visualization. Overlaying region proposals on images (as done in SoM) is straightforward but has significant drawbacks:\n* Ambiguity: The exact boundaries of the desired region are unclear.\n* Occlusion: Marks or lines can hide regions, hindering semantics, and small regions may be entirely obscured by annotations.\n* Sensitivity: The performance can be affected by the design of the marks, including shape, size, and color, as shown in prior studies.\n\n---\n\n**Q:** *The model explanation and Fig. 3 is unclear. Does the input include RGB/D, and region proposals?*\n\n**A:** Yes.\n\n---\n\n**Q:** *How many tokens represent RGB/D, respectively?*\n\n**A:** A shared visual encoder extracts 576 tokens each from the RGB image and depth map (24 * 24). Each RGB token is projected into the word embedding space through the RGB connector. Only the RGB tokens (576) are prepended before the text as a global context.\n\n---\n\n**Q:** *For region-level features, is each object represented by two tokens, one from the RGB feature and one from the depth feature? Does the Region Feature Extractor take features from the last layer of the visual encoder?*\n\n**A:** Yes. The extractor takes 576 tokens from the last visual encoder layer. These are upsampled to 9216 tokens (96 * 96) in the Feature Refinement Layer. After the Mask-pooling Layer, each object has one RGB and one depth token, which are then projected into the word embedding space through separate connectors.\n\n---\n\n**Q:** *Do the RGB and depth features share the same Region Feature Extractor? Figure 3 shows it as shared, but the appendix suggests they are independent.*\n\n**A:** The Region Feature Extractor consists of a Feature Refinement Layer (deconvolutions) and a Mask-pooling Layer (no parameters). RGB and depth features have separate Feature Refinement Layers. We will revise Figure 3 in the submission accordingly.\n\n---\n\n**Q:** *Why does the model need to extract region-level tokens separately? Why not use visual prompting like in SoMs?*\n\n**A:** Using region extractors to obtain region-level tokens avoids SoM's drawbacks:\n* Ambiguity: Boxes or masks precisely identify regions.\n* Occlusion: No overlays mean regions aren’t obscured.\n* Sensitivity: No need for annotations, eliminating concerns about mark design.\n\n---\n\n**Q:** *The model size is not reported—Is it 7B?*\n\n**A:** Yes. As mentioned in Section 3.3 (L209 submission) we use LLaMA2-7B as our base LLM. We will include this information in the main table.\n\n---\n\n**Q:** *Typos: Line 327 should refer to Fig. 5.*\n\n**A:** Thank you for pointing out the typo. Line 327 should indeed refer to Figure 5 We will revise it accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fxQftW6H6T", "reviewer": "Reviewer_ojLE", "summary": "The paper introduces a novel approach for generating 3D, region-aware annotations from 2D images, transforming scene graphs into spatial QA training data for VLMs using a combination of template-based and LLM approaches. Key contributions include:\n1. A novel pipeline for automatic generation of complex, metric spatial QA data.\n2. A proposed depth adapter to include relative depth maps as input to the VLM.\n3. Benchmarking the generated annotations against state-of-the-art methods, showing improvements in spatial reasoning tasks.", "questions": "1. How close are the questions in the evaluation set to the templated questions in the data generation pipeline? What steps are taken to ensure that the question and answer formatting do not give an evaluation advantage? If you change the evaluation question formatting, does it affect the model's performance?\n2. Does the depth adapter addition alone, trained on non-spatial data, provide a performance boost even without the added spatial training data?\n3. Why is SpatialVLM not included as a baseline, given its relevance?\n3. What is the human performance on the evaluation set or a subset, given that humans may be worse than the model at metric spatial reasoning?", "limitations": "The authors discuss limitations in the appendix. Moving limitation discussion to the main paper and addressing the questions outlined above would strengthen the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The integration of scene graphs with template-based and LLM-based QA generation, along with the use of a depth map adapter, provides a novel approach to spatial QA VLM training.\n- The experimental results are strong, demonstrating significant improvements over state-of-the-art models like GPT-4V+SoM and Llava-Next.\n- The methodology is well-explained, with clear descriptions of the data collection process and the architecture of the proposed model.", "weaknesses": "- The paper lacks a detailed analysis of how closely the questions in the evaluation set match the templated questions from the data generation pipeline. It is unclear if the formatting provides an unfair advantage to the model, and whether altering the evaluation question formatting affects model performance.\n- The paper does not include a discussion on whether the depth adapter alone, when trained on non-spatial data, improves performance. Additionally, it is unclear why SpatialVLM, a relevant baseline, was not included in the comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q:** *How close are the questions in the evaluation set to the templated questions in the data generation pipeline?*\n\n**A:** The questions from both the evaluation set and data generation pipeline are randomly sampled from a set of templates.\n\n---\n\n**Q:** *What steps are taken to ensure that the question and answer formatting do not give an evaluation advantage?*\n\n**A:** We have taken measures to avoid potential advantages from the QA format. Please see [General Response](https://openreview.net/forum?id=JKEIYQUSUc&noteId=04aWnh2vt2) (A) for clarification. \n\n---\n\n**Q:** *If you change the evaluation question formatting, does it affect the model's performance?*\n\n**A:** In [General Response](https://openreview.net/forum?id=JKEIYQUSUc&noteId=04aWnh2vt2) (B), we conduct an experiment on a GPT-4 augmented SpatialRGPT-Bench. The results demonstrate that SpatialRGPT continues to outperform the baselines even when the questions and answers differ from the training data. Additionally, in [General Response](https://openreview.net/forum?id=JKEIYQUSUc&noteId=04aWnh2vt2)  (C), we show that SpatialRGPT is state-of-the-art on a public depth-related benchmark.\n\n---\n\n**Q:** *Does the depth adapter addition alone, trained on non-spatial data, provide a performance boost even without the added spatial training data?*\n\n**A:** No. As mentioned in Line 223, the depth connector is a plugin module specifically trained on spatial-related QAs. Since it is not trained on non-spatial data and, thus, does not improve performance when spatial-related data is not available. For non-spatial data, depth inputs are not included to avoid redundant information from depth data for non-spatial tasks.\n\n---\n\n**Q:** *Why is SpatialVLM not included as a baseline, given its relevance?*\n\n**A:** SpatialVLM is not open-sourced. In Table 4 (response pdf), we provide a comparison to SpaceLLaVA, a 3rd party community implementation mentioned on SpatialVLM’s website. \n\n---\n\n**Q:** *What is the human performance on the evaluation set or a subset, given that humans may be worse than the model at metric spatial reasoning?*\n\n**A:** We show human performance on SpatialRGPT-Bench in Table 4 (response pdf). We observe that while qualitative QAs are easy for humans (97% average accuracy), quantitative QAs are extremely hard for humans (less than 50% accuracy). This supports the reviewer's suggestion that humans may be worse at metric-scale spatial reasoning.\n\n---\n\n**Q:** *The authors discuss limitations in the appendix. Moving limitation discussion to the main paper and addressing the questions outlined above would strengthen the paper.*\n\n**A:** Thank you for the suggestion, we will revise the paper accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1T6m1tOsit", "reviewer": "Reviewer_hHf2", "summary": "This paper constructs region-aware spatial reasoning QA datasets from existing sources, resulting in the Open Spatial Dataset (OSD). Based on the OSD, they develop a model called SpatialRGPT, which integrates depth information to enable effective representation of regional information and acquisition of spatial knowledge. Experimental results demonstrate the superior spatial reasoning ability of the proposed model and its potential applications in robotics.", "questions": "1. In Figure 4, the fifth QA example asks, \"What kind of vehicles would not fit in?\". Is this an example from the SpatialRGPT-Bench, and which category does it belong to? This question involves height, width, and depth of the garage area, which may not align well with the current categories and metrics used for evaluation.", "limitations": "See Weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed dataset OSD is well-crafted with open-vocabulary regions, which should benefit future research by enhancing models’ spatial reasoning abilities.\n2. Integrating depth information into the VLM is a novel approach for achieving more accurate spatial reasoning.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The SpatialRGPT-Bench is constructed through the proposed data generation pipeline, sharing the same answer formats as the OSD dataset on which the SpatialRGPT is trained. This may introduce bias when directly comparing it with other models not trained on the OSD dataset.\n2. There is a lack of clarity regarding the size of SpatialRGPT used in the experiments. In Table 1, it is compared with a 34B model (LLaVA v1.6), while in Table 2, it is compared with 7B models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q:** *SpatialRGPT-Bench is constructed through the proposed data generation pipeline, sharing the same answer formats as the OSD dataset on which the SpatialRGPT is trained. This may introduce bias when directly comparing it with other models not trained on the OSD dataset.*\n\n**A:** Please refer to the [General Response](https://openreview.net/forum?id=JKEIYQUSUc&noteId=04aWnh2vt2) for clarification. We explain the steps we have taken to ensure fair evaluation of SpatialRGPT-Bench in (A), conduct additional experiments on GPT-4 augmented SpatialRGPT-Bench in (B), and evaluate SpatialRGPT’s performance on a public benchmark, BLINK, in (C).\n\n---\n\n**Q:** *There is a lack of clarity regarding the size of SpatialRGPT used in the experiments. In Table 1, it is compared with a 34B model (LLaVA v1.6), while in Table 2, it is compared with 7B models.*\n\n**A:** As mentioned in Section 3.3 (L209), we use LLaMA2-7B as our base LLM. We will include our model size in the tables for clarity. In Table 1 (submission), we compare our model to larger models (≥ 7B) in the spatial-related benchmark, while in Table 2  (submission), we compare it to models of the same size (7B) for general VLM benchmarks.\n\n---\n\n**Q:** *In Figure 4, the fifth QA example asks, \"What kind of vehicles would not fit in?\". Is this an example from the SpatialRGPT-Bench, and which category does it belong to? This question involves height, width, and depth of the garage area, which may not align well with the current categories and metrics used for evaluation.*\n\n**A:** No, the questions in Figure 4 (submission) are not examples from the SRGPT-Bench. Examples from the SRGPT-Bench can be found in Figure 7 (submission), where each question contains only one category type (height, width, etc.).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HA0VWXfFSq", "reviewer": "Reviewer_v4dm", "summary": "This paper introduces SpatialRGPT, targeting at enhancing the spatial reasoning abilities of VLMs. The authors introduce a data curation pipeline along with a benchmark that facilitates the learning and evaluation of 3D spatial knowledge. Experiments show that SpatialRGPT thrives at spatial reasoning and performs comparable to SOTA on standard VQA benchmarks. The authors also showcase some real-world applications of such a model by using it for complex spatial reasoning and as a reward annotator for robot manipulation.", "questions": "1. Since Open Spatial Dataset plays an important role in enhancing the ability of spatial reasoning, I'm wondering if the data that targets at different aspect would affect each other, i.e. if I remove the QA pair regarding width and height, will it affect the models performance on answering big/small questions?\n2. Despite the task is targeted at region-based spatial reasoning, I'm wondering how well the model would perform on VQA tasks that includes single region but multi-hop reasoning, i.e. what is the object on the table to the right of <region1>?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces a data curation pipeline that facilitates spatial relation learning, along with a benchmark that's designed specifically for this important task, which will all be made available, making it easy to follow.\n2. This paper introduces a plug-and-play module that process depth information for VLM. Experimental results showcase that the module is useful for cases that involves reasoning among behind/front, wide/thin, and estimating distances.\n3. Results on real robot shows that the learned spatial relation representation can indeed be used for downstream tasks.", "weaknesses": "1. When constructing 3D scene graph, the author mentioned using 3D axis-aligned bounding boxes for objects to compute the width and height. This could lead to inaccurate measure of the object size. It'd be great if the author could show some results on how much does this affect the overall data quality, i.e. how many objects are measured inaccurately because of the AABB assumptions of the bounding boxes.\n2. When discussing the potential real-world application, the author showcased SpatialRGPT can be used as a dense reward annotator. However, the annotation process still needs manually defining regions of interest for every frame in the demonstration video, hindering its use at a broader impact.\n3. The width and height result in Table 1 suggests that SpatialRGPT underperforms the best model, i.e. GPT-4V by 10.5 in success rate, this is a major concern to me since GPT-4V only has commonsense knowledge on the object size and still outperforms SpatialRGPT, which is trained on object size knowledge, by a large margin. Therefore, I'm wondering if the data of object size is useful, i.e. without this portion of data, how would SpatialRGPT perform in the task of reasoning about width and height.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q:** *It'd be great if the author could show some results on how much AABB assumptions affect the overall data quality, i.e. how many objects are measured inaccurately because of the AABB assumptions of the bounding boxes.*\n\n**A:** We conduct an ablation study to examine the effect of using axis-aligned bounding boxes (AABB) versus PCA-based oriented bounding boxes (OBB). For this study, we use human-labeled OBBs from the Omni3D test set as the ground truth. We then compare the mean-square error of the width and height measurements for AABBs and PCA-based OBBs labeled by our 3D scene graph pipeline. The results are shown in Table 5 (response pdf). As noted in L747 of our paper, PCA-based OBB often lacks accuracy due to the incomplete and noisy nature of point clouds captured from a single view. \n\n---\n\n**Q:** *When SpatialRGPT functioned as a dense reward annotator, the annotation process needed to manually define regions of interest for every frame in the demonstration video, hindering SpatialRGPT’s use at a broader impact.*\n\n**A:** SpatialRGPT can be combined with video segmentation approaches such as SAM-v2. The video can be annotated by clicking on the object (point prompt) only in the first frame. \n\n---\n\n**Q:** *The width and height result in Table 1 suggests that SpatialRGPT underperforms the best model, GPT-4V by 10.5 in success rate.*\n\n**A:** The success rates presented in Table 1 (submission) are based on a threshold cap. Despite a lower success rate, SpatialRGPT still outperforms GPT-4V in terms of absolute relative error for both width and height measurements. The lower success rate is attributed to SRGPT-Bench being derived from Omni3D and operating within a closed-set setting, with objects from a limited number of commonly seen classes (e.g., human, car, chair), which are usually easy for GPT-4 models to estimate within a reasonable range. Therefore, the advantages of SpatialRGPT are not fully apparent under these conditions. Creating an open-world or more diverse 3D ground truth annotation dataset will provide a more comprehensive evaluation framework and can better demonstrate the strengths of SpatialRGPT. We leave this as future work.\n\n---\n\n**Q:** *Will the data that targets at different aspect would affect each other? i.e. if we remove the QA pair regarding width and height, will it affect the models performance on answering big/small questions?*\n\n**A:** Following the reviewer’s advice, we conduct a study to see if adding width and height data affects other types of questions. As shown in Table 7 (response pdf), adding this data slightly improved the accuracy for questions about size (like big/small, tall/short, wide/thin) but slightly worsened the accuracy for questions about the distance between objects (horizontal and vertical). This suggests that information about object size helps with size-related questions but might make distance measurements less clear.\n\n---\n\n**Q:** *How well the model would perform on VQA tasks that includes single region but multi-hop reasoning, i.e. what is the object on the table to the right of <region1>?*\n\n**A:** In Figure 1 (response pdf), we show examples of SpatialRGPT handling multi-hop reasoning; we recommend zooming in for a clearer view. In the upper left sample, the model first identifies what's to the right of Region [0] (a single apple), finds the basket there, determines what's inside the basket, and then provides spatial details about the object inside. Even though our training data doesn't specifically include such multi-hop tasks, SpatialRGPT can still manage them effectively. This indicates that the model has developed a strong understanding of spatial relationships.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for recognizing the importance of our research problem (Reviewer `y3Gs`), and for acknowledging the novelty (Reviewer `ojLE`, `hHf2`), effectiveness (Reviewer `y3Gs`), and usefulness (Reviewer `v4dm`) of our approach. Below, we address the reviewers' common feedback, particularly regarding the evaluation of SpatialRGPT.

**(A) Ensuring Fair Evaluation on SpatialRGPT-Bench**

While Reviewer `y3GS`, `ojLE` and `hHf2` pointed out that the current benchmark may be biased to the proposed model, to ensure a fair evaluation on SpatialRGPT-Bench, we took the following measures to avoid potential advantages from the QA format:
* **Number-Aware Evaluations:** As mentioned in Line 738 of the submission, for quantitative questions, we used GPT-4 to extract numerical values and units from the answers. We calculated accuracy and error metrics only on these extracted values, ensuring that the evaluation did not favor our model's response due to textual formatting.
* **In-Context Learning for Baselines:** We provided baseline models with example QA samples, including both quantitative and qualitative questions, to enable in-context learning.

**(B) Augmented/Rephrased SpatialRGPT-Bench**

Following Reviewer `ojLE` and `hHf2`'s suggestions, we conduct additional experiments by augmenting and rephrasing both questions and answers in SpatialRGPT-Bench using GPT-4. The results are shown in Table 1 (response pdf). The results show that SpatialRGPT consistently outperforms the baseline models, even when the questions and answers are different from the training data.

**(C) Evaluation Results on BLINK**

Following Reviewer `y3Gs`'s suggestion, we evaluate SpatialRGPT on BLINK’s Relative Depth Benchmark. This benchmark is particularly challenging as it assesses point-level depths, while both the point-level region input and point-level questions were not specifically included in the training of SpatialRGPT. We use bounding boxes to mark the target points and evaluate the test set online with the EvalAI server. As shown in Table 6 (response pdf), SpatialRGPT significantly outperforms the state-of-the-art, **achieving over 20% accuracy gain compared to GPT-4V-Turbo**. Our model demonstrated strong performance, highlighting its ability to generalize to new tasks without explicit training.

Meta-review: The reviewers unanimously vote for the acceptance of the paper. The rebuttal successfully addressed most of the concerns from the reviewers, including fairness in evaluation. The AC does not see any reason to turn down the reviewer opinions.
Decision: Accept (poster)