Title: SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Vision-Language Models

Spatial Reasoning
Abstract: Vision Language Models (VLMs) have demonstrated remarkable performance in 2D vision and language tasks. However, their ability to reason about spatial arrangements remains limited. In this work, we introduce Spatial Region GPT (SpatialRGPT) to enhance VLMs’ spatial perception and reasoning capabilities. SpatialRGPT advances VLMs’ spatial understanding through two key innovations: (i) a data curation pipeline that enables effective learning of regional representation from 3D scene graphs, and (ii) a flexible ``plugin'' module for integrating depth information into the visual encoder of existing VLMs. During inference, when provided with user-specified region proposals, SpatialRGPT can accurately perceive their relative directions and distances. Additionally, we propose SpatialRGBT-Bench, a benchmark with ground-truth 3D annotations encompassing indoor, outdoor, and simulated environments, for evaluating 3D spatial cognition in Vision-Language Models (VLMs). Our results demonstrate that SpatialRGPT significantly enhances performance in spatial reasoning tasks, both with and without local region prompts. The model also exhibits strong generalization capabilities, effectively reasoning about complex spatial relations and functioning as a region-aware dense reward annotator for robotic tasks. Code, dataset, and benchmark are released at https://www.anjiecheng.me/SpatialRGPT.

SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models

An-Chieh Cheng 1 , Hongxu Yin 2 , Yang Fu 1 , Qiushan Guo 2 , Ruihan Yang 1 , Jan Kautz 2 , Xiaolong Wang 1,2 , Sifei Liu 2

1 UC San Diego, 2 NVIDIA

blue sky

1

2

3

4

5

6

7

Can you confirm if......has a greater

width compared to......?

Relative Relation

In

fact,

......

might

be

narrower

than

......

.

Which of these four,......,......,......, ......is

the tallest?

......

is the tallest.

From the viewer's perspective, is the

position of......closer than that of......?

No,

......

is closer.

Metric Measurement

If  you  are  at......,  where  will  you

find......?

......

is at 1 o'clock from

......

.

How  far  apart  are......and......hori-

zontally?

They  are

15.33

meters  apart

horizontally.

Spatial Reasoning

Can you try to estimate how many

floors are in......? Think step by step.

The  height  of ...... is  204.54  feet.  As- suming  each  floor  is  about  10  feet high, the total  number  of  floors would  be  20.454.  Since  you  can't have  a  fraction  of  a  floor,  the  total number of floors would be approxi- mately 20. 1

I'm

planning

to

build

a

bridge

from......to......, is it possible?

The  distance  between  ....  and....  is

14.82  meters,  so  it  is  possible  to

build a bridge between them.

If  you are riding a motorcycle with 36  inches  wide,  do  you  think  you can  pass  through  the  area  between .... and .... ? 4 3

The  distance  between  ....  and  ....  is 38.95  inches,  so  yes,  you  can  pass through the area between .... and .... since  the  motorcycle  is  narrower than the distance between them. 4 3 4 3

1

1

2

4

6

4

6

3

4

5

6

6

6

3

3

3

7

7

3

2

3

What is the height of......?

......

is 1.38 meters tall.

4

4

1

2

A

B

C

Vision Language Models (VLMs) have demonstrated remarkable performance in 2D vision and language tasks. However, their ability to reason about spatial arrangements remains limited. In this work, we introduce Spatial Region GPT (SpatialRGPT) to enhance VLMs' spatial perception and reasoning capabilities. SpatialRGPT advances VLMs' spatial understanding through two key innovations: (i) a data curation pipeline that enables effective learning of regional representation from 3D scene graphs, and (ii) a flexible 'plugin' module for integrating depth information into the visual encoder of existing VLMs. During inference, when provided with user-specified region proposals, SpatialRGPT can accurately perceive their relative directions and distances. Additionally, we propose SpatialRGBT-Bench, a benchmark with ground-truth 3D annotations encompassing indoor, outdoor, and simulated environments, for evaluating 3D spatial cognition in VLMs. Our results demonstrate that SpatialRGPT significantly enhances performance in spatial reasoning tasks, both with and without local region prompts. The model also exhibits strong generalization capabilities, effectively reasoning about complex spatial relations and functioning as a region-aware dense reward annotator for robotic tasks. Code, dataset, and benchmark are released at https://www.anjiecheng.me/SpatialRGPT .

1 Introduction

Understanding spatial arrangements in both 2D [1, 2] and 3D [3] spaces is crucial for accurately interpreting complex visual environments. Despite the impressive advancements in Vision Language Models (VLMs) across a variety of tasks such as image classification [4], captioning [5], object detection [6], video understanding [7], and document parsing [8], etc., these models still face significant challenges with spatial reasoning. This includes difficulties [9, 10, 11] in distinguishing simple spatial concepts like "left" and "right," "above" and "below," as well as more complex relationships such as "behind" and "in front," "inside" and "outside," and "near" and "far." The ability to comprehend and reason about these spatial relationships is fundamental not only for visual understanding, but also for enabling practical applications in fields like robotics [12, 13] and augmented reality [14], where precise spatial awareness is crucial for tasks such as navigation [15], manipulation [12], and interaction with real-world environments [16].

Recently, several works [11, 17, 18] has advanced VLMs' spatial reasoning capabilities by introducing a comprehensive data generation pipeline that enables large-scale training with spatially-aware visual question answering (VQA) tasks. This approach is based on the hypothesis that the limited spatial reasoning capabilities of current VLMs are due to a lack of 3D/2D spatial knowledge in their training data. However, two critical challenges remain. First, effective spatial reasoning requires VLMs to accurately parse regional information, particularly the regions of object instances, whereas most existing VLMs are primarily designed to understand the global context of an image. When an image contains numerous instances, it becomes challenging to prompt the model to reason about the spatial relations between specific instances. This is because most VLMs function as global image parsers and do not support specifying regions for which users want to understand spatial relationships. Second, accurately perceiving spatial relations such as direction and distance cannot rely solely on RGB pixel data. Thus, the architecture needs to incorporate 3D inputs, such as depth information.

In this work, we propose SpatialRGPT, leveraging a data curation pipeline, along with a region and 3D-aware visual encoder architecture to improve the spatial reasoning capability of VLMs.

Our data pipeline automatically generates 3D, region-aware annotations from 2D images at scale by constructing a 3D scene graph for each image, where nodes represent object instances and edges denote spatial relationships. This is achieved through three scalable components: (i) open-vocabulary detection and segmentation for instance extraction, (ii) metric depth estimation, and (iii) camera calibration for projecting objects into 3D space. These scene graphs are subsequently transformed into region-aware spatial QA tasks using both template-based and large language model (LLM)-based approaches. This dual approach provides region-based VLMs with the necessary spatial knowledge and advanced reasoning capabilities to interpret complex environments. We use the collected data to train SpatialRGPT. While SpatialRGPT is designed to support region prompts, it effectively avoids the ambiguity issues found in SpatialVLM. In SpatialVLM, multiple similar objects in an image can confuse caption labels. In contrast, our pipeline naturally handles these scenarios without requiring carefully crafted rules or extensive post-processing.

Similar to RGPT [19], SpatialRGPT introduces a region representation module that allows region proposals to be included as additional inputs alongside the image. This approach enables the LLM to leverage both regional and global contexts, allowing the model to reason about relationships between local regions while maintaining an understanding of the overall scene. In addition, we propose a novel architecture that features a flexible 'plugin' module for integrating relative-depth information into the visual encoder of existing VLMs. This design allows a pre-trained visual encoder to optionally learn additional depth representation while still functioning effectively when depth inputs are absent. Our experiments demonstrate that this design can substantially improve the spatial reasoning capabilities compared to VLMs that only use RGB images as input. Furthermore, we highlight practical applications enabled by SpatialRGPT, such as serving as a region-aware dense reward annotator and a stand-alone complex spatial reasoner. Our work has four main contributions:

We present SpatialRGPT, a framework that enhances region-level spatial reasoning in VLMs by enabling effective representation of regional information and acquisition of spatial knowledge. Our novel architecture also integrates depth information flexibly, significantly improving 3D perception and analysis.

To facilitate model training, we introduce a scalable data pipeline that constructs regionaware spatial reasoning QAs from existing datasets. With the pipeline, we create the Open Spatial Dataset (OSD), encompassing 8.7M spatial concepts grounded in 5M unique regions.

To address the absence of a benchmark for evaluating spatial cognition in VLMs, we present SpatialRGPT-Bench, a comprehensive benchmark based on ground-truth 3D annotations that span indoor, outdoor, and simulated environments.

Wedemonstrate downstream applications of SpatialRGPT. Leveraging SpatialRGPT's region capabilities, we develop a region-aware dense reward annotator for robotics. Additionally, we show that SpatialRGPT can function as a stand-alone complex spatial reasoner, as well as its capacity to perform multi-hop reasoning.

2 Related work

Spatial Reasoning via Large Language Models. Recently, there has been a significant push to obtain spatial reasoning capabilities using LLMs. Initiatives [20, 21] have focused on reconstructing scenes from multi-view images, such as point clouds or neural fields, and enhancing these representations with dense semantic features. The resulting 3D representation and dense features are then integrated into an LLM. However, multi-view images are not always available, and constructing a scene explicitly with dense semantic features is resource-intensive. Additionally, the modal gap between 3D representations and language often results in decreased performance. ConceptGraph [22] avoids directly incorporating 3D representations into LLMs. Instead, it constructs a scene graph and integrates this with the LLM. Yet, recent studies [10] indicate that LLMs struggle to utilize coordinate information effectively when presented in text, which can undermine their ability to understand and reason about spatial relationships. Our research is most aligned with SpatialVLM [17], which uses 2D VLMs to understand spatial relationships and metric distances. Unlike the above approaches, the spatial understanding is encoded implicitly. The VLM directly handles the spatial relationship problem without an explicit 3D representation or scene graph. However, SpatialVLM relies on language descriptions of objects as input, while LLMs can already resolve some spatial queries even without visual data [23]. The responses can be inferred directly from the questions or derived from the world knowledge embedded in LLMs. This reliance on textual cues suggests that the training may not effectively teach VLMs to learn spatial reasoning from visual data. Additionally, SpatialVLM lacks the capability to specify regions precisely. This is particularly problematic in real-world scenarios where describing ambiguous locations or objects in language can be challenging.

Region-level Visual Language Models. KOSMOS-2 [24], Shikra [25], MiniGPT-2 [26], CogVLM [27], SPHINX [28], and LLaVA [29] have enabled MLLMs to achieve region-based image understanding. However, these methods provide region information in textual form, such as bounding box coordinates. This method heavily depends on the language decoder to understand the position. In contrast, VisionLLM [30], GPT4RoI [31], [32], and Ferret [33, 34], along with GLaMM [35], use spatial boxes with ROI-aligned features to map region-level features into the LLM word embedding space. However, bounding boxes can include unwanted background features, leading to inaccurate alignment between region descriptions and text, which complicates spatial reasoning. Recently, RegionGPT [19] and Osprey [36] have introduced visual spatial-aware modules that can directly extract pixel-level features. These models support using input masks that can accommodate regions of any shape. Despite these advancements, none of these approaches specifically focus on enhancing spatial reasoning at the region level in VLMs. Our framework is based on RegionGPT's ability to process pixel-level inputs, with the aim of deepening spatial reasoning within region VLMs.

3 Method

SpatialRGPT is a powerful multimodal language model adept at understanding both 2D and 3D spatial arrangements. It can process any region proposal, such as boxes or masks, and provide answers to spatial reasoning questions. While effective training dataset is the key to learn spatial-aware region representation, we introduce: (i) how to build 3D scene Graph from a single image, in Sec. 3.1, and (ii) how to facilitate visual representation learning from these scene graphs in Sec. 3.2. We propose a novel SpatialRGPT visual encoder architecture that flexibly leveraging monocular depth information into an existing 2D VLM, in Sec. 3.3, with training detail explained in Sec. 3.1.

3.1 3D Scene Graph from Single 2D Images

Our scene graph construction pipeline (Figure1) begins with a filtering process to remove any unsuitable images (Appx.F.1). Using open-vocabulary models, we identify and ground candidate objects, followed by lifting them into 3D space using metric depth estimation and camera calibration. We then process the point clouds (Appx. F.3) to construct the final 3D scene graph.

Open-Vocabulary Detection & Segmentation. Segmenting objects is the initial stage of building a scene graph. Our models must satisfy two criteria: (i) object descriptions, e.g., class labels, should

Reasoning QA

RGB Image

Region Masks

Pitch, Roll, Intrinsics

Point Cloud Processing

Image Collections

Open-Vocabulary

Detection & Seg.

Estimation

Camera

Calibration

Filtering

Template QA

LLM

GETREAL.BEWELL.

Gg

Y

司

adhere to an open-world setting for better generalization; (ii) mask proposals need to be highly accurate, ensuring precise contour outlines. This precision is crucial, as even small deviations can lead to significant inaccuracies in the resulting 3D bounding boxes. To this end, we first employ an open-vocabulary image tagging model [37] to identify all the object classes present in the image. Next, we use GroundingDino [38], an open-vocabulary 2D detector to determine the corresponding object bounding boxes. Finally, we apply segmentation models [39] to refine these bounding boxes into precise masks. We do not use existing dataset annotations since they either fall short due to vocabulary limitations, or use polygon annotations [40] or compressed masks [41] for segmentation.

Metric Depth Estimation. Several studies have explored the recovery of metric depth from a single image. The main challenge is to address the scale ambiguity, and one common approach [42, 43] is to use relative depth along with metric heads fine-tuned on specific metric datasets. However, these methods may tend to overfit the depth scale for particular datasets such as KITTI [44] or NYU [45], which makes them less robust for in-the-wild images. Recently, Metric3Dv2 [46] takes focal length as input and is trained end-to-end to predict metric depth and surface normals. The model is trained jointly on diverse indoor and outdoor scenes, making it less prone to overfitting to the depth distribution of specific datasets. We adopt Metric3Dv2 as our metric depth estimator and found that Metric3Dv2 together with WildCamera [47]'s camera intrinsic, is robust for images taken in real-world settings. Additionally, thanks to the joint depth-normal optimization training in Metric3Dv2, the recovered geometry is improved particularly around object edges.

Camera Calibration. Camera calibration includes (i) intrinsic estimation to back-project depth maps to 3D point clouds, and (ii) scene canonicalization to ensure that scene relations are described in a shared space. To estimate the camera intrinsic, we use the WildCamera model [47], which estimates four DoF intrinsic parameters (focal point and focal length in two dimensions). This model excels in real-world scenarios due to its scale-awareness and ability to detect image cropping. To convert the camera coordinates of the point cloud into a canonicalized geodetic coordinate system for each scene, we leverage PerspectiveFields [48], which provides per-pixel up-vectors and latitude values that can be transformed into camera extrinsics, such as pitch and roll. Using these, we derive a rotation matrix to convert the point cloud from camera coordinates to geodetic coordinates. We note that while SpatialVLM [17] uses surface segmentation (e.g., "floor," "tabletop") to identify a horizontal plane and then uses the normal axis of this plane to align the point cloud to the horizontal plane, this approach is limited by the presence of specific classes, such as floors or tables. Additionally, the plane segmentation may fail if there are not enough points for RANSAC.

Constructing 3D Scene Graph. The 3D scene graph is a collection of tuples where the nodes represent specific 3D object instances, and the edges represent the spatial relationships between the nodes. Each node is defined by the object's class , width , and height in metric scale. To create the node, we start by using the instance mask to deproject the object points from the depth map. Then, we perform canonicalization and denoising, and build 3D axis-aligned bounding boxes for each object. With the 3D bounding box, we calculate the width and height of the objects in

How wide is Region [1]?

The width of Region [1] is 7.73 feet.

Is Region [1] behind Region [4]?

No, it is in front of Region [4].

You are a visitor in a museum and see two sculptures, one in Region [0] and the other in Region [1]. If you  walk  from  one  sculpture  to the other, how far will you have walked?

You will have walked 4.85 meters.

You  are  a  helicopter  pilot flying over the city and you see  Region  [1]  and  Region [7]. Which one is higher?

The  tower  in  Region  [1]  is higher  than  the  skyscraper in Region [7].

4

1

real-world units. The edges represent the spatial relationships between the nodes within two types of relations: relative and metric. Relative relations contain left , right , above , below , behind , front , wide , thin , tall , short , big , and small . Metric relations include direction , direct distance , horizontal distance , and vertical distance between the two objects. We then traverse all the object nodes and use the point cloud centroids and bounding boxes to calculate their spatial relationships.

3.2 Learning Spatial-aware VLMs from 3D Scene Graph

In this section, we discuss converting the constructed 3D scene graph into textual representations for VLM training. One simple approach is through template-based methods via predefined handcrafted instructions. However, this approach limits the diversity of instructions and hinder the model's reasoning capabilities. Thus, we employ additional complex QAs to enhance the model's reasoning ability. Our results in Figure 4 show that blending these two types of data can lead to a generalized and complex spatial reasoning model.

Template-based Question Answering. These QAs serve as the foundation for learning basic spatial knowledge. We extract information about node attributes such as width and height, as well as relative and metric relations from the edge attributes. We create both qualitative and quantitative templates to generate questions and answers for each type of attribute, using entities in the form of Region [X] . This approach results in examples shown in the first row of Figure 2. We provide detailed templates for each attribute in Appx. F.4.

LLM-based Complex Reasoning Question Answering. We employ Llama3-70B to generate complex spatial reasoning questions to enhance the model's spatial reasoning capabilities. One approach is to input the scene graph directly into the LLMs. However, LLMs struggle to utilize 3D coordinate information effectively [10], so we opt for an alternative approach. We first construct spatial descriptions in a language format. Similar to the template-based approach, we extract attributes from the scene graph and then construct template-based spatial descriptions based on these attributes. We combine the spatial descriptions and the region tags as inputs to the LLM. The LLM is then tasked with creating a complex reasoning question and answer that is based on the description and matches the context. Examples of LLM-generated QAs are shown in the second row of Figure 2. Our LLM prompts for generating QAs are provided in Appx. F.5.

We use our automated annotation pipeline to annotate images from the OpenImages [49] dataset, which covers a wide range of subjects and is of high resolution. The resulting Open Spatial Dataset (OSD) contains 1M unique images and 5M open-vocabulary regions, each associated with a bounding box and segmentation mask. Furthermore, the dataset includes 8M template-based QAs and 700K LLM-based QAs.

3.3 VLMArchitecture

An overview of SpatialRGPT's VLM architecture is shown in Figure 3. SpatialRGPT consists of a visual encoder (Appx. G.1) to encode vision features, a region-feature extractor [19] to obtain regionlevel embeddings (Appx. G.2), linear connectors (Appx. G.3) to project multi-modal embeddings into

Visual Backbone

They are around 45 centimeters apart.

<rgb>

<depth>

What is the distance between     and     ?

<rgb>

<depth>

Region

Masks/Boxes

Region Feature Extractor

RGB Connector

Depth Connector

❄

🔥

🔥

🔥

🔥

the word embedding space, and a large language model using LLaMA2-7B for language processing. In this section, we will explain why and how we incorporate depth information into SpatialRGPT, as well as how SpatialRGPT handles tokenizations.

Plugin Module for Relative-depth Inputs. VLMs that learn solely from RGB pixels are ineffective for 3D perception tasks. Direct learning from 3D data (e.g., point clouds), presents challenges due to issues with scale and diversity. To bridge this gap, we propose using relative depth, which can be obtained through off-the-shelf models [43], to provide additional 3D information alongside RGB as input to our network. Our goal is to elicit geometric reasoning capability through depth guidance. However, this goal is non-trivial. Most VLM's visual encoders are typically only trained with text and 2D images, and simply concatenating RGB and depth features may negatively impact performance. To address this, we introduce an add-on module that seamlessly incorporates the depth information. We use the same image encoder to process the depth map and generate depth feature maps. Then, we employ an additional depth-to-language connector to project the features into the language domain. The depth connector's weights are trained only on spatial-related QAs. This flexible design allows the 2D visual encoder to leverage additional depth representation while still functioning when depth inputs are not presented, thus avoiding the need for a vast amount of training data.

Tokenization and Prompt Format. We generate multi-turn conversation data following [29, 19] for each image and make the image the initial input for the first instruction, providing contextual information. Specifically, we incorporate a prefix prompt: ' <image> \ n ". The <image> is a special token that acts as a placeholder, which would be replaced by the image-level embedding from the vision encoder. When specific mask regions are mentioned in the user input, we use special tokens <region> and <depth> as placeholders. Each region token will be substituted with the corresponding region RGB embedding and depth embedding. All image-level, regionlevel RGB/depth tokens and text tokens are interleaved and fed as the input to the LLM for an auto-regressive generation.

3.4 Training and Inference Paradigm

SpatialRGPT training includes three stages [50]: (i) Connector Feature Alignment, (ii) Visual Language Pre-training, and (iii) Visual Instruction-tuning. During the first stage, CC3M imagecaption pairs are used to pretrain the RGB connector as [29, 51, 52]. In the second stage, the visual language corpus from MMC4 [53] and COYO [54], along with region understanding datasets from [19] and our OSD dataset, are used to pretrain the LLM and connectors (Figure 3). Finally, at stage three, we fine-tune all weights of the VLM on visual language instruction-following datasets, using a combination of the instruction tuning dataset from [29], region-level instruction tuning data [19], and our OSD dataset. Detailed data blend of the visual instruction data is in Appx. H.1. For training region-level data and our OSD, we randomly sample from different modalities (e.g., box, mask) for each sample to ensure the model is versatile to the input modality. At inference time, SpatialRGPT can take both boxes or masks as input. For the results shown in the main paper, if the segmentation is available, we use the mask; if not, we use the box provided and apply SAM to segment the corresponding mask.

4 Experiments

We evaluate the effectiveness of our proposed SpatialRGPT in three aspects: (1) spatial reasoning benchmarks (Section 4.1), (2) standard vision-language benchmarks (Section 4.2), and (3) real-world applications (Section 4.3).

4.1 3D Spatial Reasoning Benchmarks

Currently, there are no visual-language benchmarks that specifically focus on VLM's ability to understand 3D spatial concepts like metric distance or size differences between objects. Recently, SpatialVLM created a spatial reasoning VQA benchmark using human labelers to annotate spatial information on 2D images, but this benchmark is not publicly available. To address this gap, we develop SpatialRGPT-Bench, a spatial reasoning VQA benchmark using data from both urban (nuScenes [58], KITTI [59]) and indoor (SUNRGBD [60], ARKitScenes [61]) environments, as well as simulated scenes (Hypersim [62]). These datasets cover various potential applications and include diverse object types, enhancing our benchmark's thoroughness. We use preprocessed 3D cuboids for each object from Omni3D [63], all positioned within a unified 3D camera coordinate system and categorized by object classes. With these 3D cuboid annotations, we developed a conversational benchmark using our data generation pipeline. This benchmark comprises 657 qualitative and 749 quantitative VQA pairs, covering 88 distinct classes. All the samples come from the validation or test splits of the original datasets and are unseen by SpatialRGPT during the training phase. Please see Appx. E for statistics and examples of SpatialRGPT-Bench.

Can you estimate the age of

Region

[1]? Think step by step.

The height of

Region [1] is 3.77 feet.

Since

the

average

height

of

a

3-year-old

is

around  3.5  feet,  so

Region [1] is likely a 3-year-old.

In Region [1], the individual appears

to be a toddler, possibly around 1 to

3 years old, judging from their size,

posture,  and  the  style  of  walking,

which suggests early childhood.

GPT4-V

How many people do you think can

sit on Region [1]? Explain why.

The

width

of

Region

[1]

is

1.2

meters.  Since  the  average  human

width  is  around  0.5  meters,  two

people can sit on Region [1].

Region [1] highlights a purple sofa

with  a  vibrant  pattern,  capable  of

comfortably  seating  three  people.

This estimation is based on the stan-

dard dimensions of sofas that usual-

the  size  and  arrangement  resemble

what is shown in the image.

GPT4-V

Can you estimate how much time it will take

for the person in Region [1] to reach the loca-

tion  where  Region  [2]  is  now?  Say  if  the

person is Usain Bolt.

The distance between Region [1] and Region

[2] is 10.84 meters. If Usain Bolt can run at a

speed of  10  meters  per  second,  it  will  take

him  1.084  seconds  to  reach  the  location

where Region [2] is now.

Considering that the person is Usain Bolt, we

would  need  some  additional  information

such as the distance between the two regions

and  the  average  speed  Usain  Bolt  could

maintain over that distance in a running set-

ting.

GPT4-V

If you were the person at Region [1],

consider  Region  [2],  Region  [3],  and

Region [4],  which  one  is  standing  at

your 10'oclock direction?

The person in Region [1] will see the

person in Region [2] at 10 o clock.

For the person in Region [1], Region

[3] is in the 10 o'clock direction. This

region is to the upper left from their

perspective, aligning with the general

direction  indicated  by  the  10  o'clock

position on a clock.

GPT4-V

Based on the spatial dimension of Region

[1], what kind of vehicles would not fit in?

The Region [1] is 2.7 meters in height, so

vehicles  taller  than  2.7  meters  would  not

fit in.

In  the  image  you've  shown,  Region  [1]

covers the garage area. To determine what

type of vehicles might not fit in this garage

based on its spatial dimensions, we need

to estimate the size of the garage. From the

image, the garage appears to be a typical

residential garage. Standard single-car ga-

rages usually have a width of about 10 to

12 feet and a depth of about 20 to 24 feet.

GPT4-V

Region2

Region3

Region4

Region1

Region1

egion1

Region1

8

We consider three categories of models as baselines:

Blind LLMs w/ Language Referral. The blind [10] LLM model relies solely on text and generates answers using only the content of the question. To enhance this approach, we prepend the object class to each question. This method serves as a baseline to gauge how much spatial reasoning can be derived from purely existing world knowledge. We choose GPT-4 to represent this baseline, as it is the most advanced model for encapsulating comprehensive world knowledge.

VLMs w/ Language Referral. The setup is similar to the blind LLMs but includes access to visual content, which could allow the model to answer better than a blind LLM. We employ current state-of-the-art VLMs, GPT-4V and LLaVA-v1.6-34B [56], as baselines for this category.

Region-aware VLMs. This category explores models with region-level capabilities similar to our method. The models do not receive any language captions or object class information related to the region of interest; they rely solely on their visual processing capabilities. We equip GPT-4V [55] and LLaVA-v1.6-34B with Set of Marks (SoM) [57] to enable region-referring capabilities. Additionally, we include KOSMOS-2 [24], a VLM capable of taking bounding box inputs to reference objects, and RegionVILA (RegionGPT [19] with VILA [50] pre-training). RegionVILA-7B also serves as an ablation baseline to our method; it shares the same model architecture as our SpatialRGPT-7B ( rgb ) variant but is trained without our specialized spatial VQA dataset.

We use GPT-4 to evaluate the response for each model; please see Appx. J for details. For qualitative QAs, GPT-4 scores the alignment between the model's response and the correct answer as 0 or 1. For quantitative QAs, GPT-4 standardizes numerical values across units into meters; we then calculate accuracy and error metrics. We present the results in Table 1. The upper rows of the table show accuracy (correct vs incorrect or failed to answer) for qualitative QAs. The lower rows report on

Model

↑

)

Acc. (%)

CLIP [64]

58.9

-

RegionCLIP [65]

58.3

-

LLaVA-7B [29]

-

40.0

Shikra-7B [25]

-

53.9

GPT4RoI-7B [31]

-

64.0

PVIT-7B [66]

-

64.5

69.3

-

RegionGPT-7B [19]

70.0

80.6

SpatialRGPT-7B

69.7

79.9

SpatialRGPT-VILA-1.5-3B

72.5

82.5

SpatialRGPT-VILA-1.5-8B

72.9

82.9

Model

Acc. (%)

58.9

Gemini Pro [68]

50.0

Claude 3 OPUS [69]

57.3

GPT-4V-

preview

[55]

58.9

GPT-4V-

Turbo

[55]

66.9

GPT-4o [55]

64.5

InstructBLIP-13B [51]

50.0

Yi-VL-34B [70]

53.2

LLaVA-v1.5-13B-xtuner [71]

54.0

LLaVA-v1.6-34B [56]

64.5

MiniGPT-4-v2-7B [26]

49.2

InstructBLIP-7B [51]

50.8

LLaVA-v1.5-7B-xtuner [71]

50.8

CogVLM-7B [27]

50.8

LLaVA-v1.5-7B [72]

51.6

LLaVA-InternLM2-7B [73]

52.4

SpatialRGPT-7B

82.3

SpatialRGPT-VILA-1.5-8B

87.9

quantitative QAs, detailing their success rate (answers within ± 25 % of the ground truth value) and the absolute relative error [43, 42]. We exclude answers that failed to produce a numerical response from the relative error calculations. The results show that SpatialRGPT significantly outperforms baselines in terms of success rate for qualitative QAs and maintains the lowest error rate for quantitative QAs. Interestingly, we found that blind LLMs and VLMs with language referrals achieved commendable success rates for quantitative QAs, especially for questions related to width and height. This suggests that LLMs can accurately answer specific spatial questions using their extensive world knowledge. Additionally, our SpatialRGPT-7B variant demonstrates improved performance over the SpatialRGPT7B ( rgb ) variant, especially in scenarios where relative depth information can be used to resolve ambiguities, such as distinguishing between behind/front, wide/thin, and estimating distances.

4.2 Public Vision-language Benchmarks

General Benchmarks. In this section, we evaluate whether integrating spatial VQA data and depth information affects performance on other VQA tasks. We compared our models with VILA-1.5-3B, which is trained on the same general VQA datasets. As shown in Table 2, our variants performed similarly to the baselines and slightly better on the VQA-v2 and MMVet datasets. These results align with findings from [17], indicating that VLMs generally underperform on spatial reasoning tasks but can improve with specific spatial VQA training without compromising general VQA performance.

Region & Spatial Benchmarks. We follow the evaluation protocol from RegionGPT [19] and report object classification results using ground-truth boxes on the COCO-2017 validation set. As shown in Table 3, SpatialRGPT outperforms the baselines, demonstrating its strong region cognition capabilities. We further evaluate SpatialRGPT on BLINK [9]'s Relative Depth Benchmark. This benchmark is particularly challenging as it assesses point-level depths, while both the point-level region input and point-level questions were not specifically included in the training of SpatialRGPT. We use bounding boxes to mark the target points and evaluate the test set online with the EvalAI server. As shown in Table 4, SpatialRGPT significantly outperforms the state-of-the-art, achieving over 20% accuracy gain compared to GPT-4V-Turbo. Our model demonstrated strong performance, highlighting its ability to generalize to new tasks without explicit training.

4.3 Real-world Applications

Complex Spatial Reasoning. In this application, we aim to explore whether SpatialRGPT can function as a complex spatial reasoner on its own. Unlike the system mentioned in [17], which uses GPT-4 to handle reasoning tasks and employs VLM solely for answering basic spatial queries, SpatialRGPT directly integrates these capabilities. We provide examples in Figure 4, where we compare SpatialRGPT's responses to those from GPT-4V using real-world samples. Our model demonstrates the ability to address complex spatial questions based on its own spatial knowledge. This suggests that SpatialRGPT has developed a robust representation of spatial learning and that this knowledge has effectively generalized to enhance its intrinsic language reasoning abilities.

What is the object on the table to the

right  of  Region  [0],  and  what  is  its

height and width?

The object on the table to the right of

Region [0] is a coffee maker. It is 19.1

inches  in  height  and  12.4  inches  in

width.

What is the object on the table to the

right of Region [0], and what is its dis-

tance to Region [0]?

The object on the table to the right of

Region [0] is a potted plant, and it is

13.9 inches away from Region [0].

What is the object to the left of Region

[0], and what is its height and width?

The object to the left of Region [0] is a

candle, and it is 10.49 inches in height

and 3.39 inches in width.

What is the object in the basket to the

right of Region [0], and how far is it?

The object in the basket to the right of

Region [0] is a peach, and it is 10.42

inches away.

0

2

4

Distance (inches)

Region0

RegionO

egion

Regiono

20

GT

SRGPT

15

SRGPT_Depth

10

5

0

2

3

4

Frame Index

Multi-hop Reasoning. In Figure 5, we show examples of SpatialRGPT handling multi-hop reasoning. In the upper left sample, the model first identifies what's to the right of Region [0] (a single apple), finds the basket there, determines what's inside the basket, and then provides spatial details about the object inside. Even though our training data doesn't specifically include such multi-hop tasks, SpatialRGPT can still manage them effectively. This indicates that the model has developed a strong understanding of spatial relationships.

Region-aware Dense Reward Annotator. Recently, [17] has shown that VLMs can function as dense reward annotators for robotics tasks by specifying tasks in natural language and having the model annotate rewards for each frame in a trajectory. However, this approach can be constrained by the language's ambiguity, especially when multiple identical objects are present or when targeting a small, specific region in a scene, which can be difficult to describe precisely with language alone. Given that SpatialRGPT is equipped with region-aware capabilities, we can directly specify the regions of interest. To study this application, we conducted a real robot experiment. Specifically, we defined two regions using bounding boxes (one for the fingertip and one for a green cube) and tasked SpatialRGPT to annotate rewards using the distance between the two regions. The results, shown in Figure 6, indicate that the estimated distance between the fingertip and its target cube decreased monotonically as the fingertip moved towards its goal. Also, our depth variant performs slightly better than the RGB variant. This demonstrates SpatialRGPT 's effectiveness as a region-aware dense reward annotator, offering a more precise and efficient alternative to language-only approaches.

5 Discussion

Conclusion. We introduce SpatialRGPT, a novel framework designed to enhance the spatial reasoning capabilities of Vision Language Models (VLMs). By integrating a region representation module and a flexible plugin for depth information, SpatialRGPT allows VLMs to effectively perceive spatial arrangement at both local and global scopes. Our data curation pipeline facilitates the learning of 3D spatial knowledge from scene graphs, while SpatialRGPT-Bench provides a comprehensive benchmark for evaluating spatial cognition across diverse environments. The results demonstrate significant improvements in spatial reasoning tasks while showcasing the model's ability to reason complex spatial relations and perform as dense reward annotators for robotic applications.

Limitations. One limitation of our work is the use of Axis-Aligned Bounding Boxes (AABBs), which can result in inaccuracies in label representation. A more accurate alternative is oriented bounding boxes (OBBs), but implementing them requires precise object pose estimation, which remains challenging due to the lack of open-world solutions. The most accurate approach would be human labeling [74], while this requires significant effort. We leave these for future work.

Acknowledgement. This work was supported, in part, by the Qualcomm Innovation Fellowship.

References

Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In CVPR , 2019. 2

Long Chen, Hanwang Zhang, Jun Xiao, Xiangnan He, Shiliang Pu, and Shih-Fu Chang. Counterfactual critic multi-agent training for scene graph generation. In ICCV , 2019. 2

Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In CVPR , 2020. 2

Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In ICCV , 2023. 2

Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or. Myvlm: Personalizing vlms for user-specific queries. arXiv preprint arXiv:2403.14599 , 2024. 2

Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-vlm: Openvocabulary object detection upon frozen vision and language models. In ICLR , 2023. 2

De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. In ECCV , 2024. 2

Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, et al. Kosmos-2.5: A multimodal literate model. arXiv preprint arXiv:2309.11419 , 2023. 2, 7

Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390 , 2024. 2, 9

Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Sasha Sax, and Aravind Rajeswaran. Openeqa: Embodied question answering in the era of foundation models. In CVPR , 2024. 2, 3, 5, 8

Jianrui Zhang, Mu Cai, Tengyang Xie, and Yong Jae Lee. Countercurate: Enhancing physical and semantic visio-linguistic compositional reasoning via counterfactual examples. In ACL , 2024. 2

Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. In ICRA , 2024. 2

Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872 , 2024. 2

Mikhail Konenkov, Artem Lykov, Daria Trinitatova, and Dzmitry Tsetserukou. Vr-gpt: Visual language model for intelligent virtual reality applications. arXiv preprint arXiv:2405.11537 , 2024. 2

Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In ICRA , 2023. 2

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. In ICML , 2023. 2

Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In CVPR , 2024. 2, 3, 4, 9, 10, 19, 20, 23

Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics. In CoRL , 2024. 2

Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision language model. In CVPR , 2024. 2, 3, 5, 6, 7, 8, 9, 22

Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 3d concept learning and reasoning from multi-view images. In CVPR , 2023. 3

Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. In NeurIPS , 2023. 3

Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In ICRA , 2024. 3, 20

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330 , 2024. 3

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. In ICLR , 2024. 3, 8

Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195 , 2023. 3, 9

Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 , 2023. 3, 9

Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079 , 2023. 3, 9

Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575 , 2023. 3

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS , 2023. 3, 6, 9, 22

Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. In NeurIPS , 2023. 3

Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601 , 2023. 3, 9

Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. In ICLR , 2024. 3, 9

Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In ICLR , 2024. 3

Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, TsuJui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.07973 , 2024. 3, 22

Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In CVPR , 2024. 3

Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In CVPR , 2024. 3

Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. arXiv preprint arXiv:2306.03514 , 2023. 4

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 , 2023. 4

Lei Ke, Mingqiao Ye, Martin Danelljan, Yifan Liu, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Segment anything in high quality. In NeurIPS , 2023. 4

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014. 4, 22

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV , 2023. 4

Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. In CVPR , 2023. 4, 9

Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR , 2024. 4, 6, 9

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRR , 2013. 4

Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV , 2012. 4

Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. arXiv preprint arXiv:2404.15506 , 2024. 4

Shengjie Zhu, Abhinav Kumar, Masa Hu, and Xiaoming Liu. Tame a wild camera: In-the-wild monocular camera calibration. In NeurIPS , 2023. 4

Linyi Jin, Jianming Zhang, Yannick Hold-Geoffroy, Oliver Wang, Kevin Matzen, Matthew Sticha, and David F. Fouhey. Perspective fields for single image camera calibration. In CVPR , 2023. 4

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV , 2020. 5, 19, 25

Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR , 2024. 6, 8, 22, 23

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurIPS , 2023. 6, 9

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML , 2023. 6

Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. In NeurIPS , 2023. 6

Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/ coyo-dataset , 2022. 6

R OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article , 2(5), 2023. 7, 8, 9, 23

Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal capabilities in the wild, 2024. 7, 8, 9

Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441 , 2023. 7, 8

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR , 2020. 7

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR , 2012. 7

Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR , 2015. 7

Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Yuri Feigin, Peter Fu, Thomas Gebauer, Daniel Kurz, Tal Dimry, Brandon Joffe, Arik Schwartz, and Elad Shulman. ARKitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data. In NeurIPS , 2021. 7

Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In ICCV , 2021. 7

Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild. In CVPR , 2023. 7, 18

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021. 9, 22

Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In CVPR , 2022. 9

Chi Chen, Ruoyu Qin, Fuwen Luo, Xiaoyue Mi, Peng Li, Maosong Sun, and Yang Liu. Positionenhanced visual instruction tuning for multimodal large language models. arXiv preprint arXiv:2308.13437 , 2023. 9

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 , 2023. 9

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023. 9

Claude-3-family. https://www.anthropic.com/news/ claude-3-family. 9

Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652 , 2024. 9

XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning llm. https://github. com/InternLM/xtuner , 2023. 9

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR , 2024. 9

Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. In arXiv preprint arXiv:2403.17297 , 2024. 9

Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models. arXiv preprint arXiv:2409.09788 , 2024. 10, 24

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. NeurIPS , 35, 2022. 19

Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 , 2023. 19

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV , 2023. 22

Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich. Discriminability objective for training descriptive captions. In CVPR , 2018. 22

Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR , 2016. 22

Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In ECCV , 2020. 22

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR , 2017. 22

Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. A corpus of natural language for visual reasoning. In NAACL , 2017. 22

Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmrc: Machine reading comprehension on document images. In AAAI , 2021. 22

Desmond Elliott, Stella Frank, Khalil Sima'an, and Lucia Specia. Multi30k: Multilingual english-german image descriptions. In ACL , 2016. 22

Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI , 2019. 22

Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In CVPR , 2021. 22

Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR , 2019. 22

Feng Liu, Tao Xiang, Timothy M Hospedales, Wankou Yang, and Changyin Sun. ivqa: Inverse visual question answering. In CVPR , 2018. 22

Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR , 2019. 22

Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV , 2019. 22

Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, Jose G. Moreno, and Jesús Lovón Melgarejo. Viquae, a dataset for knowledge-based visual question answering about named entities. In ACM SIGIR , 2022. 22

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR , 2017. 22

Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In CVPR , 2017. 22

Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3det: Vast vocabulary visual detection dataset. In ICCV , 2023. 22

Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR , 2019. 22

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV , 2017. 22

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP , 2014. 22

Unsplash. https://unsplash.com/. 25

Appendix Table of Contents

A Ablation Study on Augmented SpatialRGPT-Bench

We conduct additional experiments by augmenting and rephrasing both questions and answers in SpatialRGPT-Bench using GPT-4. The results are shown in Table 5. The results show that SpatialRGPT consistently outperforms the baseline models, even when the questions and answers are different from the training data.

B Ablation Study on Metric-Scale Width and Height Data

We conduct an ablation study to see if adding width and height data affects other types of questions. As shown in Table 6, adding this data slightly improved the accuracy for questions about size (like big/small, tall/short, wide/thin) but slightly worsened the accuracy for questions about the distance between objects (horizontal and vertical). This suggests that information about object size helps with size-related questions but might make distance measurements less clear.

Below / Above

Left / Right

Big / Small

Tall / Short

Wide / Thin

Behind / Front

-

width & height

99.1

99.0

75.8

90.8

82.8

92.1

90.5

+

width & height

99.1

+0

99.0

+0

80.1

+4.3

91.9

+1.1

87.5

91.8

-0.3

90.5

+1.2

Direct Distance

Horizontal Distance

Vertical Distance

Height

Direction

-

width & height 41.2

69.3

54.8

22.8

21.2

95.1

+

width & height 41.2 +0

65.6 -3.7

51.9 -2.9

49.6 +26.8

57.9 +36.7

95.3 +0.2

C Ablation Study on Bounding Box Types

We conduct an ablation study to examine the effect of using axis-aligned bounding boxes (AABB) versus PCA-based oriented bounding boxes (OBB). For this study, we use human-labeled OBBs from the Omni3D test set as the ground truth. We then compare the mean-square error of the width and height measurements for AABBs and PCA-based OBBs labeled by our 3D scene graph pipeline. The results are shown in Table 7. PCA-based OBB often lacks accuracy due to the incomplete and noisy nature of point clouds captured from a single view.

D Ablation Study on Different Input Modalities

As mentioned in Section 3.4, SpatialRGPT can take both boxes and masks as input during the inference phase. In this study, we aimed to test the impact of box and mask inputs on our SpatialRGPT-Bench. We presented the results in Table 8, where we observed a slight drop in performance when using boxes, but in general, the performance was very close. This suggests that the random modality strategy used during training is effective.

E Statistics and Samples of SpatialRGPT-Bench

Figure 7 presents key statistics from our SpatialRGPT-Bench, including counts for QA categories, data sources, and objects. We categorize the QA data into 12 distinct types, evenly divided between relative relationships and metric measurements. Notably, some datasets, such as SUNRGBD, emphasize closeobject scenarios. To reduce bias, we source our data from a diverse range of datasets following [63]. We also show six samples from our SpatialRGPT-Bench in Figure 8.

Direction

Direct

Distance

Height

Horizontal

Distance

Vertical

Distance

148

107

133

133

122

106

Below/

Above

Wide/

Thin

Short

Left/

Right

Behind/

Front

Big /

Small

120

104

112

105

110

106

ARKitScenes

hypersim

KITTI

nuScenes

140

160

120

100

80

60

40

20

F Implementation Details for Data Pipeline

In this section, we aim to provide a detailed implementation of our data annotation pipeline and intermediate results obtained through each component.

How far is......from......horizontally?

and

are 31.21 feet apart horizontally.

0

1

0

1

Could you tell me the vertical size of......?

is 3.67 feet tall.

0

0

How tall is

in terms of height?

is 4.65 inches in height.

0

Does

have a larger size compared to

?

Correct,

is larger in size than

.

0

0

1

1

Is the position of

less distant than

?

No.

0

1

Does

have a lesser width compared to

?

No,

is not thinner than

.

0

1

0

1

0

<regiono>

<regiono>

<regionl>

<regiono>

<regionl>

<regioni>

<regiono>

<regiono>

<region1>

<regiono>

F.1 Filtering.

Recent VLMs often benefit from the broad capabilities gained through training with large-scale 2D image datasets [75, 49]. However, many images in these datasets are unsuitable for developing spatial reasoning QA. For instance, some images may be computer screenshots, paintings, collages, or simply a piece of text. Similar to SpatialVLM [17], we use a CLIP-based open-vocabulary classification model [76] to identify and exclude these unsuitable images. We follow the labeling used in SpatialVLM but have made a few adaptations to better suit the data distribution of the OpenImage [49] dataset. We show the labels we use in Listing 1. With this process, we filtered out 700K samples from the 1.7M OpenImage samples.

F.2 Metric Depth Estimation

As stated in the main paper, we choose Metric3Dv2 as our metric depth estimator. We have observed that Metric3Dv2 and WildCamera's camera intrinsic perform well on images taken in natural environments. In this section, we present the predicted normal maps from the depth model on OpenImages. These normal maps can be viewed as a proxy to estimate the quality of the reconstructed geometry's edges.

F.3 Point Cloud Processing

Here, we detailed how we process the point clouds into scene graphs.

Canonicalization. Our canonicalization method is straightforward. After obtaining the pitch and roll through PerspectiveFields, we transform the point cloud into a canonicalized space using the inverse of the rotation matrix. Figure 10 illustrates the successful alignment of the ground surface with the z-axis angle after canonicalization. This process ensures that the axis-aligned bounding box accurately represents the vertical information of the objects, such as height and vertical distance. Our simple yet effective approach liberates our method from surface segmentation and RANSAC. We have empirically found this procedure robust for most natural images taken by cameras in real-world conditions.

Before Canonicalization

After Canonicalization

Denoising and constructing axis-aligned bounding box. The point clouds obtained from singleview depth may contain noise. Following [17, 22], we carry out several denoising steps based on the approach to filter out outliers and unwanted points, thereby improving the robustness and accuracy of the bounding box. Initially, we eliminate statistical outliers from the object points and then downsample the data to a lower resolution. Subsequently, we use DBSCAN to further remove noise. If the points of an object are fewer than ten after DBSCAN clustering, we exclude that object area. Finally, we employ Open3D to create axis-aligned bounding boxes for each object. The pseudocode for our denoising process is as in Listing 2.

Listing 2: Point cloud denoising steps.

F.4 Open Spatial Dataset QA Templates

We provide samples for each category of QA in the templates that we use to generate QAs mentioned in Section 3.1.

F.5 LLMPrompts for Complex QA

G Implementation Details for SpatialRGPT Architecture

G.1 Visual Backbone.

For SpatialRGPT-7B, we adopt a pre-trained OpenAI CLIP-L model [64] as the visual backbone. As for SpatialRGPT-VILA-1.5-3B/8B, we use 384 × 384 image resolutions with SigLIP [77] to include more visual details for the model, which can help with vision language tasks that require fine-grained details [50] and are beneficial for region-level representations [34].

G.2 Region-feature Extractor.

We adopt the region feature extraction technique in [19]. To begin with, we use a feature refinement module consisting of a 2-layer deconvolution network designed to upscale the original feature map. Then, we employ MaskPooling to extract and average the refined features from the masked area.

G.3 Multi-modal Connector

To bridge representations from various modalities (e.g., image to language, depth to language), we employ a simple linear layer. Following the approach suggested in [50], using a straightforward connector helps the LLM to concentrate more on processing visual inputs, thereby enhancing generalization. We implement two separate connectors, one for image embeddings and another for depth embeddings, to ensure that each modality is handled distinctly. This separation prevents the mixing of modalities, which could otherwise compromise the effectiveness of the model. Note that for SpatialRGPT-VILA-1.5-3B/8B, we follow VILA-1.5 and use a two-layer MLP as our connector.

H Implementation Details for Training SpatialRGPT

H.1 Instruction Tuning Data

Here, we list the instruction tuning data we use in addition to the OSD dataset. Includes general instruction tuning datasets from LLAVA-1.5 [29], LAN-style instructions from VILA [50] (listed in Table 10) and the region-level instruction tuning data from [19] (listed in Table 11) that we use in stage three of the training.

H.2 Hyperparameters

Please refer to VILA's paper on the implementation of the hyperparameters used in the first two stages. In the instruction fine-tuning stage, the maximum learning rate is reduced to 2e-5, and the batch size is adjusted to 16. All other hyperparameters remain the same as in the pre-training stage.

I Experimental Setting and Details

I.1 Experiments Compute Resources

Open Spatial Dataset. Our Open Spatial Dataset uses images from OpenImages, which contains a total of 1.7 million images. Our data preprocessing pipeline was tested on a system with 8 GPUs. The filtering process for 1.7 million images takes 4 hours and results in 1 million samples. The camera calibration and metric depth estimation each took around 4 hours. Note that the depth estimation requires our estimated camera intrinsics as input, so these two processes cannot be parallelized. The open-vocabulary detection and segmentation process takes 8 hours. As the process involves sequential operations, we did not specifically optimize it for parallelization. For LLM-based QA synthesis, we employ LLama3-70b using sglang backend, which takes 12 hours. In general, the total time required to convert OpenImages into 3D scene graphs is within a day, and constructing the QAs takes another half.

SpatialRGPT Training. The first two stages of Spatial RGPT are inherited from VILA [50], which is trained on 16 A100 GPU nodes, with each node having 8 GPUs. The training times for each stage of the 7B model are as follows: connector initialization takes 4 hours, visual language pre-training takes 30 hours. The depth connector is further pre-trained using 2 A100 GPU nodes, taking 4 hours. The final visual instruction-tuning is also experimented on 2 A100 GPU nodes, taking 12 hours.

SpatialRGPT-Bench. The SpatialRGPT-Bench dataset is created from ground truth 3D cuboids and human-annotated labels. Masks only need to be generated when bounding boxes are provided. We use SAM-HQ in our data pipeline to convert the bounding boxes into masks, which takes approximately 4 hours to process 10,000 samples. After this, we synthesize QA and randomly select 1,500 samples. Subsequently, we conduct human verification to filter out incorrect annotations, which takes a day to complete.

J Benchmark Evaluation Details

Our benchmark poses a challenge in evaluation due to the possibility of multiple correct answers in different units. Typically, human trials, like those used by [17], could handle this but are often too slow and costly, mainly as our benchmarks include over a thousand samples. As an alternative, we employ GPT-4 [55] to assess correctness. The evaluation process involves providing a question, the correct answer, and the model's response to the LLM. For qualitative questions, GPT-4 determines if the model's response aligns with the correct answer by assigning a score of 0 or 1. For quantitative questions, GPT-4 extracts numerical values from both the correct answer and the model's response, converting them to the same unit (such as meters). We then measure the accuracy and error of the model's response based on this standardized unit. We provide prompts we use in Table 13 and Table 12.

K More Discussion on Limitations

For the most accurate object detection, oriented bounding boxes (OBB) are preferred over axis-aligned bounding boxes (AABB). As illustrated in Figure 11, the dimensions obtained from AABBs can differ from those obtained with OBBs. There are two methods to compute an OBB. A simple method involves calculating the OBB using Principal Component Analysis (PCA) of the object's convex hull, which provides an approximate minimal bounding box. However, this approximation often lacks accuracy due to the incomplete and noisy nature of point clouds captured from a single view. Furthermore, this method still cannot handle extreme cases when objects are partially elevated (see Appdx C). The most precise method involves determining the OBB based on the object's pose, which is currently challenging due to limitations in obtaining accurate object poses. Future improvements could include integrating available pose estimation approaches. However, currently, there are no open-vocabulary solutions for object pose estimation, so this remains an area for future research. Another direction, explored in subsequent work (e.g., Q-Spatial Bench [74]), addresses this limitation by leveraging human labeling.

Oriented Bounding Box

Axis-aligned Bounding Box

L Broader Impacts

SpatialRGPT serves as a general-purpose visual assistant, similar to other VLMs. It offers potential benefits and risks due to its integration of LLMs. SpatialRGPT shares similar concerns with LLMs, such as output hallucinations, inherited biases from base models, and energy consumption during upscaling. Evaluating SpatialRGPT's performance is also challenging, particularly in accurately measuring the spatial information. This is an area for future enhancement, especially in the field of robotics, which values safety. Despite these challenges, releasing SpatialRGPT to the research community would be beneficial, as it would foster further development and improvement of robotics applications.

M Licenses

The training data we use, OpenImages [49], is released under Apache License 2.0 .

Our paper contain images from Unsplash [98], which is released under Unsplash License , allowing use of photos for free, including for commercial purposes, without attributing the photographer or Unsplash.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction clearly state four main contributions made in the paper. Our experimental results also support our claim.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have a dedicated section to discuss the limitations of our work. Please refer to Appx. K.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [No]

Justification: Our paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have included detailed information on data pipeline implementation (Appx. F, architecture design(Appx. G, and training details(Appx. H.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided instructions on data access and preparation, including how to access the raw, preprocessed, intermediate, and generated data. The data pipeline, data, model weights, and benchmark will be publicly available upon paper publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed experimental setups in Appx. I.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In Figure 6, we plot the std for five runs.

Guidelines: In Figure 6, we plot the standard deviation of our predicted distance using five runs. Note that we did not include error bars in the main table as both the training and evaluation are costly.

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, we report the type of computing resources, memory, and time required to reproduce our experiments in Appx. I.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms in every respect with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have a dedicated section to discuss the societal impacts of our work. Please refer to Appx. L.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our research builds upon existing datasets rather than creating new ones from the internet. So our work does not pose such risks to the best of our knowledge.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets used in our work are commonly used datasets with open access. We have adhered to their licenses and provided citations to give them credit.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: The data pipeline, data, model weights, and benchmark will be publicly available upon paper publication.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our research does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our research does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SpatialRGPT, targeting at enhancing the spatial reasoning abilities of VLMs. The authors introduce a data curation pipeline along with a benchmark that facilitates the learning and evaluation of 3D spatial knowledge. Experiments show that SpatialRGPT thrives at spatial reasoning and performs comparable to SOTA on standard VQA benchmarks. The authors also showcase some real-world applications of such a model by using it for complex spatial reasoning and as a reward annotator for robot manipulation.\n\nSTRENGTHS:\n1. The paper introduces a data curation pipeline that facilitates spatial relation learning, along with a benchmark that's designed specifically for this important task, which will all be made available, making it easy to follow.\n2. This paper introduces a plug-and-play module that process depth information for VLM. Experimental results showcase that the module is useful for cases that involves reasoning among behind/front, wide/thin, and estimating distances.\n3. Results on real robot shows that the learned spatial relation representation can indeed be used for downstream tasks.\n\nWEAKNESSES:\n1. When constructing 3D scene graph, the author mentioned using 3D axis-aligned bounding boxes for objects to compute the width and height. This could lead to inaccurate measure of the object size. It'd be great if the author could show some results on how much does this affect the overall data quality, i.e. how many objects are measured inaccurately because of the AABB assumptions of the bounding boxes.\n2. When discussing the potential real-world application, the author showcased SpatialRGPT can be used as a dense reward annotator. However, the annotation process still needs manually defining regions of interest for every frame in the demonstration video, hindering its use at a broader impact.\n3. The width and height result in Table 1 suggests that SpatialRGPT underperforms the best model, i.e. GPT-4V by 10.5 in success rate, this is a major concern to me since GPT-4V only has commonsense knowledge on the object size and still outperforms SpatialRGPT, which is trained on object size knowledge, by a large margin. Therefore, I'm wondering if the data of object size is useful, i.e. without this portion of data, how would SpatialRGPT perform in the task of reasoning about width and height.\n\nQUESTIONS:\n1. Since Open Spatial Dataset plays an important role in enhancing the ability of spatial reasoning, I'm wondering if the data that targets at different aspect would affect each other, i.e. if I remove the QA pair regarding width and height, will it affect the models performance on answering big/small questions?\n2. Despite the task is targeted at region-based spatial reasoning, I'm wondering how well the model would perform on VQA tasks that includes single region but multi-hop reasoning, i.e. what is the object on the table to the right of <region1>?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper constructs region-aware spatial reasoning QA datasets from existing sources, resulting in the Open Spatial Dataset (OSD). Based on the OSD, they develop a model called SpatialRGPT, which integrates depth information to enable effective representation of regional information and acquisition of spatial knowledge. Experimental results demonstrate the superior spatial reasoning ability of the proposed model and its potential applications in robotics.

STRENGTHS:
1. The proposed dataset OSD is well-crafted with open-vocabulary regions, which should benefit future research by enhancing models’ spatial reasoning abilities.
2. Integrating depth information into the VLM is a novel approach for achieving more accurate spatial reasoning.
3. The paper is well-written and easy to follow.

WEAKNESSES:
1. The SpatialRGPT-Bench is constructed through the proposed data generation pipeline, sharing the same answer formats as the OSD dataset on which the SpatialRGPT is trained. This may introduce bias when directly comparing it with other models not trained on the OSD dataset.
2. There is a lack of clarity regarding the size of SpatialRGPT used in the experiments. In Table 1, it is compared with a 34B model (LLaVA v1.6), while in Table 2, it is compared with 7B models.

QUESTIONS:
1. In Figure 4, the fifth QA example asks, "What kind of vehicles would not fit in?". Is this an example from the SpatialRGPT-Bench, and which category does it belong to? This question involves height, width, and depth of the garage area, which may not align well with the current categories and metrics used for evaluation.

LIMITATIONS:
See Weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach for generating 3D, region-aware annotations from 2D images, transforming scene graphs into spatial QA training data for VLMs using a combination of template-based and LLM approaches. Key contributions include:\n1. A novel pipeline for automatic generation of complex, metric spatial QA data.\n2. A proposed depth adapter to include relative depth maps as input to the VLM.\n3. Benchmarking the generated annotations against state-of-the-art methods, showing improvements in spatial reasoning tasks.\n\nSTRENGTHS:\n- The integration of scene graphs with template-based and LLM-based QA generation, along with the use of a depth map adapter, provides a novel approach to spatial QA VLM training.\n- The experimental results are strong, demonstrating significant improvements over state-of-the-art models like GPT-4V+SoM and Llava-Next.\n- The methodology is well-explained, with clear descriptions of the data collection process and the architecture of the proposed model.\n\nWEAKNESSES:\n- The paper lacks a detailed analysis of how closely the questions in the evaluation set match the templated questions from the data generation pipeline. It is unclear if the formatting provides an unfair advantage to the model, and whether altering the evaluation question formatting affects model performance.\n- The paper does not include a discussion on whether the depth adapter alone, when trained on non-spatial data, improves performance. Additionally, it is unclear why SpatialVLM, a relevant baseline, was not included in the comparisons.\n\nQUESTIONS:\n1. How close are the questions in the evaluation set to the templated questions in the data generation pipeline? What steps are taken to ensure that the question and answer formatting do not give an evaluation advantage? If you change the evaluation question formatting, does it affect the model's performance?\n2. Does the depth adapter addition alone, trained on non-spatial data, provide a performance boost even without the added spatial training data?\n3. Why is SpatialVLM not included as a baseline, given its relevance?\n3. What is the human performance on the evaluation set or a subset, given that humans may be worse than the model at metric spatial reasoning?\n\nLIMITATIONS:\nThe authors discuss limitations in the appendix. Moving limitation discussion to the main paper and addressing the questions outlined above would strengthen the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SpatialRGPT, a framework designed to enhance region-level spatial reasoning in Visual Language Models (VLMs) by incorporating 3D and region-aware visual encoder architecture. The authors present a scalable data pipeline to generate region-aware spatial reasoning questions and answers from existing datasets, resulting in the creation of the Open Spatial Dataset (OSD). To evaluate the model's performance, they introduce SpatialRGPT-Bench, a comprehensive benchmark with ground-truth 3D annotations. The paper demonstrates practical applications of SpatialRGPT, such as serving as a region-aware dense reward annotator for robotics and a stand-alone complex spatial reasoner.\n\nSTRENGTHS:\n- The paper addresses the important problem of enhancing the spatial perception capabilities of multimodal LLMs.\n\n- It creates a large-scale training dataset with millions of examples.\n\n- The paper is well-organized and easy to follow, clearly explaining the authors' motivations at each step.\n\n- The effectiveness of the approach is demonstrated not only in vision-language tasks but also in embodied tasks.\n\nWEAKNESSES:\n- The biggest weakness, in my opinion, is the evaluation. The evaluation using SpatialRGPT-Bench shares the same data creation pipeline as the training data. This means the good performance on SpatialRGPT-Bench might just reflect the model learning the language style of the training data. Using GPT-4 for evaluation further biases the assessment towards responses that include numerical language, as seen in the teaser example: “The height of..1....is 204.54 feet. Assuming each floor is about 10 feet high, the total number of floors would be 20.454. Since you can’t have a fraction of a floor, the total number of floors would be approximately 20.” This type of response, while technically correct, doesn’t align with normal logical thinking. Therefore, a proper evaluation should be conducted on benchmarks like BLINK, especially those related to 3D tasks like depth.\n\n- In Table 2, SpatialRGPT-Depth underperforms the original VILA in 6 out of 8 benchmarks. \n\n- What if we use an off-the-shelf pretrained 3D detector on SpatialRGPT-Bench and then use an LLM to answer questions based on the cuboids from the 3D detector? On other benchmarks like BLINK, is SpatialRGPT better, or is the data curation method of first extracting 3D scene graphs and then using an LLM to summarize better?\n\n- Does stage three of training (Visual Instruction-tuning) require updating all model parameters?\n\n- Does the training data need to overlay all region proposals on the original images, like in Fig. 2, similar to Set-of-Marks?\n\n- The model explanation is unclear, and Figure 3 is confusing. So, the input includes RGB, depth maps, and region proposals (masks or bounding boxes)? Then, a shared visual encoder extracts global features from the RGB image and depth map, and independent connectors project the global RGB/depth feature embeddings into the word embedding space? How many tokens represent RGB and depth, respectively? For region-level features, is each object represented by two tokens, one from the RGB feature and one from the depth feature? Does the Region Feature Extractor take features from the last layer of the visual encoder? Do the RGB and depth features share the same Region Feature Extractor? Figure 3 shows it as shared, but the appendix suggests they are independent.\n\n- Why does the model need to extract region-level tokens separately? Couldn’t the region proposal information be included in the image-level token sequence through visual prompting, like in Set-of-Marks (SOM)? If LLAVA+SOM were trained on OSD, it might also work.\n\n- The model size is not reported—Is it 7B?\n\n- Typos: Line 327 should refer to Fig. 5.\n\nQUESTIONS:\nPlease answer the questions in weakness section.\n\nLIMITATIONS:\nPlease refer to the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-18"}