RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics
Authors: Enshen Zhou
Jingkun An
Cheng Chi
Yi Han
Shanyu Rong
Chi Zhang
Pengwei Wang
Zhongyuan Wang
Tiejun Huang
Lu Sheng
Shanghang Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Spatial Referring
Spatial Reasoning
Vision-Language Model
Robotics
Primary area: reinforcement_learning

Abstract: Spatial referring is a fundamental capability of embodied robots to interact with the 3D physical world. However, even with the powerful pretrained VLMs, recent approaches are still not qualified to accurately understand the complex 3D scenes and dynamically reason about the instruction-indicated locations for interaction. To this end, we propose RoboRefer, a 3D-aware vision language model (VLM) that can first achieve precise spatial understanding by integrating a disentangled but dedicated depth encoder via supervised fine-tuning (SFT). Moreover, RoboRefer advances generalized multi-step spatial reasoning via reinforcement fine-tuning (RFT), with metric-sensitive process reward functions tailored for spatial referring tasks. To support SFT and RFT training, we introduce RefSpatial, a large-scale dataset of 20M QA pairs (2x prior), covering 31 spatial relations (vs. 15 prior) and supporting complex reasoning processes (up to 5 steps). In addition, we introduce RefSpatial-Bench, a challenging benchmark filling the gap in evaluating spatial referring with multi-step reasoning. Experiments show that SFT-trained RoboRefer achieves state-of-the-art spatial understanding, with an average success rate of 89.6%. RFT-trained RoboRefer further outperforms all other baselines by a large margin, even surpassing Gemini-2.5-Pro by 12.4% in average accuracy on RefSpatial-Bench. Notably, RoboRefer can be integrated with various control policies to execute long-horizon, dynamic tasks across diverse robots (e,g., UR5, G1 humanoid) in cluttered real-world scenes.

RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics
Enshen Zhou!.3 Jingkun An'; Cheng Chi Yi Han Shanyu Rong Chi Zhang?
Pengwei Wang?.
Zhongyuan Wang'. Tiejun Huang Lu Sheng' Shanghang Zhang 23t
"Scoool of Software, Beihang University 2 State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University BBeijing Academy of Artificial Intelligence zhouenshen@buaa edu. en anjinnkun028gmal.. com chichengdbaai. ac
1shengebuaa edu. shanghangepku. edu.
Spatially Constrained Instruction
Grasp the apple located in front of logo side of leftmost cup.
farthest sushi on the yellow plate the second left column.
Reasoning Process
Single-stepI: Locate farthest sushi in the second column.
Single-step2: Locate yellow plate on the in this column.
Spatially Constrained Instruction
Carry the apple to the nearest table, aligned with original apple row.
Place it between the plate nearest and the sauce dish.
Reasoning Process
Single-step::
nearest to the observer.
Single-step2: Loate the soy sauce dish.
Single-step3: Identify free space between these two objects.
Multi-Step Spatial Referring with Reasoning
Real-world Manipulation and Navigation
Figure 1: Spatial referring in complex 3D environments demands not only precise single-step spatial understanding but also multi-step spatial reasoning to resolve intricate references step-by-step, thereby enabling efficient control of diverse robots across tasks (e.g., manipulation, navigation).
Abstract
Spatial referring is fundamental capability of embodied robots interact with the 3D physical world. However, even with the powerful pretrained vision lan- guage models (VLMs), recent approaches are still qualified to accurately understand the complex 3D scenes and dynamically reason about the instruction- indicated locations for interaction. To this end, propose RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding by integrating disen- tangled but dedicated depth encoder via supervised fine-tuning (SFT). Moreover, RoboRefer advances generalized multi-step spatial reasoning via reinforcement fine-tuning (RFT), with metric-sensitive process reward functions tailored for spa- tial referring tasks. To support SFT and RFT training, we introduce RefSpatial, large-scale dataset of 20M QA pairs (2x prior), covering 31 spatial relations (vs. 15 prior) and supporting complex reasoning processes (up to 5 steps). In addition, we present CelSpatial-Benchee challenging benchmark filling the gap
Equal contribution
Corresponding author
Project leader
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

in evaluating spatial referring with multi-step reasoning. Experiments show that SFT-trained RoboRefer achieves ofate-of-the-art spatial understanding, with an average success rate of 89.6%. RFTT-rained RoboRefer further outperforms all other baselines by large margin, even surpassing Gemini-2.5-Pro by 17.4% average accuracy on RefSpatial- with various control policies to exccute long-horizon, dynamic tasks across diverse robots (e.g. UR5, GI humanoid) in cluttered real-world scenes. See the project page at
Notably, RoboRefer can be integrated
Introduction
Open-world spatial intelligence is crucial for embodied AI, as robots must understand and reason about 3D scenes to interact effectively in complex environments. As one vital topic in this field, spatial referring, which bridges spatial intelligence and embodied AI by formalizing how agents interpret and act upon spatially constrained instructions, has received increasing attention. Specifically, given sensor observations (e.g., RGB or RGB-D) and a spatially constrained instruction, the spatial referring task aims to predict precise point that satisfies complex spatial relations within the instruction. This predicted point can serve various downstream embodied functions as navigation waypoints, manipulation targets, or placement locations, enabling wide robotic applications, shown in Fig 1.
Spatial referring task comprises two distinct levels of complexity: (1) Single-step spatial under- standing, which forms the foundation of spatial perception by accurately recognizing objects' spatial properties (e.g., position, orientation) and their spatial relations (e.g., distance, direction). This level, where most current research [1-7] concentrates, provides the essential perceptual basis for complex spatial referring. (2) Multi-step spatial reasoning, which transcends basic understanding through compositional reasoning to resolve complex spatial references sequentially. Despite its importance for sophisticated spatial intelligence, this capability remains underexplored. Thus, this work attempts to address this gap by integrating both levels for comprehensive spatial referring. In Fig 1, one must first identify the plate closest to the observer and locate the desired soy sauce dish, then determine the free space between them, which is increasingly challenging more spatial constraints are introduced.
Specifically, existing vision-language models (VLMs) [8-11] ---ssd methods mainly attempt to enhance the first level, i.e., single-step spatial understanding by integrating 3D inputs. However, they either demand costly 3D reconstruction of multi-view images [12, 13]. causing modality gaps or treat depth as RGB- inputs 31, 14] via a shared image encoder, risking modality interference and degrading pretrained image encoders, requiring additional co-training data for compensation. In contrast, the second level, i.e., multi-step spatial referring with reasoning, remains underexplored due to the scareity of suitable datasets, limiting current models' capability and preventing exploration of how single-step understanding might support it. Morcover, current VLMs depend heavily on supervised fine-tuning (SFT) for implicit reasoning, risking memorizing answers over explicit reasoning and thereby hindering generalization and accuracy in open-world spatial referring.
In this work, we propose RoboRefer, a 3D-aware VLM that not only acquires precise spatial under- standing via SFT but also exhibits generalized strong reasoning capabilities for spatial referring via reinforcement fine-tuning (RFT). Specifically, for single-step spatial understanding, RoboRefer em- ploys dedicated depth encoder to enhance precise spatial perception without interfering RGB branch. To enable multi-step spatial reasoning, we design an RFT stage after SFT with explicitly annotated reasoning processes. This stage allows RoboRefer to break down complex spatial referring tasks into sequential analytical steps. In each step. RoboRefer can leverage the spatial understanding gained in SFT and refine the intermediate reasoning precision with our proposed metric-sensitive process reward functions, thus making more accurate point predictions. To our best knowledge, RoboRefer is the first 3D-aware reasoning VLM for multi-step spatial referring with explicit reasoning.
To advance spatial referring, we introduce RefSpatial, a large-scale dataset of 2.5M high-quality examples with 20M QA pairs (2x prior [3]). Leveraging diverse data sources from 2D/3D/Simulation, this dataset can teach a general VLM to achieve spatial referring in a bottom-up manner. Specifically, 2D web images provide fundamental spatial concepts and broad depth perception (indoor and outdoor), 3D embodied videos refine fine-grained spatial understanding of indoor scenes for robotics, and simulated data with ground- ground-truth reasoning processes encourage multi-step spatial referring (up

---

to 5 steps). Notably, RefSpatial includes 31 spatial relations, far exceeding 15 found in previous datasets [2, 31, and each sample contains RGB-D data to support depth aligmment in SFT stage.
We evaluate our SFT-trained model on existing single-step spatial reasoning benchmarks (e.g., CV- Bench [15], BLINK [16]), achieving SOTA performance with average success rate of 89.6%. To address the lack of multi-step spatial referring benchmarks, we introduce RefSpatial-Benee comprising 200 real-world images with manually annotated tasks for object location and placement. Over 70% of the samples require multi-step reasoning (up to 5 steps) and are annotated with precise masks. Our model consistently outperforms all baselines on this benchmark, even surpassing Gemini- 2.5-Pro by an average of 17.4%. Moreover, in Fig. 1 and Sec. 4.4, RoboRefer can execute long- horizon, dynamic tasks in cluttered real-world scenes with various control policies, exhibiting strong generalization across robots (e.g., UR5, G1 humanoid) and tasks (e.g., manipulation, navigation).
Our contributions summarized as follows: (1) We propose RoboRefer, 3D-aware reasoning VLM trained using sequential SFT-RFT strategy with metric-sensitive process reward functions to achieve spatial referring.
(2) We construct RefSpatial, a well-annotated dataset tailored for spatial referring, facilitating both SFT and RFT training, and introduce Re[Spatial-Benchs a benchmark that fills the gap in evaluating spatial referring with multi-step reasoning. (3) Extensive experiments
show that RoboRefer generalizes well, surpasses baselines in spatial understanding and referring with reasoning, and efficiently controls diverse robots across tasks the real world.
2 Related work
Spatial Understanding with VLMs. Spatial understanding [16-24] focuses object-centric
properties (e.g., position, orientation) and inter-object relations (e.g., distance, direction), while spatial reasoning [25-36] draws higher-level inferences over such information. Recent advances in VLMs [8- 11, 37-57] enhance these two abilities via two paradigms: (1) tool-based approaches [7, 14, 58-64]
that integrate VLMS with vision foundation models [65-78] to extract and reason spatial cues and (2) data-driven methods, which fine-tune VLMs using pseudo-3D annotations [1, 6]. real-world
3D datasets [2, 3], or simulated data [4, 79]. However, existing datasets lack multi-step reasoning annotations critical for spatial referring tasks, and a benchmark for evaluating such abilities remains unavailable. We thus introduce a new dataset and benchmark specifically tailored for spatial referring.
Referring with VLMS for Robotics Referring, also known as Referring Expression Comprehension (REC) [80-87], leverages unambiguous descriptions to localize unique region/point in image, and has seen great progress via VLMs [88-93]. Unlike Phrase Localization [94-96] and Generalized Visual Grounding [97-101], which address ambiguous multiple referents, REC focuses one single target-an emphasis crucial for robotics, especially in pick-and-place tasks requiring precise object identification and destination [102-106]. While 2D REC relies object attributes (e.g., color) and image- plane localization (e.g., top right of the image), real-world scenarios for robotics require 3D spatial reasoning to localize (e.g.. "near" vs. "far"). Although efforts [107-109] like RoboPoint [5] incorporate basic spatial cues via images to meet such expectations, they often struggle with complex environments and instructions required for spatial referring. Thus, propose RoboRefer, 3D-aware framework that employs multi-step reasoning to ensure precise spatial referring for robotics.
Reinforcement Fine-tuning for VLMs. Reinforeement Fine- Fine-tuning (RFT) [110-114] is post- training strategy that aligns models with human preferences or specific goals via feedback, comple- menting SFT [115, 116], which adapts prained models using task-oriented data. Recent advances in LLM--based reasoning [114, 117-120] have shifted RL in VLMs toward visual reasoning [121- 1271. grounding [128-130], segmentation [131] and trajectory prediction [132]. However, most methods rely solely on 2D perception, limiting their ability to handle spatial referring tasks that re- quire 3D spatial reasoning. To address this, we propose two-stage training strategy: (1) incorporate depth information during SFT to strengthen spatial understanding; (2) RFT stage then leverages intermediate perception outputs powered by SFT to enable multi-step spatial referring with reasoning.
3 Method
We first formulate the spatial referring task (Sec. 3.1.. Then, we elaborate on RoboRefer, including its architecture and training strategies (Sec. 3.2). Finally, we describe the construction of the ResSpatial dataset (Sec. 3.3) and necessary training details about RoboRefer (Sec. 3.4).
3

---

Stagei-SFT
Stage2-RFF RFT
Task Instruction
Input RGB(D)
Place object between the pen holder and keyboard, lined up with the cup's logo.
The available placement point
Place the object
0.510)
between the peo
/
* 1
holder and keyboard, lined
Vision Encoder
cup's logo.
Projector
LLM
VLM
K Depth Encoder
A Depth Projector
Frozen during Depth Alignment Trainable during Both Full Steps
Answer
<thinloStep1. Stinko/Stpp1..Sop2..... Sttpp...otppeeeee.. Step3..Step4 Step44.</thinie
Input RGB(D)
<answer> (0.341, 616))/answer>>
Reasoning Process
Step1: [Posttion] [pen point 139, 0.628)
Step3: [Position] [cup] point 239, 0.383)
Step5: (0.341, 0.516)
Step2: [Pasition] [keyboard] point (8,586, 0.592)
Step4: [Orientation] [cup's logo direction] Vector (8.312, 0.316, -8. -8.896)
Step1
Step2
Step3
Step4
Output
Figure 2: Overview RoboRefer. RoboRefer can perform single-step precise spatial understanding from RGB(D) imputs with spatially constrained instructions (enabled by the SFT stage introducing depth modality), and multi-step spatial referring with explicit reasoning (powered by the reinforce- ment fine-tuning stage and leveraging spatial understanding within each step learned in SFT).
3.1 Problem Formulation
We formulate spatial referring predicting single 2D point (Ir, y) in image space specify target location or destination, given visual inputs O (e.g., RGB or RGB-D) from the sensor and a textual instruction L. This instruction encodes not only single-step spatial properties such size (e.g., large, small), position (e.g., relative or ordinal location), orientation (e.g. front-facing), and spatial relations (e.g.. distance, direction), but also requires multi-step spatial reasoning. For example, "Place the object between the pen holder and keyboard, lined with the cup's logo." (See Fig. 2) becomes more complex as multiple spatial constraints are combined.
Unlike region-based based 2D referring methods [88, 93, 101], this point-based faasedooommaa formulation more suitable and generalizable for robotics. Compared to the 2D bbox, point naturally map to 3D coordinates via depth, providing accurate spatial anchors. By leveraging predicted points for navigation, grasping, accurately
placement, this formulation enables multi-task learning and execution. Moreover, localize a visible object part occlusion scenarios, while 2D bbox often includes irrelevant objects.
3.2 RoboRefer: A 3D-aware reasoning VLM for spatial referring
VLM Architecture, In Fig. 2, RoboRefer employs separate RGB and depth encoders extract features, which are then aligned via projectors with the L.M for VQA or point prediction. As 3D cues are vital for spatial understanding, 2D VLMs pretrained solely on RGB lack accurate 3D perception. Recent methods [1, 3, 14] avoid explicit 3D representations by treating depth as an image-like modality and sharing the RGB encoder, but this causes modality interference, degrading the pretrained encoder and requiring additional RGB co-training to compensate. To address this, we propose simple yet effective approach: a dedicated depth encoder and projector, initialized from their RGB counterparts. Notably, during joint RGB and RGB-D training, the image encoder remains unaffected by depth input, while the depth encoder updated independently. This design not only avoids modality interference and preserves general VQA performance without extensive RGB-only co-training, but also improves spatial understanding through enhanced perception depth cues (e.g., distance, near-far relations, and perspective-base size variations). See Appx. D.I for details.

---

Supervised Fine-tuning. We adopt NVILA [38] as our base VLM: however, its 2D-only pretraining limits spatial understanding. To address this, we propose two-step SFT. (1) Depth alignment. In Fig. 2, we first train depth projector to align the newly introduced depth space with the textual space, leveraging RGB-D annotations of the RetSpatial (see Sec. 3.3). In this step, only the depth projector is updated. (2) Spatial understanding enhancement. We fine-tune all parameters on the RefSpatial, including single-step fine-grained annotations and multi-step reasoning data with explicit reasoning processes, and additional instruction-folloiing datasets [87, 133, 134]. Therefore, the model is jointly optimized on RGB and RGB-D inputs, with separate updates for the image and depth encoders. This process not only enhances single-step spatial understanding via the new depth modality but also bolsters implicit multi-step reasoning through data with explicit reasoning processes, providing a "cold start" for the subsequent RFT stage. As result, this SFT-trained model exhibits improved capability for multi-step spatial referring tasks. Please check Appx. D.3 for details.
Reinforcement Fine-tuning. Though SFT employs data with precise reasoning, it tends to memorize answers rather than generalize to novel spatial constraints. We thus design a subsequent RFT stage using Group Relative Policy Optimization (GRPO [114]) with multi-step reasoning data from RefSpatial. To guide RFT for more accurate point predictions, we first define two outcome reward (i.e., only care about whether the output answer is correct) functions: (1) Outcome Format Reward (RoF) for structured reasoning and clarity; and (2) Point LI Reward (Rp) granting a score of 1 if the final prediction falls within specific range near the ground-trut point, and 0 otherwise. To enhance intermediate reasoning precision, we exploit key-step perception annotations from RefSpatial and design specialized metric-sensitive process reward functions: (1) Process Format Reward (Rpr). ) enforcing the format "[Perception Type] [TTareet Object]:"; (2) Accuracy Reward (cAc.), which applies to steps included in the key-step perception annotations. For each relevant step, we measure the prediction error using a specific metric, according to the perception type (e.g., L1 distance for positions between ground-truth points and predicted points). Notably, this design order- invariant and does not constrain the reasoning trajectory to a fixed sequence. We sample N responses {ai..... a from the current policy (initialized from the SFT model) encourage exploration. Each response receives combined reward (r, = Ro)(a;) + Ra,) + aRpp + Ac (a;)). where a is set to 0.25, Rewards are normalized within each group to compute relative advantages
(A; = ri mr)mm(((r))) sud(r,)) ), which are then used to update the policy, reinforcing high-quality responses and suppressing suboptimal ones. A -l-deveeeeecee regularization term stabilizes updates by constraining them near the reference policy. Notably, the SFT initialization provides a strong prior, enabling rapid adaptation to output formats and supporting accurate, step-wise spatial reasoning by using the spatial understanding learned from SFT. Fig 2 shows that the RFT-trained model generalizes well to tasks like 4-step spatial referring, progressively handling intricate spatial relations, and yielding precise point predictions. For more details about the RFT training and reward design, please see Appx. D.4.
3.3 RefSpatial dataset
3.3.1 Overview
RefSpatial is a comprehensive dataset integrating 2D images from OpenImages [135], 3D embodied videos from CA-IM [136], and simulated scenes from Infinigen [137] using Objaverse [138] assets (See Fig. 3 (a)). RefSpatial's 's key features are: (1) Fine-Grained Annotations. While prior spatial datasets [2, 3] simplify object reference by limiting each category to single instance per scene, RefSpatial includes multiple objects of the same category. Moreover, each object is annotated with hierarchical captions from broad categories (e.g., "cup"") precise spatial referents (e.g., "the third cup from the left", "the cup closest to the "amera")-enabling unambiguous spatial referring Beyond basic spatial concepts, relations,
in cluttered environments. (2) point coordinates, and point depth predictions, the dataset supports multi-step spatial reasoning by annotating detailed reasoning processes (all simulated data), addressing limitations in existing datasets. (3) High Quality. We rigorously filter data to maintain quality. Retain 466k OpenImages containing rext-referable, spatially relevant objects (down from 1.7M); sample 100k frames from CA-1M with text-identifiable 3D bounding boxes (down from 2M); and manually check and annotate
3k Objaverse-LVIS assets with semantic orientation labels (down from 46k). (4) Large Scale. Comprising 2.5M samples and 20M QA pairs, our dataset spans qualitative VQA, quantitative queries on object attributes/relationss, and point coordinate prediction (Fig. 3(b).. (5) Rich Diversity. RefSpatial spans indoor and outdoor scenes, covers common embodied scenes and integrates 31
5

---

Dataset Statistics
Stutte Data Distibutton
Conkiter Types Diaributian
2D Web Image
Qurllitstive
Embrstee Video
Puro Proketro
Abom
Domm
Smot
ProcssAnsstadio
ny W'Aazsoning
(a)) Distribution
(6). Question Types Distribution
(c). Statics Types Spatial Relations
20 Web image Pipeline
Template CA
Object Location
Detection Seg
GA
Depth
and brown shippigrertaateee cantalere thare which neartes slgnw paler wertectl? cargo stip ar higher erange and brown shippay cantettaree right, althengt latter ray seglilghe imuge perepessiive
Camera Colibration
image
Foundation
Graph
Data Type & Example Data Entry
+30 Embodied Video Pipeline
Template
Reasoning
Object Location
Placement
Q: Pleass point out the free the floor between the chairwith painting above lafimost chair under the table.
A: [(0.359, e. 778)]
Filtered Freme
Top-down
Free Space Annotation
Data Crample Data Entry
Simutation Data Pipeline
Object
Object Location
Object Placement
with Reasoning
Object
Reasoning
Please point out second leftmost frant- prant-facing plarm clock
Regsoning: Step3: (Position lefhmost alarm clock) 199, 0.667). Step2: (Fonhiolthee vecond lefomest alarm clock) (8. (8.510, 0.255.
A:
[[0. (00, 0525)
Annotatee Assett
Scone
Image
Data Type & Example Data Entry
Figure 3: RefSpatial: 2.5M data samples from 2D/3D/Simulated sources, with 31 spatial relations.
distinct spatial relations (See Fig. (c)), fostering precise spatial understanding during SFT. (6) Easy Scalability, Our pipeline seamlessly scales spatial referring data using diverse sources, including 2D images, 3D videos with bounding boxes, and simulation assets. See Appx. for more dataset details.
3.3.2 Data Recipe
In Fig. 3, we present the dataset recipe that progressively integrates 2D, 3D, and simulated data to enable general VLMs to adapt to spatial referring tasks, thereby enhancing bottom-up spatial understanding and reasoning. (1) 2D Web Images aim to endow the model with core spatial concepts and comprehensive depth perception across both indoor and outdoor scenes. mitigate depth scale and category discrepancies between indoor and outdoor scenes, leverage the large-scale, diverse 2D web image dataset, OpenImage [135]. However, directly extracting 3D-aware spatial information challenging. Inspired by prior work [1. 36], transform 2D images into pscudo-3D scene graphs. In detail, after high-quality filtering (from 1.7M to 466K images), we further enhance the data using Qwen2.5-VV V- [11] and heuristic for generating hierarchical region captions, capturing both coarse labels and -reigra spatial references, differentiating our approach from previous methods. We then construct scene graphs via object deteetiion/semmentttion depth estimation, and camera intrinsic estimation, using object captions as nodes and spatial relations as edges. Finally, we generate QA pairs via camplate-based the annotated captions. (2) 3D Embodied Videos want to provide the model with focused spatial understanding of indoor scenes, with a finer- fineregraine perception of spatial relations and concepts. We therefore leverage the richly annotated CA-IM IM [136]. After fine- fine-grained filtering (from 2M 100K frames), construct 3D scene graphs with more diverse spatial relations, enabled by precise 3D bounding boxes compared to 2D approaches. Moreover, we generate top top-down occupancy maps that encode the object positions, orientations, and metric distances (e.g., "10em right of the chair"), enabling accurate spatial referring for placement. (3) Simulation Data arms the model with multi-step referring capabilities with spatial reasoning. While 2D and data enable single-step spatial understanding. they are less scalable for multi-step spatial referring with reasoning. Therefore, we leverage procedurally generate scene layouts [137], using manually verified assets [138] (from 46k to 3k) with semantic orientation annotations [7]. Tasks purposefully designed foster multi-step
I.M-basee approaches, augmented by object-location QA derived from

---

Table 1: Performance on the single-step spatial understanding benchmarks across different model
types. Top-1 Top-2 accuracies are represented using bold text, and underlines.
Method
Input
CV Be-Bench [15]
BLINK [16]
RoboSpatial SAT [4] EmhSpptial [22]
2D-Relation 3D-Depth 3D-Distance 2D-Relation 3D-Depth
Proprictary Models
GPT-4o [8]
RGB
84.62
86.50
83.33
82.52
78.23
77.20
68.67
63.38
Gemini-2.5 2.5-PPo [9]
RGB
93.54
91.00
90.67
91.61
87.90
77.24
70.59
76.67
RGB
74.15
85.83
84.17
74.83
67.74
60.73
40.67
33.33
Open
Langunge Modely
NVILA-2B [38]
RGB
70.15
79.67
60.00
67.83
62.10
51.79
31.33
47.34
NVILA 8B [58]
RGB
91.54
91.83
90.67
76.92
76.61
59.35
63.33
67.72
Owen-2.5-VL-77 [11]
RGB
82.15
60.17
69.00
64.34
60.98
49.59
30,00
40.20
Owen [11]
Spatial Specializt Models
SputialBot 3B [14]
RGB D
69.38
77.33
60.83
67.83
67.74
72.36
63.33
50.66
SputialRGPT:8R
81.12
89.51
SpaceLLaVAA113B [6]
RGB
63.69
66.83
70.17
72.73
62.90
61.00
62.67
49,40
ReboPoint-13B [5]
RGB
75.85
77.83
44.50
60.84
61.29
69.90
46.60
49.31
RoboReler Variants
RohoRefer 2B-SFT
RGB
96.15
95.83
90.67
83,92
88.71
82.93
71.33
70.66
RohoRefer-2R-SFF
RGB-D
96.31
97.17
90.83
87.41
91.13
82.93
82.00
71.10
RoboRcfer-8B-SFT
RGB-D
91.61
92.74
84.55
72.53
Table 2: Performance on current referring and multi-step spatial referring benchmarks. L. and P. denote benchmark's Location and Placement parts: U. indicates unseen compositional spatial relations during SFT/RFT. Top-1 & Top-2 accuracies are represented using bold text, and underlines.
Proprictary Models
Referring Speciallst Models
RoboRefer Variants
Benchmark
Gemini-2.5-Poe [9]
SpaceLLLVA [6]
RoboPoint [5] Molmo-7B [15]
Molmo-728 [15]] 2B-SFT 8B-SFT 2B-RFT
RoboReilt
21.3
49.8
72.8
75.9
74.2
Where2Place
61.9
11.8
46.8
45.0
63.8
66.0
70.0
71.0
RoboSpatial
40.2
16.0
41.3
38.0
40.9
66.4
70.8
71.3
PuSppntal-Bencc--L
46.96
5.82
22.87
21.91
45.77
47.00
52.00
Re!Spatial- Benul--Bechh- PP.
24.21
4.31
9.27
12.85
14.74
48.00
$3.00
54.00
Re/Spatial Bench U.
27.14
4,02
8.40
12.23
21.24
33.77
37.66
41.56
spatial referring and generate corresponding data. We assume that the generated code reflects optimal reasoning, with each line translated into textual form and intermediate results filled into structured formats (e.g., coordinates, distances), as shown Fig. 2, Fig. 3, and Appx. D.4.2, yielding QA pairs with reasoning annotations. For more demonstrations about RefSpatial, please refer to Appx. F.
3.4 Training Details
We adopt NVILA [38] (2B/8B) as the base model and apply SFT to obtain RoboRefer-SFF Due to computational limits, RFT applied only to the model, yielding RoboRefer-RFT. SFT has two steps: the first uses only the RefSpatial; the second trains tuning (1/20 the size RefSpatial [1),,33, 134], and referring datasets[87]. Notably, RefSpatial is reused with both RGB and RGB-D inputs in the second step to enforce the image encoder to learn spatial understanding beyond depth cues. Thus, the model supports both RGB-on -nly and RGB-D inference, with depth optionally inferred via relative depth estimation model [139]. Finally, RFT stage uses the multi-step reasoning data from RefSpatial to train. See Appx. for details.
mixture of RefSpatial, instruction
Experiments
4.1 Single-step Spatial Understanding
We evaluate on public single-step spatial understanding benchmarks. including CV-Bench [15]. the BLINK [16] validation split, RoboSpatial [2] configuration part, SAT [4], and EmbSpatial [22]. Check Appx. E.2 for more evaluation details. The following parts present our analyses.
7

---

Table 3: Performance on general referring benchmarks. Table 4: Performance on general VLM B. and denote Bounding Box and Point. Top-1/2 benchmarks. We also show the advantage of dedicated depth encoder (E. Encoder). We use the same evaluation protocol of CV- Bench/BLINK in Sec. 4.1.
accuracies are indicated by bold/underlined text.
RefCOCO Output val / testA testB val testA testB
RefCOCO+
RefCOCOg val / test
Method
Grounding Specialist Models
Gro
BBox 90.6/ 88.2 88.2 89.0 75.9
86.1/870 /87.0
Benchmark
NVILA-2B Shared F. Deeicated E. (Ours)
RoboRefer-2-SFF
Open-Source Viston-Longuugg Models
Public Vision Language Benchmarks
Qwen2.5-XV 5-VB-22B
BBox / 0 89.7 88. 98.9/9 99.22/2227377 89.9/90.3
MMF
1547
1541
1583
Qwea2.5-VV S-VL722
Point 99.2/ 96.5 93.8 90.4/ 86.7
91.5/92.0
MMBenchaer
78.63
76.23
77.73
Qwen2.5-VI--228
P. 95.4 97.0 94.2 99.5/99.91 88.2
92.5/92.5
OK-VQA
61.93
65.81
POPE
81.96
81.93
83.44
RoboReler Variants
Single-step Spatial Understonding Reachmarks
RohoRefer-2B. SBFT
Point 95.5 96.7 93.8 91 /44.3 /66.3
93,1/9366
RoboRofer-8B--3
Point 96.6 97.7 94.7 91. /8.7
94.3/ 94.1
CV-Bench BLINK
93.99 80.02
94.22
RoboRcler-2BRRFT
Point 95.6 96.6 93.9
92.0 94.2 86.3 /93.63
64.97
85.26
Locate the free area facing direction of the second chair from frant the left side.
free space where new abject be placed to form a parallelogram with three existing objects table.
Identify the free area the table between the mouse and the speaker to the right the monitor.
Unseen
Unseen Reasoning Step
Reasoning Step= 3
Unseen Reasoning Step =
G.P
Ours
G.P
M.M X
R.P
Ours X
G.P x
M.M
M.M X
R.P
Locate the pillow which is closest to the right nightstand.
Point out second object from right to left on the platform with the banana.
Identify the leftmost
white
second shelf from the top white shetf.
Seen Reasoning Step=
Unseen Reasoning Step
Ours
G.P x
Unseen Reasoning Step = 3 G.P
Ours o G.P
M.M
R.P x
Ours
M.M * R.P
M.M
R.P
Figure 4: RefSpatial-Bench results. G.P., M.M., and R.P. donate Gemini-2.5-Prr [9], Molmo- 72B [15], and RoboPoint [5]. RoboRefer-RFT excels in unseen and multi-step cases.
SFT stage enables strong spatial understanding. In Tab 1, trained solely RefSpatial, RoboRefer- SET surpasses all spatial specialist models on these benchmarks, even surpassing Gemini-2.5-PP by 5% (absolute) average. Moreover, our 2B variant outperforms NVILA- AB by 21.7% (absolute).
Depth input improve 3D spatial understanding during inference. Tab. 1, we find that incorpo- rating depth information during inference leads to relative improvements in 3D benchmarks compared to 2D ones by 1.5%, although our model exhibits strong spatial understanding with RGB input alone by reusing the RefSpatial dataset with both RGB and RGB-D inputs during SFT's second step.
4.2 Multi-step Spatial Referring
We first evaluate current robotic referring benchmarks, namely RoboRefIt [140] (location) and Where2Place [5]/RoboSpatial [2] (placement), all limited reasoning steps. To evaluate more complex multi-step spatial referring, we propose RefSpatial-Bench, a challenging benchmark based on real-world cluttered scenes. It contains two subsets, Location and Placement, each with 100 images. Notably. 77 images involve spatial relation combinations unseen in RefSpatial. Over 70% requires multi-step reasoning (up to steps), including precise ground-truth masks. More details about RefSpatial can be found in Appx. For metrics, we report the average success rate of predicted points within the mask. We evaluate RoboRefer using RGB-D inputs by default, with depth maps generated from RGB images via DepthAnyyhing V2 [139]. See Appx. E.3 for more details.
RFT stage fosters better reasoning ability. shown in Tab. 2, 2B-RFT variant outperforms all baselines, exceeding the prior SOTA (Gemini-2.5-Ppp [9]) by 17.4% (absolute) on EitSpatial-Benchh We find that although Gemini-2.5-PP excels in 2D referring (e.g., color, image-space localization). struggles with 3D spatial relations involving distance (e.g., identifying the eecond-farthest object),
8

---

Table 5: Simulation Results
Table 6: Real-world robot evaluation requiring spatial referring.
Method
L.1 L.3 Avg. Time(s)
Success Rate(()) 1 Execution
Manipulation or Navigation tasks with spatial referring
OpenVT.A RehoPPint
Success Rate(%) 1
Ours
Octo
51.2 12.7 0.0 43.2
Pick specifii hamburger closest mug nearest the camera and place in front of the teddy bear.
0.00
0.00
80.00
OpenVLA
51.6 13.1 0.0 43.6
75.3
72.4
Pick the apple front of the leftmost cup's logo side, navigate the nearest table, and place aligned with the apple row.
0.00
0.00
60.00
Ours
81.4 73.1 80.0 79.2
29
Task with Spatial Referring: Pick the hamburger closest to the nearest the camera and place it front of the teddy bear.
mug moved, target changed
teddy bear moved, target target changed
Figure 5: Real-Worll Evaluation. The purple point denotes the current target predicted by our model.
reducing overall performance when multiple spatial constraints are combined. Fig. 4 shows complex multi-step spatial reasoning cases from RefSpatial-Bent -eatihhhlBeehhhhh with model comparisons.
RFT stage provides powerful generalization ability. In the RefSpatiil- Benaall-BBehhheeee Unseen row of Tab. 2, we evaluate novel spatial relation combinations omitted during our SFT/RFT training. The 2B-RFT model exceeds 2B-SFT by 9.1% (absolute), indicating that SFT overfits the training distribution, whereas the RFT model better generalizes by leveraging learned spatial knowledge, consistent with prior findings [141]. Fig. 4 shows results these unseen combinations across different models.
4.3 Public vision-language benchmarks
RefSpatial enhance 2D general referring ability. We also evaluate 2D referring capability on the ReCOCO/+/g [87]. Since our model predicts a single point, we deem a prediction correct the point lies within the ground-trute Souund-truthha bounding box. As this evaluation differs from standard visual grounding protocols, we additionally assess Dwen-2.5VL-22
[11] by using either its predicted point or the center of its predicted box as baselines. In Tab. 3, our method surpasses baselines, indicating that our dataset not only supports 3D spatial referring but also enhances 2D referring performance.
Joint RGB and RGB-D training preserves commonsense knowledge. In Tab. 4, we assess how spatial and depth information influences overall VQA performance by comparing RoboRefer-2B-SBF with the baseline NVILA-2B [38], trained on standard VQA datasets. Our model achieves comparable or slightly superior results, corroborating insights from SpatialVLM [6] and SpatialRGPT [1]. These findings indicate that although VLMs often struggle with spatial reasoning, targeted spatial VQA training, especially with combined RGB and RGB-D data enriched by general visual instruction datasets, can enhance spatial understanding without compromising overall VQA performance.
4.4 Simulator and Real-world Evaluation for Robotics
RoboRefer can be integrated into the system as useful tool. We evaluate our model on the Open6DOR [59] V2 position track, comparing against VLA-based baselines (pretrained Octo [142], LIBERO-finttueed VLe [143]) and SoFar [7], which integrates Florence-2 [93], SAM [78], GPT-4o, and GSNet. RoboRefer serves as lightweight alternative to Florence-2 and GPT-4o for object localization and placement. By using single target point predicted by RoboRefer, the system can generate more accurate masks and corresponding grasp poses than those from 2D boxes under occlusion in cluttered scenes, yielding 6.8% absolute improvement in success rate (Tab. 5). Its compact size also reduces execution time by 27.5% relative to GPT-4o. See Appx. E.4 for details.
Spatial referring from RoboRefer is crucial for real-world robots. In Tab. 6, only our method can handle long-horizon tasks requiring complex multi-step spatial referring in cluttered and dynamic environments. These tasks are challenging, as the robot must precisely identify objects and their placement to satisfy spatial constraints that may change over time. In Fig. 5, integrating RoboRefer with open-loop policy enables rapid updates at 2.5 Hz. Thus, when the mug nearest the camera

---

is moved, the robot adapts by grasping the hamburger closest to the mug's new position and also readjusts placement after the teddy bear's 90º rotation, preserving correct spatial alignment. Notably, spatial referring unifies both manipulation and navigation under single formulation. This allows the G1 humanoid to navigate while performing spatially constrained pick-and-place actions (Fig. 1), thereby enabling more complex, long-horizon tasks. Check Appx. E.5 for more details.
4.5 Ablation Study
Data recipe is critical for SFT training. Ablation results in Tab. 7 reveal that combining 2D, 3D, and simulated data yields optimal performance. As noted in Sec. 3.3, 2D data spans indoor/outdoor scenes, enabling depth learning across scales; its removal severely outdoor-centric BLINK [16]. Mean-
Table 7: Ablation Studies. S.D. means simulated data. P.R. de- notes process reward. We use the same evaluation protocol in Sec. 4.1 and Sec. 4.2.
degrades performance while, 3D data captures embodied indoor environments and miti- gates the Sim2Real gap, benefiting foduoe- CV-Bench [15]. Finally, simulated data broadens spatial diversity. This tripartite data composition is thus key to effective SFT training.
Data Recipe Depth |Spatial Understanding 2D 3D S.D. Encooder[CV-Bond Bench BLINK,..
SFT Variants (28)
84.17
74.48
Dedicated depth encoder preserves image understanding. We compare dedicated and shared image-depth encoders during SFT. In Tab. 4, the dedicated encoder better maintains image understand- ing under limited RGB-only data (1/20 RefSpatial QA), while the shared encoder harms general performance. Though prior work[ ] adopts shared encoder, it (1) requires over twice as much RGB- only data compared to spatial-relate data for co-training; (2) tar- gets region-level depths, differing from our full-image approach.
X
81.83
74.61
75.10
91.24
85.27
94.77
89.27
Reward P.R.
Depth
Spatial Referring
Encoder
RefSpatia--Bocch
RFT Variants (28)
40.00
48.00
Depth encoder improves both spatial understanding and rea-
$3.00
soning. Recent VLMs [3, 107] show that large-scale spatial train-
ing enables implicit 3D understanding (e.g., depth, distance, 3D boxes) from images alone. To assess this, we fine-tune NVILA-2B [38] on RefSpatial without the depth encoder, followed by continued RFT. Results indicate that depth improves single-step spatial understanding, consistent with MM-Spatial [3], and yields greater gains in multi-step spatial referring. We attribute this to: (1) the need for precise coordinate prediction in spatial referring, unlike VQA's multiple-choice; (2) cumulative reasoning across steps, amplifying the utility of depth cues.
Process reward advances the accuracy of intermediate perception. Tab 7 shows a 5-point improvement with process reward, which leverages key step annotations from RefSpatial to refine step-wise perception, thereby predicting more accurate points with complex spatial relations.
Conclusion and Future work
In this paper, we introduce RoboRefer, a novel 3D-aware VLM that addresses spatial referring through the combination of both single-step accurate understanding and multi-step spatial reasoning. In detail, we enhance 3D perception with a separate depth encoder via SFT, and enable generalized multi-step spatial referring via RFT with our proposed metric-sensitive process reward functions. We also present RefSpatial, a large-scale, well-designed dataset for SFT and RFT training, with RefSpatial- Bench, benchmark tailored to evaluate spatial referring. Extensive experiments show the effectiveness of RoboRefer and highlight its potential for a broad range of robotic applications.
Our future work will focus on two main directions. (1) Enhancing the model's understanding of human priors and intent. As discussed in Appx. G. human instructions are often brief and ambiguous, even when the correct location is unique. Potential solutions include exploring procedural synthesis of intent-aware data or improving model performance through co-training with intent-rich datasets. (2) Improving the model's 's 3D perception capabilities. Our current models predominantly rely on qualitative spatial relations (e.g., left, right) and predict 2D image-plane coordinates, necessitating depth- conversion to 3D, as discussed in Appx. A. Future directions include directly modeling quantitative geometry to enable precise 3D reasoning or direct prediction of 3D points and visual traces, which are more challenging if combined with spatially constrained instructions.
10

---

Acknowledgement
This work was supported by the National Natural Science Foundation of China (62132001, 6247601 I), Capital's Funds for Health Improvement and Research (CFH 2024-2-40611), and the Fundamental Research Funds for the Central Universities. We sincerely thank Jiayuan Zhang, Jiawei He for their valuable discussions and insightful feedback about the method. We also sincerely appreciate Yusu Deng's s excellent figure design (e.g., teaser figure, pipeline overview) and demo reviewing work.
References
[1] An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language models. NIPS, 2024.
[2] Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d vision- laniugla models for robotics. CVPR, 2025.
[3] Erik Daxberger, Nina Wenzel, David Griffiths, Haiming Gang, Justin Lazarow, Gefen Kohavi, Kai Kang, Marcin Eichner. Yinfei Yang. Afshin Dehghan, al. Mm-spatial: Exploring 3d spatial understanding in multimodal 1ls. arXiv, 2025.
[4] Arijit Ray, Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A Plummer, Ranjay Krishna, Kuo-Hao Zeng, et al. Sat Spatial aptitude training for multimodal language models. arXiv, 2024.
[5] Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vasion-language model for spatial affordance prediction for robotics. arXiv, 2024.
[6] Boyuan Chen, Zhuo Xu, Sean Kinmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision- lision-language models with spatial reasoning capabilities. In CVPR, 2024.
[7] Zekun Qi, Wenyao Zhang, Yufei Ding, Runpei Dong, Xinqiang Yu, Jingwen Li, Lingyun Xu, Baoyu Li, Xialin He, Guofan Fan, Sofar: Language-grounde orientation bridges spatial reasoning and object manipulation. arXiv,, 2025.
[8] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv, 2023.
[9] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv, 2023.
[10] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jac Sung Park. Mohammadreza Salchi, Niklas Muennighoff, Kyle Lo. Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-arr multimodal models, arXiv, 2024.
[11] Shuai Bai, Kegin Chen, Xuejing Liu, Jialin Wang, Wenbin Gc, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv, 2025.
[12] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. NIPS, 2023.
[13] Yining Hong, Chunru Lin, Yilun Du Zhenfang Chen, Joshua Tenenbaum, and Chuang Gan.
3d concept learning and reasoning from multi-view images. In CVPR, 2023.
[14] Wenxiao Cai, laroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang. Hao Dong, and
Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. ICRA, 2025.
[15] Peter Tong, Ellis Brown. Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal lms. NIPS, 2024.
[16] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In ECCV, 2024.

---

[17] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In CVPR, 2022.
[18] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
[19] Baoxiong Jia, Yixin Chen, Huangyue Yu. Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. ECCV, 2024.
[20] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnckk and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
[21] Emilia Szymanska, Mihai Dusmanu, Jan- Willem Buurlage, Mahdi Rad, and Mare Pollefeys. Space3d-bench: Spatial 3d question answering benchmark. In ECCV 2024 Workshop, 2024.
[22] Mengfei Du, Binhao Wu, Zejun Li, Xuan-Jing Huang, and Zhongyu Wei. Embspatial--ende Benchmarking spatial understanding for embodied tasks with large vision-language models. In ACL. (Volume 2: Short Papers). 2024.
[23] Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Kraehenbuehl, and Vladlen Koltun. Does spatial cognition emerge in frontier models? ICLR, 2025.
[24] Enshen Zhou, Cheng Chi, Yibo Li, Jingkun An, Jiayuan Zhang, Shanyu Rong, Yi Han, Yuheng
Ji, Mengzhen Liu, Pengwei Wang, et al. Robotracer: Mastering spatial trace with reasoning in vision-language models for robotics. arXiv preprint arXiv:25 /2. / /3660, 2025.
[25] Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's" up" with vision-language models? investigating their struggle with spatial reasoning. In EMNLP, 2023.
[26] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 2023.
[27] Navid Rajabi and Jana Kosecka. Towards grounded visual spatial reasoning in multi-modal vision language models. arXiv, 2023.
[28] Kanchana Ranasinghe, Satya Narayan Shukla, Omid Poursaeed, Michael S Ryoo, and Tsung- Yu Lin. Learning localize objects improves spatial reasoning visual-Ilms. In CVPR, pages 12977-12987, 2024.
[29] Fatemeh Shiri, Xiao-Yu Yu Guo, Mona Far, Xin Yu, Reza Haf, and Yuan-Fang Li. An empirical analysis on spatial reasoning capabilities of large multimodal models. In EMNLP, 2024.
[30] Phillip Y Lee, Jihyeon Je, Chanho Park, Mikaela Angelina Uy, Leonidas Guibas, and Minhyuk Sung. Perspective-aware reasoning in vision-language models via mental imagery simulation. arXiv, 2025.
[31] Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is picture worth thousand words? delving into spatial reasoning for vision language models. NIPS, 2024.
[32] Yihong Tang, Ao Qu, Zhaokai Wang, Dingyi Zhuang, Zhaofeng Wu, Wei Ma, Shenhao Wang. Yunhan Zheng, Zhan Zhao, and Jinhua Zhao. Sparkle: Mastering basic spatial capabilities in vision language models elicits generalization to composite spatial reasoning. arXiv, 2024.
[33] Yuecheng Liu, Dafeng Chi, Shiguang Wu, Zhanguang Zhang, Yaochen Hu, Lingfeng Zhang, Yingxue Zhang, Shuang Wu, Tongtong Cao, Guowei Huang, et al. Spatialcot: Advancing spa- tial reasoning through coordinate alignment and chain-of ofain-of-thought for embodied task planning. arXiv, 2025.
[34] Yang Liu, Ming Ma, Xiaomin Yu, Pengxiang Ding, Han Zhao, Mingyang Sun, Siteng Huang. and Donglin Wang. Ssr: Enhancing depth perception in vision-language models via rationale- guided spatial reasoning. arXiv preprint arXiv:2505. 12448, 2025.
[35] Wufei Ma, Luoxin Ye, Celso de Melo, Alan L Yuille, and Jieneng Chen. Spatiallim: A compound 3d-informed design towards spatially-intelligeett large multimodal models. In CVPR, 2025.
[36] Wufei Ma, Yu-Cheng Chou, Qihao Liu, Xingrui Wang, Celso de Melo, Jieneng Chen, Jianwen Xie, and Alan Yuille. Spatialreasoner: Towards explicit and generalizable 3d spatial reasoning. arXiv, 2025.
12

---

[37] AI Anthropic. The claude 3 model family: Opus. sonnet, haiku. 3laude-3 Model Card, 1:1, 2024.
[38]
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang. Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv, 2024.
[39] Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, et al. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. NIPS, 2023.
[40] Yiran Qin, Enshen Zhou, Qichang Liu, Zhenfei Yin, Lu Sheng, Ruimao Zhang, Yu Qiao, and Jing Shao. Mp5: multi-modal open-ended embodied system in minecraft via active perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16307-16316, 2024.
[41] Yiran Qin. Ao Sun, Yuze Hong, Benyou Wang. and Ruimao Zhang, Navigatediff: Visual predictors are zero-shot navigation assistants. arXiv preprint arXiv:2502.138% 2025.
[42] Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang. Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, et al. Worldsimbench: Towards video generation models as world simulators. arXiv preprint arXiv:2410. 18072, 2024.
[43] Zeren Chen, Zhelun Shi, Xiaoya Lu, Lehan He, Sucheng Qian, Zhenfei Yin, Wanli Ouyang, Jing Shao, Yu Qiao, Cewu Lu, al. Rh20t-p: A primitive-level robotic dataset towards composable generalization agents. arXiv preprini arXiv:2403. 19622, 2024.
[44] Enshen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang, Ruimao Zhang, Lu Sheng. Yu Qiao, and Jing Shao. Minedreamer: Learning to follow instructions via chain-of-imagination for simulated-world control. arXiv preprint arXiv:2403. /2037, 2024.
[45] Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang. Mengdi Zhao, Yao Mu, Pengju An, et al. Robobrain: A unified brain model for robotic manipulation from abstract to concrete. arXiv preprint arXiv:2502.2
2025.
[46] Yuzhou Huang. Yiran Qin, Shunlin Lu, Xintao Wang, Rui Huang, Ying Shan, and Ruimao Zhang. Story3d-agent: Exploring 3d storytelling visualization with large language models. arXiv preprint arXiv:2408./ 1801, 2024.
[47] Lijun Li. Zhelun Shi, Xuhao Hu, Bowen Dong, Yiran Qin, Xihui Liu, Lu Sheng, and Jing Shao. T2isafety: Benchmark for assessing fairness, toxicity, and privacy in image generation. arXiv preprint arXiv:2501.12612 2025.
[48] Jinguo Zhu, Weiyun Wang. Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv, 2025.
[49]
Jiazhao Zhang, Kunyu Wang. Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-based vlm plans the next step for vision-and siion-and-langeee language navigation. Robotics: Science and Systems, 2024.
[50] Jiazhao Zhang, Kunyu Wang, Shaoan Wang, Minghan Li, Haoran Liu, Songlin Wei, Zhongyuan Wang, Zhizheng Zhang, and He Wang. Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks, 2024.
[51] Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Lily Lee, Kaichen Zhou, Pengju An, Senqiao Yang, Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Multimodal state space model for efficient robot reasoning and manipulation. arXiv e-prints, pages arXiv-2406. 2024.
[52] BAAI RoboBrain Team, Mingyu Cao, Huajie Tan, Yuheng Ji, Xiansheng Chen, Minglan Lin, Zhiyu Li, Zhou Cao, Pengwei Wang, Enshen Zhou, et al. Robobrain 2.0 technical report. arXiv preprint arXiv:2507.0200
2025.
[53] Yuheng Ji, Huajie Tan, Cheng Chi, Yijie Xu, Yuting Zhao, Enshen Zhou, Huaihai Lyu, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang, et al. Mathsticks: A benchmark for visual symbolic compositional reasoning with matchstick puzzles. arXiv preprint arXiv:2510.004:
2025.
13

---

[54] Qizhe Zhang, Mengzhen Liu, Lichen Li, Ming Lu, Yuan Zhang, Junwen Pan, Qi She, and Shanghang Zhang. Beyond attention or similarity: Maximizing conditional diversity for token pruning in milms. arXiv preprini arXiv:2506. 10967, 2025.
[55] Rui Li, Zixuan Hu, Wenxi Qu, Jinouwen Zhang, Zhenfei Yin, Sha Zhang, Xuantuo Huang, Hanqing Wang. Tai Wang, Jiangmiao Pang, et al. Labutopia: High-fidelity simulation and hierarchical benchmark for scientific embodied agents. arXiv preprint arXiv:2505.22634, 2025.
[56] Shaoan Wang, Jiazhao Zhang, Minghan Li, Jiahang Liu, Angi Li, Kui Wu, Fangwei Zhong, Junzhi Yu. Zhizheng Zhang, and He Wang. Trackvla: Embodied visual tracking the wild. arXiv preprint arXiv:2505. 2025.
[57] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 2023.
[58] Chenyang Ma, Kai Lu, Ta- Yang Cheng, Niki Trigoni, and Andrew Markham. Spatialpin: Enhancing spatial reasoning capabilities of vision- vision-language models through prompting and interacting 3d priors. In NIPS, 2024.
[59] Yufei Ding, Haoran Geng, Chaoyi Xu, Xiaomeng Fang. Jiazhao Zhang, Songlin Wei, Qiyu Dai, Zhizheng Zhang, and He Wang. Openódor: Benchmarking open-instruction 6-dof object rearrangement and vlm-based approach. In IROS, 2024.
[60] Yiran Qin, Li Kang. Xiufeng Song, Zhenfei Yin, Xiaohong Liu, Xihui Liu, Ruimao Zhang, and Lei Bai. Robofactory: Exploring embodied agent collaboration with compositional constraints. arXiv preprint arXiv:2503. 16408, 2025.
[61] Kefei Zhu, Fengshuo Bai, YuanHao Xiang, Yishuai Cai, Xinglin Chen, Ruochong Li, Xing- tao Wang, Hao Dong, Yaodong Yang, Xiaopeng Fan, et al. Dexflywheel: A scalable and self- self-improving data generation framework for dexterous manipulation. arXiv preprint Stti::2500.238299 2025.
[62] Huajie Tan, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Yaoxu Lyu, Mingyu Cao, Zhongyuan Wang, and Shanghang Zhang. Roboos: A hierarchical embodied framework for cross- embodiment and multi-agent collaboration. arXiv preprint arXiv:2505.08 03673, 2025.
[63] Enshen Zhou, Qi Su, Cheng Chi, Zhizheng Zhang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, and He Wang. Code-as-monitor: Constraint-aware visual programming for reactive and proactive robotic failure detection. arXiv preprint arXiv:2412.044 2244455, 2024.
[64] Yi Han, Cheng Chi, Enshen Zhou, Shanyu Rong, Jingkun An, Pengwei Wang, Zhongyuan Wang, Lu Sheng, and Shanghang Zhang. Tiger: Tool-integrated geometric reasoning in vision-language models for robotics. arXiv preprint arXiv:25/0.0718// 181, 2025.
[65] Linyi Jin, Jianming Zhang, Yannick Hold-Geoffrry, Oliver Wang, Kevin Blackbumn-Matzen Matthew Sticha, and David F Fouhey. Perspective fields for single image camera calibration. In CVPR, 2023.
[66] Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin. Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. In CVPR, 2024.
[67] Mengzhen Liu, Mengyu Wang. Henghui Ding, Yilong Xu, Yao Zhao, and Yunchao Wei. Segment anything with precise interaction. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 3790-3799, 2024.
[68] Luigi Piccinelli, Christos Sakaridis, Yung-Hsu Yang, Mattia Segu, Siyuan Li, Wim Abbeloos, and Luc Van Gool. Unidepthy2: Universal monocular metric depth estimation made simpler. arXiv, 2025.
[69] Aleksei Bochkovskii, AmaĀGI Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second. arXiv, 2024.
[70] Chengbo Yuan, Rui Zhou, Mengzhen Liu, Yingdong Hu, Shengjie Wang, Li Yi, Chuan Wen, Shanghang Zhang, and Yang Gao. Motiontrans: Human vr data enable motion-level learning for robotic manipulation policies. arXiv preprint 2509.:25009177999 2025.
14

---

[71] Zhe Li, Weihao Yuan, Yisheng He, Lingteng Qiu, Shenhao Zhu, Xiaodong Gu, Weichao Shen, Yuan Dong, Zilong Dong, and Laurence T Yang. Lamp: Language- motion pretraining for motion generation, retrieval, and captioning. arXiv preprint arXiv:2410.070933 2024.
[72] Zhe Li, Yisheng He. Lei Zhong, Weichao Shen, Qi Zuo, Lingteng Qiu. Zilong Dong, Lau- rence Tianruo Yang, and Weihao Yuan. Mulsmo: Multimodal stylized motion generation by bidirectional control flow. arXiv preprint arXiv:24/2.09901, 2024.
[73] Zhe Li. Cheng Chi, Yangyang Wei, Boan Zhu, Yibo Peng, Tao Huang, Pengwei Wang. Zhongyuan Wang, Shanghang Zhang, and Chang Xu. From language loco- motion: Retargeting-free humanoid control via motion latent guidance. arXiv preprint arXiv:2510. /4952, 2025.
[74] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Nacem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding. localization, and dense features. arXiv, 2025.
[75] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In CVPR, 2023.
[76] Shengjie Zhu, Abhinav Kumar, Masa Hu, and Xiaoming Liu, Tame wild camera: in-the-wild monocular camera calibration. NIPS, 2023.
[77] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, 2023.
[78] Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloc Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. ICI.R, 2025.
[79] Xingrui Wang, Wufei Ma, Tiezheng Zhang, Celso M de Melo, Jieneng Chen, and Alan Yuille. Pulsecheck457: A diagnostic benchmark for 6d spatial reasoning of large multimodal models. CVPR, 2025.
[80] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016.
[81] Kees van Deemter, Ielka van der Sluis, and Albert Gatt. Building semantically transparent corpus for the generation referring expressions. In CVPR, 2006.
[82] Dave Golland, Percy Liang, and Dan Klein. game-theoretic approach generating spatial descriptions. In EMNLP, 2010.
[83] Margaret Mitchell, Kees van Deemter, and Ehud Reiter. Natural reference to objects in visual domain. In Proceedings of the 6th international natural language generation conference, 2010.
[84] Margaret Mitchell, Kees Van Deemter, and Ehud Reiter. Generating expressions that refer to visible objects. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2013.
[85] Nicholas FitzGerald, Yoav Artzi, and Luke Zettlemoyer. Learning distributions over logical forms for referring expression generation. In EMNLP, 2013.
[86] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014.
[87] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, 2016.
[88] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. ICLR, 2024.
[89] Zhiliang Peng. Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv, 2023.
[90] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal 11m's 's referential dialogue magic. arXiv, 2023.
15

---

[91] Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang. Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-modal lims. arXiv, 2023.
[92] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open- open-ended decoder for vision-centric tasks. NIPS, 2023.
[93] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In CVPR. 2024.
[94] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-senteen models. In ICCV, 2015.
[95] Mingzhe Wang, Mahmoud Azab, Noriyuki Kojima, Rada Mihalcea, and Jia Deng. Structured matching for phrase localization. In ECCV, 2016.
[96] Bryan A Plummer, Arun Mallya, Christopher M Cervantes, Julia Hockenmaier, and Svetlana Lazebnik. Phrase localization and visual relationship detection with comprehensive image- language cues. CVPR, 2017.
[97] Chi Xie, Zhao Zhang, Yixuan Wu, Feng Zhu, Rui Zhao, and Shuang Liang. Described object detection: Liberating object detection with flexible expressions. NIPS, 2023.
[98] Shuting He, Henghui Ding, Chang Liu, and Xudong Jiang. expression comprehension. arXiv, 2023.
Gree: Generalized referring
[99] Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression
segmentation. In CVPR, 2023.
[100] Yutao Hu, Qixiong Wang, Wenqi Shao, Enze Xie, Zhenguo Li, Jungong Han, and Ping Luo. Beyond one-to-one: Rethinking the referring image segmentation. In CVPR, 2023.
[101] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang, Chun- yuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, 2024.
[102] Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In CoRL, 2021.
[103] Weiyu Liu, Chris Paxton, Tucker Hermans, and Dieter Fox. Structformer: Learning spatial structure for language-guidee semantic rearrangement of novel objects. In ICRA, 2022.
[104] Weiyu Liu, Yilun Du, Tucker Hermans, Sonia Chernova, and Chris Paxton. Structdiffusion: Language-guided creation of physically-valid structures using unseen objects. In RSS, 2023.
[105] Wentao Yuan, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. M212: Multi-task
masked transformer for object-centric pick and place. arXiv, 2023.
[106] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In CoRI., 2022.
[107] Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean- Baptiste Alayrac, Montser- rat Gonzalez Arenas, Travis Armstrong. Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv, 2025.
[108] Soroush Nasiriany, Fei Xia. Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. In ICML, 2024.
[109] Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, al. Magma: A foundation model for multimodal ai agents. arXiv, 2025.
[110] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv, 2022.
[111] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv, 2022.
16

---

[112] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly reward model. NIPS, 2023.
[113] Jingkun An, Yinghao Zhu, Zongjian Li, Enshen Zhou, Haoran Feng, Xijie Huang, Bohua Chen, Yemin Shi, and Chengwei Pan. Agfsync: Leveraging ai- ti-generated feedback for preference optimization in text-to-image generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 1746-1754, 2025.
[114] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv, 2024.
[115] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv, 2021.
[116] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. NIPS, 2023.
[117] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv, 2024.
[118] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv, 2025.
[119] Guibin Zhang. Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, et al. The landscape of agentic reinforcement learning for Imms: survey, arXiv preprint arXiv:2509. N00::2509.02547, 2025.
[120] Heng Zhou, Hejia Geng, Xiangyuan Xue, Li Kang, Yiran Qin, Zhiyong Wang, Zhenfei Yin, and Lei Bai. Reso: A reward-driven orga-organ lim-based multi-agent system for reasoning tasks. arXiv preprint arXhh:2503.023990 2025.
[121] Yi Yang. Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng. Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. RI-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv, 2025.
[122] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv,, 2025.
[123] Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al. Mm-rlhf: The next step forward in multimodal I1m alignment. arXiv, 2025.
[124] Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang. Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv, 2025.
[125] Li Kang, Xiufeng Song, Heng Zhou, Yiran Qin, Jie Yang. Xiaohong Liu, Philip Torr, Lei Bai, and Zhenfei Yin. Viki-r: Coordinating embodied multi-agent cooperation via reinforcement learning. arXiv preprint arXiv:2506. 09049, 2025.
[126] Huajie Tan, Sixiang Chen, Yijie Xu, Zixiao Wang, Yuheng Ji, Cheng Chi, Yaoxu Lyu, Zhongxia Zhao, Xiansheng Chen, Peterson Co, et al. Robo-dopamine: General process reward modeling for high-precision robotic manipulation. arXiv preprint arXiv:2512.23703: 2025.
[127] Jiayua Zhang, Kaiquan Chen, Zhihao Lu, Enshen Zhou, Qian Yu, and Jing Zhang. Prune4web:
Dom tree pruning programming for web agent. arXiv preprint arXiv:2511.21398 2025.
[128) Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv, 2025.
[129] Yufei Zhan, Yousong Zhu, Shurong Zheng, Hongyin Zhao, Fan Yang, Ming Tang, and Jinqiao Wang. Vision-rl: Evolving human free alignment in large vision- languaee models via vision- guided reinforcement learning. arXiv, 2025.
[130] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable rl-style large vision- languaee model. arXiv preprint arXiv:2504.076155 2025.
17

---

[131] Yuqi Liu, Bohao Peng. Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-zero Reasoning-chain guided segmentation via cognitive reinforcement. arXiv, 2025.
[132] Zirui Song, Guangxian Ouyang, Mingzhe Li, Yuheng Ji, Chenxi Wang, Zixiang Xu, Zeyu Zhang, Xiaoqing Zhang, Qian Jiang, Zhenhao Chen, et al. Maniplvm-rl: Reinforcement learning for reasoning in embodied manipulation with large vision-language models. arXiv preprint arXiv:2505. 16517, 2025.
[133] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning, In ICLR, 2024,
[134] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024.
[135] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJC,, 2020.
[136] Justin Lazarow, David Griffiths, Gefen Kohavi, Francisco Crespo, and Afshin Dehghan. Cubify Scaling indoor 3d object detection. arXiv preprint 2421v:2412.04458, 2024.
anything:
[137] Alexander Raistrick, Lingjie Mei, Karhan Kayan, David Yan, Yiming Zuo, Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexandropoulos, Lahav Lipson, et al. Infinigen Photorcalistic indoor scenes using procedural generation. In CVPR, 2024.
indoors:
[138] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, 2023.
[139] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Heng- shuang Zhao. Depth anything v2. Advances in Neural Information Processing Systems, 37:21875-2D1 2024.
[140] Yuhao Lu, Yixuan Fan, Beixing Deng, Fangfu Liu, Yali Li, and Shengjin Wang. VI-grasp: a 6-dof interactive grasp policy for language-orienter objects in cluttered indoor scenes. In IROS, 2023.
[141] Tianzhe Chu, Yuexiang Zhai, Jihan Yang. Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv, 2025.
[142] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-sourg generalist robot policy. arXiv, 2024.
[143] Moo Jin Kim. Karl Pertsch. Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, et al. Openvla: An open-source ssiionlanguage--cction model. In CoRL, 2024.
[144] Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, and Jiaolong Yang. Moge-2: Accurate monocular geometry with metric scale and sharp details. arXiv preprint arxiv:2507.02544. 2025.
[145] Bowen Wen, Matthew Trepte, Joseph Aribido, Jan Kautz, Orazio Gallo, and Stan Birchfield. Foundationstereo: Zero-shot stereo matching. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5249- 52600 2025.
[146] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025.
[147] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv, 2021.
[148] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias NieBner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
[149] Johanna Wald, Torsten Sattler, Stuart Golodetz, Tommaso Cavallari, and Federico Tombari. Beyond controlled environments: 3d camera re-localization in changing indoor scenes. In ECCV, 2020.
18

---

[150] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In ECCV, 2014.
[151] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019.
[152] Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. CVPR, 2019.
[153] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. ICV, 2017.
[154] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal
policy optimization algorithms, arXiv preprint wwXivv/7070063443 2017.
[155] Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Amaud, al. Openeqa: Embodied question answering in the era of foundation models. CVPR, 2024.
[156] Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, and Lu Wang. Process reward models that think. arXiv, 2025.
[157] Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang. Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can lb Ilm surpass 405b Ilm? rethinking compute-optimal test-time scaling. arXiv, 2025.
[158] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In /CML, 2023.
[159] Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics, 2023.
19

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please check Sec .1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification:
Please check Appx. F.
20

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please see Sec. 3,4 and Appx. A/B/C/D for details.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibilityy which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Since we are still organizing the dataset and related code, we will open source them as soon as possible. In Appx. A/B/C/D, we try to explain how to access the raw data, preprocessed data, generated data, and details about all experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/ public/guie Baaiai//guidess/oeeeuubbae
wttpsuuiiisisiioosssoioooiiomeme for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. cc/ ccc/public/pee wwtccc/ggggee/eeeeeeeeee Cttps//oooSSbbmmeec wwsssnoolsliii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
22

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Please see Sec. and Appx.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: All our experiments involve VLMs. With the temperature to the output becomes deterministic, eliminating variability and thus making error bars unnecessary. Meanwhile, measuring the inference cost of various VLMs is computationally expensive.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please check Appx. D.I.
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neeue http://wuwspppppppppppeeeeeeeeee cc/puublic/fhiicsuuuuee
Answer: [Yes]
Justification: Please check Appx. C Cnd Appx. D.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please check Appx. G.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
24

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please see Appx. I.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please see Appx. C and Appx. I.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
25

---

Answer:
[No]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [No]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We declare the LLM usage in Appx AA.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer to LLM policy (https: //w.riurps...// cc/Confferecces/22222 for what should or should not be described.
26

---

Appendix Table of RoboRefer
A Discussion
28
Implementation Details and Samples of RefSpatial Dataset
29
B.1
2D Web Image
29
B.1.1
Multi-Stage Image Filtering
B.1.2
Pseudo-3D Scene Graphs Construction
32
B.1.3
Hierarchical Object Description Generation
34
B.1.4
Generating Diverse QA Pairs via Pseudo-3D Scene Graphs
36
B.2
3D
Embodied Video
39
B.2.1
Why Use CA-IM and How to Pre-process It
39
B.2.2
Inherent Challenges and Limitations in CA-IM
40
B.2.3
Addressing Limitations: Object Annotation and Bounding Box Filtering
41
B.2.4
3D Object Description Generation and Scene Graph Construction.
42
B.2.5
Free Space QA Generation for Object Placement
42
B.2.6
Generating Diverse QA via precise 3D annotations
46
B.3
Synthetic Data Generation in the simulator
48
B.3.1
Indoor Scene Generation
48
B.3.2
3D Asset Selection and Preparation
48
B.3.3
3D Asset Annotation with LLM
49
B.3.4
Scene Population and Data Rendering
51
B.3.5
Question-Answer Pair Generation
52
Implementation Details and Samples of RefSpatial-Bena
52
Implementation Details for RoboRefer
54
D.1 Architecture
54
D.2
Training Data
D.3 SET Training Details
55
D.4 RFT Training Details
56
D.4.1
Sampling Action Groups
56
D.4.2
Reward Design and Policy Update
56
Experimental Setting and Details
57
E.1
Experiments Compute Resources
57
E.2
Spatial Understanding Benchmarks
E.3
Spatial Referring Benchmarks
58
E,4
Simulation Evaluation..
59
E.5
Real-world Evaluation
59
E.5.1
UR5 Manipulation
59
27

---

E.5.2 G1 Humanoid Mobile Manipulation
59
E.6 More Ablation Studies
60
More Demonstrations
61
G More Discussion on Limitations and Future Work
61
H Broader Impacts
61
I Licenses
61
A Discussion
Distinction from SpatiaIRGPT. Our approach differs from SpatialRGPT in key aspects: (1) Task Setting: Our model addresses more challenging spatial referring task, where takes spatially constrained textual instruction input. requires multi-step spatial reasoning with learned spatial knowledge to precisely localize the referred object as a 2D point step-by-step. In contrast, Spatial- RGPT addresses a simpler VQA task and relies on extemally provided region information input for specific object referring. (2) Model Usage: Unlike SpatialRGPT, which needs additional masks or detection tools generate masks or 2D boxes as inputs for object reference and simplify referring tasks, our model can use textual descriptions for object referencing (see I.283), which better aligns with real-world robotic applications. (3) Data Pipeline: Our data pipeline adopts a more structured, progressive design than SpatialRGPT. It first uses 2D image data to teach core spatial concepts and general depth perception across diverse indoor and outdoor scenes. Next, accurate 3D data enhances fine-grained spatial understanding in indoor settings for robotics. Finally, simulation data introduces multi-step spatial referring with reasoning. This staged approach yields stronger spatial understanding and reasoning than SpatialRGPT, which relies solely on data generated from 2D images and lacks precise spatial perception for more complex spatial referring tasks. (4) Training Pipeline: Our training pipeline includes process-based RFT after SFT, further to improve multi-step reasoning and generalization for spatial referring tasks, whereas SpatialRGPT is trained with SFT only.
Justification for RFT. RFT brings two main benefits: (1) Generalization to Unseen Cases: In Tab. 2 in the main paper (RefSpatial-Bende Unseen raw), which features novel combinations of spatial relations absent from RefSpatial dataset, our 2B-RFF RFT model surpasses 2B-SFT BF-SFT by 9.1% in accuracy, showing the strong generalization enabled by the RFT stage. (2) Enhance Multi-Step Reasoning Ability: In the Tab. 8, the RFT-based model consistently outperforms the SFT-based model across varying reasoning steps, especially at larger steps, showing the RFT stage's effectiveness enhancing multi-step reasoning.
Why is NVILA chosen as the backbone? In Tab in the main paper, NVILA outperforms other open-source VLMs under comparable model scales, such as Qwen 2.5-VL (even 72B), in spatial understanding. Enhancing a strong baseline with our dataset and training strategy further validates their effectiveness. Notably, our dataset is model-agnostic and transferable to other backbones. Despite partial training on the RefSpatial dataset, Qwen2.5-VL-7B still shows notable improvements on spatial understanding benchmarks in the Tab. 9 below.
Depth-to-3D mapping assumption. The depth-to-3D mapping assumption is essential in real-world evaluation, as our model predicts only 2D image-plane points, while real-world tasks typically require 3D coordinates for grasping, placement, or navigation. While depth noise and partial observations are important real-world challenges, our setting follows prior work [5, 45], which assumes that accurate depth-to-3D mapping feasible given known camera intrinsies and extrinsics sufficient for common manipulation and navigation tasks. Moreover, these challenges can be effectively mitigated via the following strategies: (1) Depth noise can be mitigated by recent advances in monocular depth estimation [68], monocular geometry prediction [144], and stereo methods [145]. cases of severe noise, we employ FoundationStereo [145] in real-world settings to mitigate this issue. (2) Partial views are mitigated in our method by leveraging pixel-level target points. Further improvement is possible by incorporating RoboRefer as a spatially-aware planner for active perception.
28

---

The effect of depth noise on the model's accuracy and robustness, In real-world experiments, we utilize a relative depth estimation model, DepthAnything v2, obtain relative depth as the model's depth input, thereby effectively reducing depth noise from real camera. We also evaluate success rates under depth noise in real-world settings (see Tab below). Depth maps generated from the strong monocular relative depth estimation model (i.e., Depth Anything V2) offer the highest robustness and success. Despite depth noise from a real camera, RoboRefer maintains great performance by leveraging RGB priors due to mixed RGB and RGB-D training during the SFT stage.
Table 8: We report the success rates (%) of 2B-SFT and 2B-RFT model at each reasoning step on -ennhh
Benchmark
Reasoning Step Num.
2B-SFT
2B-RFT
Gain
Step 1
63.33
66.67
+3.34
Step 2
39.58
43.75
+4.17
RelSpatial-Rencc-LLoattii
Step 3
27.27
36.36
+9.09
Total
47.00
52.00
+5.00
Step 2
55.56
55.56
+0.00
Step 3
41.67
41.67
+0.00
Rei'Spatial- -rSSpatill-Bench-
Step 4
41.67
45.83
+4.16
Step 5
0.00
25.00
+25.00
Total
48.00
54.00
+6.00
B Implementation Details and Samples of RefSpatial Dataset
In this section, we present a comprehensive exposition of the implementation details and representative data samples underpinning the construction of the RefSpatial dataset. As this dataset is intended to equip general VLMs with the ability to adapt to spatial referring tasks, thereby enhancing spatial understanding and reasoning in a bottom-up manner, we meticulously design a multi-data-soune generation pipeline. We elaborate the three components of this pipeline follows:
2D Web Image (Appx. B.1): We present a 2D data pipeline comprising image filtering, pseudo-3D scene graph construction, hierarchical referential description generation-from coarse categories to fine-grained spatial referents-and diverse QA pair creation.
. 3D Embodied Video (Appx. B.2): This section outlines the 3D data selection process from CA- IM [136], discusses its limitations and mitigation strategies, and presents methods for enriched scene graph construction compared to the 2D data source. We further describe QA generation framework that leverages detailed 3D annotations (e.g., depth maps, oriented 3D bounding boxes) capture richer spatial relations. Finally, we detail how to generate QA pairs for the problem of "feasibility assessment for object placement in free space".
Synthetic Data from Simulator (Appx. B.3): We describe how to synthesize 3D scenes, select and annotate digital assets, efficient scene assembly and rendering, and the generation of QA pairs grounded in these simulated scenes.
In the following subsections of each section, we detail the employed models, prompt design rationale, data processing steps, filtering criteria, and illustrative examples, providing a clear and thorough overview of the construction pipeline and core technical details of the RefSpatial dataset.
B.1 2D Web Image
B.1.1
Multi-Stage Image Filtering
2D Web Images aim to endow the model with basic spatial concepts and comprehensive depth perception across both indoor and outdoor scenes. Here we use Openlmage [135] as 2D data source.
Overall Motivation and Goals for Filtering. The OpenImages dataset offers vast collection of 2D internet images (1.7 M images in training split) with extensive visual diversity, but many images, such as text-only graphics, QR codes, medical scans, or abstract art, lack relevance for spatial
29

---

Table 9: Performance on the single-step spatial understanding benchmarks across different model
types. Top-1 Top-2 accuracies are represented using bold text, and underlines.
CV-Rench [155]
BLINK na! [16]
Method
RoboSpatial [2] SAT [4] EmbSparial
2D-Relation 3D-Depth 3D-Distance
2D-Relation 3D-Depth
Qwen-2.5-VI. (hase)
82.15
60.17
69.00
64.34
60.98
49.59
30.00
40.20
Qwen-2.5-VL-7B (finetuned)
95.85
95.00
90.83
83.22
84.68
69.92
85.75
76.32
NVILA-8B (base)
91.54
91.83
90.67
76.92
76.61
59.35
63.33
67.72
RoboReler- SB-SF SFT (linetuned)
96.90
98.33
93.50
91.61
92.74
84.55
86.67
72.53
Table We report the success rates (%) of real- woald evaluation performance when using depth from DeythAny V2 and Real Camera.
Real-world
Depth from Depth-Anything Depth from
Pick the specific hamburger closest to the mag nearest to the camera.
80
70
Place the hamburger in front the toddy bear.
90
90
Pick the apple in front of the leflmost cup's logo side. existing apple row.
80
80
apple aligned
understanding and referring with reasoning. To curate dataset tailored for these tasks, employ a two-stage filtering pipeline: a coarse pre-filterigg using SigLIP2 [74], followed by fine-grained selection via Owen2.5-VLL [11], retain images rich in spatial semanties.
Stage 1: Initial Coarse Filtering. We employ the si siappp-gganttop-pptcchii initial filtering to efficiently discard low-quality or off-theme images (e.g., irrelevant scenes or content lacking multiple everyday objects). This step greatly reduces data volume, streamlining subsequent processing, Specifically, the SigLIP2 model is guided by predefined positive and negative textual labels. Positive labels represent desired image characteristics, while negative labels describe undesired content. For each image, the model computes cosine similarity between its embedding and all label embeddings. The label with the highest similarity is selected; it belongs to the positive set, the image is retained, otherwise discarded. Label sets manually refined iteratively balance recall and precision, ensuring relevance while excluding noise. These labels act as semantic anchors for visual-text alignment. In this stage, only 934k images qualified to be retained. Positive and negative label lists are provided in Listings and For more details about the compute resources needed in this stage, please Appx. E.
model for
Listing Positive Labels used during SigLIP2 filtering.
Positive Labels
"Kid-distance observation of some objects on a table", "Some objects on the desktop", "Distant view of
animals", "Xid-distance observation of some animals", "Distant view of one object", "Xid-distance observation of one object", "Distant view of some objects",
"Kid-distance observation of some objects", "Distant view of person",
"Xid-distance observation of person", "Distant view of some people", 'Nid-distance observation of some people", "Distant view of indoor scene", "Distant view of outdoor scene", "Distant view of traffic", "Distant view of Urban architecture"
]
Listing 2: Negative Labels used during SigLIP2 filtering.
Negative Labels
"Macro shot an animal", "Macro shot of one object", "Macro shot of a person", "Macro shot of flovers",
30

---

"A piece of text",
"A person displayed in front of white background",
product displayed in front a white background", screenshot of the graphics user interface",
A dimly 1it environment"
SigLIP2 preserves images rich in object diversity, depth cues, and scene context (indoor/outdoor) through the above labeling process, but its performance declines certain image types, including:
1. Paintings/Artworks: Especially those with visible brushstrokes or canvas textures.
2. Low-Light Scenes: Images with minimal illumination and strong shadows.
3. Grayscale Photographs: Black- anthwee imagery lacking color cues.
4. Distorted Images: Those exhibiting warping, mirroring, or other geometric anomalies.
5. Multi-Scene Collages: Images containing three or more distinct scenes with hard borders.
SigLIP2 struggles to detect and interpret these categories reliably, highlighting the need for secondary, fine-grained filtering stage.
Due to SigLIP2's limitations in handling certain visual content mentioned above, we introduce a fine-grained filtering stage using the Qwen2.5- model to further improve dataset quality. This step ensured the final images are clear, authentic, and well-suited for spatial understanding and reasoning required for spatial referring tasks. Qwen2.5- VL processed 93-4k images filtered by SigLIP2, retaining 846k. Although Qwen2.5-VL. offers higher filtering precision, its slower speed necessitated the use of SigLIP2 for initial fast filtering, significantly improving overall efficiency. To ensure accurate and consistent fine-grained filtering, we adopt a structured prompt engineering strategy for the Qwen2.5-VE LL model. The process begins with system prompt (See Listing 3) that defines the model's role as an image analysis expert, specifying key visual attributes to assess and negative categories to detect, and enforcing strict workflow. For each image, corresponding user prompt (See Listing 4) instructs the model to determine whether the image belongs to any predefined negative categories. The model's response follows a structured format: the segment after pipe symbol ()) is "Yes", the image classified as negative and discarded; otherwise, it is retained. This prompting scheme ensures that the model adheres to consistent output, enhancing the reliability of filtering outcomes. Please refer to Appx. E, for details on the computational requirements of this stage.
Stage 2: Fine-grained Filtering
Listing 3: System Prompt for Qwen2.5-VL-7B filtering.
system_ prompt
You an image analysis expert. Follow this workflow rigidly:
1. Content Analysisss:
Inspect: Main subjects, artistic style, visual characteristics Check: Lighting intensity, color channels, geometric integrity, composition structure
2. *Category Verification** (YES if matches ANY):
a) Painting/Artxorf Visible trushatrokes/canc texture
b) Dim Lighting Very low brightness, heavy shadows
B&W Photo Grayscale only (0 color channels)
Distorted Image barping/mirrorian anomalies
e) Multi-image Collage >=3 distinct scenes with hard borders
3. ** Structured Response**:
Output EXACTLY in this format: [[Analysis sentence]. Yes/No"
Analysis must contain observable evidence
Final ansser MUST use pipe separator
Examples VALID responses:
"This image is a composite created by stitching together multiple smaller images, with distinct white borders visible between the individual components. Yes"
31

---

"This image features vibrant colors, is neither an artistic painting composite of multiple images, and does conform any the specified categories.
Listing 4: User Prompt for Qwen2.5-VL-7B filterin..
user_prompt
Analyze if this image belongs to ANY of these categories:
Painting/artwork
2. Dim lighting
3. Black-and Black-and-white
4. Geometric distortion
5. Multi-image collage
Respond EXACTLY FORMATTED as: [Your evidence - based analysis]. Yes/No
Visualizing Overall Filtering Results. Fig. 6 presents visual overview, showing the effectiveness of our multi-stage filtering pipeline. The first row shows the output of SigLIP2, which effectively removes images lacking spatial semantics, such macro shots of animals, people, flowers, textual content, and GUI screenshots. The second row demonstrates how 1ree2.5-vVL71 further eliminates unsuitable categories, including artworks, dimly lit scenes, black-and-whitt images, geometrically distorted content, and image collages. The third row displays the retained images after both stages, which exhibit rich spatial relationships, confirming their suitability for spatial understanding and reasoning and the overall quality of our dataset.
creators
SigLIP2 filtering
person
user graphics
Qwen-VL
filtering
Painting AArtwork
B&W Photo
Multi-iage Collage
Dim Lighting
Distorted Image
Remain
Distant view traffic
Some objects on the desktop
Distant view outdoor stene
Distant view some people
Distant view of some antmals
Figure 6: Visual overview of the multi-stage filtering results. Row 1: Images discarded by SigLIP2 due to insufficient spatial context (e.g., close-ups, text). Row 2: Additional filtering by Qwen2.5-Vi LL removes non- natural content (e.g., artwork, collages). Row 3: Remaining high-quality images suitable for spatial understanding and referring.
B.1.2 Pseudo-3D Scene Graphs Construction
Although filtered 2D images contain some spatial cues, QA pairs containing sufficient 3D spatial information (e.g., "near" "far", distances) derived directly from these 2D images challenging. Inspired by prior work [1, 6], we construct pseudo-3D scene graphs from 2D images to enhance the generation of QA pairs with rich 3D spatial semantics. In these graphs, nodes represent object attributes, while edges encode inter-object spatial relations. We detail the process of converting 2D images into pseudo-3D scene graphs below.
Object Detection and Annotation. Although the OpenImages dataset provides annotations, its limited vocabulary and coarse labeling insufficient for open-world scenarios. To address this,
32

---

leverage oftt-offtthearr foundation models for enhanced object detection and annotation. Specifically, our scene graph construction pipeline integrates the Recognize Anything Model (RAM) [66] and GroondingDINO [101] to assign semantic labels and bounding boxes to key objects in filtered raw 2D images. The workflow is broadly as follows:
1. Semantic Labeling via RAM: RAM analyzes each image generate category labels for recognized objects. Its broad recognition capability ensures comprehensive semantic coverage, guiding subsequent localization.
2. Bounding Box Localization via GroundingDINO: The labels from RAM are used as text prompts for GroundingDINO,, open-vocabulary detector that localizes the corresponding objects and outputs precise bounding boxes.
Although recent VI.M, Le., Florence-2-Large supports instruction-based recognition and detec- tion simultaneously, we find that combining RAM with GroundingDINO yields superior results. In Fig. Florence-2 (top) often produces ambiguous redundant detections (e.g. vague labels, multiple boxes for single object, or single boxes covering multiple objects), which are unsuitable for precise object referring. contrast, GroundingDNO++RAM (bottom) generates concise labels and one-to-one object-boundigg bounding- box mappings, better satisfying the requirements of referring tasks.
Florence-2-large
blue glass facade
tall skyscraper
tall buildings
tall buildings
several cars
several cars
several cars
several cars
GroundingDINO + RAM
building
building
building
building
Figure 7: Object detection comparison: Florence-2 (above), Grounding (below)
3D-aware information Extraction. To further extract 3D-aware information from 2D images, we adopt UniDepth V2 [68] for metric depth estimation due its recent state-of-the-ar performance, surpassing models such as DepthPro and Metric3Dv2 across multiple benchmarks. For camera intrinsic prediction, we employ WildeCamera [76]. Together, these models enable robust 3D point cloud reconstruction of the scene. Based on previously annotated object bounding boxes, we apply SAM 2.1 [78] to generate instance masks. Each resulting Pseudo-3D scene graph comprises object labels (via RAM), 2D bounding boxes (via GroundingDINO), instance masks (via SAM 2.1),
33

---

and object-level point clouds (via UniDepth v2 with WildeCamera), resulting in axis-aligned 3D bounding boxes. A visualization is provided Fig. 8.
Figure 8: Scene graph visualization with image, detected objects, and corresponding point clouds.
B.1.3 Hierarchical Object Description Generation
While 3D scene graphs encode basic object categories, real-world scenes often contain multiple instances per category. Prior datasets [2, 3] simplify reference by assuming single object per category, limiting their utility in spatial referring tasks. To overcome this, we augment object descriptions with attributes and spatial relations, enabling finer-grained disambiguation among similar instances of the same category. We present our two-stage generation pipeline below.
Stage 1: Generating Object and Image Dense Descriptions in image space. We first generate detailed descriptions for each detected object and the entire image, employing the Qwen2.5-VL-7B model. This stage involves both object-level captioning and comprehensive image-level captioning. The global captions are essential for providing contextual grounding to downstream large language models (QwQ-32B during LLM QA generation (detailed in Appx. B.1.4), enhancing the relevance and accuracy of the outputs. Prompt templates are detailed in Listings 5 and 6. In particular, the object_ caption, user text, _octt__roppttt uses dynamic placeholder [class [class_name], which is filled with the object category predicted by the RAM model (See Appx. B.1.2).
Listing 5: Prompts for Image Caption Generation with Qwen-VL.
image_ cetee_cppioonststeeee srrrrin___yttet_tte_rrrommm are an expert image analysis assistant. Your task is generate detailed and comprehensive description of the image
.."
Please focus accurately capturing all visual elements present
in the image, including objects, scenery, colors, shapes, textures, and lighting.
Your description should be clear, precise, and professional. Additionally, ensure that your description begins with either "this image' 'the image'.
Please carefully examine the provided image and generate detailed description.
Include all visible elements such as objects, scenery, colors, shapes, textures, and lighting.
Ensure that your description is thorough, accurate, and complete, and that it starts with either 'this image' the image'.
34

---

Listing 6: Prompts for Object Caption Generation with Qwen-Vis VL.
abject_caption.
prompt
You are visual localization analyzer working with TWO distinct
images:
1. MUOSITION-REPEREIN REFERENCE] (First Image):
Contains ONLY location clues with background
Strictly use ONLY for determining spatial position (left/right/ upper/lower/ center)
Ignore all visual features except object placement
[DETAIL-SOURE] (Second Image):
Shows the object 's TRUE APPEARANCE without background
Extract EXCLUSIVELY from this: color, texture, material, shape Never infer details from the first image
Generate phrase in pattern: [Color] [Material] [Object] at [Position
Example: "Matte black laptop the left" NOT "Red -eed-bosk boxed laptop"
abject_ccation.c user text_prompt ...
[class_name] name] marked by red box in FIRST image and fully For the [class
shown in SECOND image:
COLOR/MATERIAL: Must
from SECOND image
POSITION: Only from FIRST image placement
Forbidden actions:
x Mention 'red box' or background elements
Use location terms in second image
Combine features
images
Describe the "class_name) marked by red box in FIRST image and fully shown in SECOND image with this format: [Object]
[Position]
Samples:
"Brushed metal water at bottle left"
"Glossy ceramic mug at upper center"
"Faded denim jacket at lower right"
Stage 2: Generating Object Description with Spatial Cues. To enhance referential specificity in object captions, particularly when multiple instances of the same category coexist, we adopt heuristic strategy that appends spatially indicated relative information, such as "the third chair from the front". This method leverages 3D object positions from the scene graph (See Appx. B.1.2). By comparing same-category objects along the three principal axes (front-back, left-right, top-bottom), we identify the axis with the largest spatial variation the primary arrangement direction to guide relative spatial reference generation. Once the main sorting axis is identified, we retrieve appropriate templates from predefined library (see Listing 7) augment the initial object descriptions. These templates are designed to capture diverse natural language patterns. For instance, for a row of chairs arranged left right, templates may include: "[dense_cappiion). which is the (ordinal) (class _ammss___mme____ from left to right," or
). the (ordinal) (class name) ) in the left-to-right sequence". Here, dense _ccppiinnnnnncc denotes the initial description generated by the Qwen2.5-VL. model, ordinal indicates the object's position in the sorted sequence, and class_ aam__amee name is the category label predicted by RAM. This spatially-aware enhancement is applied only when multiple instances the same category detected to avoid redundancy. If object appears only once, its original dense caption used directly. To ensure spatial diversity, we set a variance threshold across the three principal axes; images with multiple same-category objects but low variance on all axes are discarded, resulting in a final set of 466k images. By integrating spatial ordering with visual descriptions, this heuristic enables the generation of precise and diseriminative referential expressions, essential for producing high-quality, unambiguous question-ansee answer pairs.
Listing Templates for Spatial Order Description Enhancement.
TEMPLATES
35

---

"left_to_right"
"(dense_ caption,, which is the {ordinal} {class_name) from left to right", cidense_caption) marked as the {ordinal} {class_names in a riight arrangement",
1,
" togghttt__leftte
"(dense_ caption,, the fordinal} {class_name} viewed from the right", "(dense_caption)) the [ordinal} (class_name) from the right",
1,
"[dense_caption), "[dense caption), which appears as the fordinal} "alass_name] when vieved from the front", ('dense_caption) positioned the (ordinal) {class_name) in to-ckk order",
1,
"back_tte to "back_to_front":
('dense_caption) which is counted as the {ordinal) { plass_named starting from the back", [(dense_caption]] the (ordinal) {class_name) in back-to- front sequence",
1.
" "top -cop-to to_ttonn
" ((dense_caption) the {ordinal} {class_name) viewed from the top",
cclense_caption) placed as the {ordinal} {class_name) when sorted from top to bottom",
l,
botton to_
[edense_caption)) which ranks as the fordinal} (class_name) bottom order", "[dense "[dense_caption) caption), arranged as the {ordinal} {class_name when ordered from the bottom",
Examples of Object and Image Descriptions This part qualitatively shows the representative examples with the generated descriptions. As shown in Fig. 9, we present two types of object captions. The top row shows simple captions produced by Qwen2.5-VL for single-instance object categories, where spatial ordering is unnecessary. The bottom row includes captions augmented with spatial order information to distinguish multiple instances of the same category. Additionally, Fig. 10 demonstrates Qwen2.5-VL's ability to generate detailed global descriptions of entire images used in the following.
B.1.4 Generating Diverse QA Pairs via Pseudo-3D Scene Graphs
After constructing scene graphs and generating hicrarchical object descriptions, leverage this information to generate diverse QA pairs from pseudo-3D scene graphs to support SFT training for improved single-step spatial understanding.
Template, Choice and Fact QA Generation. We first adopt template-based based method to generate structured preliminary QA pairs, multiple-choice questions, and factual statements. The templates are derived from scene graph information (e.g., object attributes, positions) and refined hierarchical object descriptions. When designing QA templates using pseudo-3D scene graphs from 2D images, we explicitly account for the spatial ambiguity inherent in single-view pseudo-3D representations (e.g., inaccuracies in monocular depth, lack of object orientation, absence of 3D oriented bounding boxes). Consequently, our QA templates from 2D data source focus mainly on qualitative spatial understanding and reasoning, while incorporating quantitative cues only when they be inferred reliably from 2D or pseudo-3D signals. The spatial concepts covered the QA templates fall into the following categories:
1. Relative position relations: capture spatial layouts (left/right, above/bclow, front/behind).
36

---

black frying pan at left
yellow cup upper left
white book lower
black chair left
black motorcycle left, the second motorcycle from the left
left, which the first from the left
black goose right, placed the first goose when sorted from right to left
red whrch the first barrel in back-to- front order
Unique-category category captions. Bottom: Spatially-awe aware captions for the same categories. Red boxes indicate referenced objects.
Figure 9: Generated object descriptions. Top: Unique
image captures bustiing bright lighttng and clear buildings. showcasing a variety architectural styles. On the left side of the street, there are older, multi story buildings with brick facades and classic designs, featuring large windows and decorative cornices. prominent sign reading "TENNESSEE" is visible, suggesting the location could be Tennesses. The right side of the street features taller, contamporary glass building, reflecting surrounding environment, which indicates blend of vchicles parked the
scone, street
during the daytime as historic and modern
new architecture
itself vell-maintaised, typical many moving, suggosted by some visible headlights taillights. ounership. sidewalls spacious,
of sedans and SUWs reflecting typical with lumpposts lining street, providing warm glow that enhances the urban amblance. the background, lush green hillside rises, offering natural contrast the urban setting. The hill is densely covered with greon follage, siggesting a park natural reserve, which may be significant feature in this area. The everall scone activity and nature. The lighting the soft and natural, indicative suggentig
lively har harmonious blend of urban and morning
Figure 10: Visualization of generated image detailed descriptions.
2. Relative size comparisons: describe object attributes (e.g.. bigger/smaller, taller/shorter, wider/thinner) often inferred from image-plane projections.
3. Quantitative information from 2D or pseudo-3D label: include spatial reasoning based on estimated depth maps, 2D object coordinates, and coarse monocular depth approximations.
Accordingly, we design diverse QA templates in these types for spatial understanding:
1. Templates assess spatial and size relations:
Position relation: "Is [A] the left of [B] ?
Size comparison: "Which object is larger, [A] or [B]?"
2. Templates query 2D point of uniquely identified object '[A]' (defined in Appx. B.1.3):
"Where [A] located? Provide
coordinates." (e.g., "Where is the red apple at left, which is the second apple from left to right, located? Provide its 2D coordinates." "
3. Templates query attributes at specific 2D point '[X]' (formatted as '(x. y)'):
Depth retrieval: 'What is the depth at point [X]?" (e.g., "What is the depth at (0.528, 0.317)?")
Object identification: "Which object is at point [x] (e.g.. "Which object is al (0.753, 0.839)?")
We further design fact templates to generate declarative statements, forming a structured basis for prompting Reasoning LLM to produce richer and more natural QA pairs. Example templates include:
37

---

1. Approximate depth: "Point [A] and the camera are [X] apart." " (based on depth estimation).
2. Precise 2D object location: "[A] is located at point [X] ", "[A] is to the right of [B]
Reasoning QA Generation. To generate more natural, complex, and diverse QA pairs beyond templated formats, we leverage powerful reasoning LLM, QwQ-32B [146]. It takes the factual statements, initial QA pairs, and multiple-choice questions (if available) as input, along with global image captions and precise object descriptions. QwQ-32B then produces more challenging and conversational spatial reasoning QA. The prompt design is shown in Listing 8.
Listing 8: Prompt for QwQ-32B QA Diversification and reasoning QA Generation.
You helpful assistant tasked with generating spatial reasoning- based questions and answers from provided descriptions of scenes.
Rules:
1. **le have three types of input informationss:
** [+[ene]]**: general description of the entire image, which
provides context for the objects and their surroundings.
**Example:**
[Scene]: The image shous tranquil lakeside with a small wooden dock the right and calm, reflective water in the center. The sky overcast.
** [Objects]*:: A list containing one or more object labels separated "I"
=*Eapple:***
[Objects]: teal glossy water at lower center green bamboo dock at lover right.
[Objects Description)::: Provides spatial comparative details betveen those objects.
**Example:** [Objects Description]: teal glossy water at lower center is taller than green bamboo dock at lover right.
2. ** When crafting a Question**:
** Always use the provided [Scene] description as context** to ensure the question aligns with the overall image.
**Mention all object labels from [Objects]++ in the question.
**Do not modify or paraphrase the object labels**; they must appear
**exactly*s as given in '[Objects]'.
**Do not assume or invent additional scene details** beyond what is provided in '[Scene]'.
+DDo not reveal the specific details [Objects Description]** (
like which object is taller, shorter, wider, etc.).
Alwaya generate questions related to the description using the object labels from [Objects].
in the The question should read from **an observer 's perspective**. The description should always be used to ansver and not leak into the question.
Each object label in '[Objecte]' **must appear exactly Question.
3. **When crafting an Answer**:
**neiioo at least one object label from [Objects]*s in the anaver. **Use the '[Objects Description] to provide correct ansver*t. **Ensure the answer provided '[Scene] and '[Objects]'*..
concise, factual, and directly related the
**You may restate or summarize the relevant details from '[Objects Description] but do not introduce new assumptions**.
38

---

Here several examples:
[Scene]: The image depicts a modern living room with a large window allowing warm sunlight to enter. The room has a wooden floor, a patterned rug in the center, and coffee table with few magazines neatly stacked positioned centrally, facing the television mounted on the opposite wall. To the left of the sofa, a black metal chair with cushioned seat is placed beside a tall bookshelf filled with an assortment of books and decorative items. The furniture arrangement leaves an open pathway between the sofa the chair. [Objects]; yellow leather sofa at lower center, black metal chair on the left.
it. yellos leather sofa is
[objects Description]: The path between yellow leather sofa lower center and black metal chair the left "Question"; You are a cleaning robot that is meter wide. Now you are standing in living room and see the image; you want to move from here to the door that leads to the backyard. Do you think go through the path between the yellow leather sofa lower center and the black metal chair on the left?
1.5 meters.
"Answer": The path between the yellow leather sofa at lower center and the black metal chair on the left is 1.5 meters, so yes, the robot go through the path between the yellow leather sofa at lower center and the black metal chair the left since it is wider than the robot's width.
[Scene]:
The image shoucases a modern kitchen with wooden countertop that extends across the space, separating the cooking area from the dining area. On the left side of the countertop, holds a variety of fresh produce. A red fresh apple placed on the left side of the bowl, while a bright fresh orange sits neatly the right side. Behind the fruit boul, glass pitcher filled with orange juice and stack of white ceramic plates are visible. sink,
fruit bowl
Natural light streams reflecting off the stainless steel appliances and giving the space a bright, clean feel.
from large window above
[Objects]: red fresh apple the left, fresh orange on the right. [Objects Description]: red fresh apple the left side of fresh orange on the right.
the left positioned on
"Question": You see two fruits, a red fresh apple on the left and a fresh orange "Answer"; The red fresh apple on the left is more the left.
the right. Which one is more on the left side? Now its your turn!
Training data visualization. For specific examples of training data generated from 2D web images and their visualizations, please refer to Appx. F, which contains detailed sample presentations.
B.2
3D Embodied Video
B.2.1 Why Use CA-1M and How Pre-process
Rationale for Selecting CA-1M as the 3D Data Source. To enable fine- fine-grained spatial reasoning in indoor environments, we adopt Apple's 'ss open-source CA-IM [136] dataset as our primary 3D data source. CA-1M aligns closely with our objectives due to the following key attributes:
1. Dense 2D/3D Annotations: CA-IM provides fer-frame 2D/3D oriented bounding boxes, enabling spatial localization (e.g., 3D spatial occupancy) and accurate interaction modeling.
2. Comprehensive Camera and Depth Data: The inclusion of camera intrinsies, extrinsics, and depth maps supports accurate 3D reconstruction and geometric reasoning.
39

---

3. Large-Scale Coverage: Its extensive volume enables training/evaluation of VLMs at scale.
These features offer strong foundation for constructing 3D scene graphs and generating spatial reasoning data involving 3D geometry, object interactions, and egocentric understanding.
Comparative Analysis with Alternative 3D Datasets. While several high-quality 3D datasets exist, they fall short of meeting the specific requirements of our project, motivating our selection of CA-IM
1. ARKitScenes [147]: As predecessor to CA-1M, it provides only global 3D bounding boxes without per-frame 3D or 2D annotations. Projecting these 3D boxes to 2D yields oversized and contain irrelevant objects. Additionally, its annotations are less comprchensive, and image resolution also much lower compared to CA-IM.
2. ScanNet V2 [148]: Lacks 3D bounding boxes, making object orientation estimation infea- sible. Although EmbodiedScan introduces per-frame 3D bounding boxes, still lacks 21) annotations, and projected 21D boxes remain imprecise and contain irrelevant objects.
3. 3RScan [149]: Suffers from low image quality, hindering spatial information extraction and limiting usability.
In summary, despite some limitations (See Appx. B.2.2), CA-IM provides large-scale egocentric video, per-frame 2D/3D annotations, depth, and camera parameters, making CA-1M the most suitable choice for generating rich and complex 3D spatial data from an embodied perspective.
Pre-processing. Video datasets capturing continuous activities, such CA-1M, exhibit high tem- poral redundancy, as consecutive frames often contain near-identical visual content with minor variations. Processing all frames is computationally intensive and yields redundant samples with limited informational gain for model training. To mitigate this, we adopt frame sampling strategy, selecting frame every 20 frames. This reduces redundancy while preserving meaningful scene and viewpoint transitions. The resulting subset maintains scene diversity and supports efficient downstream processing, including 3D scene graph construction and question-answer generation. In Fig. 11, the top row shows four consecutive frames before sampling, revealing minimal variation; the bottom row, sampled 20-frame intervals, demonstrates significantly greater scene variation.
Before sampling
time step
After sampling
time step
Figure 11:
Comparison of frame sequences before and after sampling. Top: Four consecutive frames from the original video. Bottom: Corresponding frames sampled every 20 frames, exhibiting increased temporal variation between adjacent frames.
B.2.2 Inherent Challenges and Limitations in CA-IM
Although the CA- 1M dataset is chosen for its large scale, egocentric perspective, and rich annotations, presents inherent limitations that undermine scene graph construction if unaddressed.
Ambiguous or Meaningless Object Annotations. major issue is the prevalence of ambiguous semantically insignificant object annotations. Many instances are difficult to interpret, even for
40

---

humans, due to unclear boundaries or a lack of identifiable semantics, For example, some bounding boxes enclose small, indiscernibee regions such as a patch of wall or vague background elements (see Fig. 12). Incorporating such annotations into model training introduces noise, hampers spatial understanding. and may mislead the model's perception object relationships.
Widespread Absence of Semantic Labels. Another major limitation of CA-IM is the lack of semantic labels for most annotated objects. Unlike datasets (e.g., ARKitScenes and ScanNet V2), which provide object category annotations, CA-IM includes only a few structural categories (e.g.. floors, walls, doors), leaving the majority of object instances unlabeled. Although bounding boxes indicate the presence of objects (i.e., labeled as "object"), their categories (e.g., "chair", "table") remain unknown. This absence of semantic information hinders spatial understanding, making impossible to generate category--depener queries such as "Is the red chair to the left of the table?"
Figure 12: Example of some meaningless annotations in CA-
B.2.3 Addressing Limitations: Object Annotation and Bounding Box Filtering
To address the limitations of the CA-1M dataset, particularly the lack of semantic labels and the prevalence of noisy bounding boxes (See Appx. B.2.2 above), we develop a dedicated annotation and filtering pipeline. This enhances the dataset's usability for downstream tasks such 3D scene graph construction and spatial reasoning. We detail the two-stage pipeline below.
Stage 1: Initial Annotation and 2D Bounding Box Prediction. We employ multi-model combi- nation strategy to annotate unlabelled objects and refine bounding boxes in CA- IM video frames. Specifically, integrated GroundingDINO, RAM, and Florence-2 to perform object semantic labeling and 2D bounding box prediction. Our approach yields semantically meaningful and visually coherent object bounding boxes compared to the often sefeculttto-diseer bounding boxes from the original CA-1M annotations. To maximize recall, we intentionally lower the confidence thresholds of Grounding and RAM, ensuring the inclusion of low-confidence but potentially relevant objects. These candidates are retained for further validation in subsequent matching and filtering stages.
ambiguous
Stage 2: Bidirectional 2D Bounding Box Matching. To associate model-predicted 2D bounding boxes (with semantic labels) with unlabeled boxes in CA-1M, propose bidirectional matching and refinement strategy. This not only enables semantic annotation of meaningful objects but also filters out noisy ambiguous CA-1M bounding boxes, thus enhancing the utility of its 3D annotations (e.g., 3D oriented bounding boxes). The strategy consists of two steps:
1. Matching CA-1M Bounding Boxes Model Predictions. We first match original CA-IM 2D bounding boxes to model-predicted bounding boxes based on the IoU metric. Due to the sparsity, occlusions, and fragmentation of annotation, multiple CA-1M boxes may correspond to single prediction, resulting in many-to-one matches.
2. Refining Model Predictions via One-to-One Mapping. To resolve many-to-one matches, retain only predicted boxes that match least one CA- 1M bounding box. For each, assign the CA-1M 1M bounding box with the highest loU as its unique match. This enforces a one-to-one correspondence, eliminating redundant or weakly aligned CA-IM boxes. The result is a refined set of original bounding boxes with strong semantic alignment.
Visualizing Matching Results. Fig. 13 shows the effectiveness of our bounding box matching procedure. The first row shows successfully matched CA-1M bounding boxes annotated with RAM-ereddittdd object labels, while the second row highlights unmatched bounding boxes, typically corresponding to ambiguously annotated objects. This demonstrates that our method reliably filters
41

---

out uncertain annotations and assigns semantic labels to clearly identifiable objects. For subsequent scene graph generation, we adopt the model-predictte bounding boxes instead of the original CA- 1M ones, they better align with the visible object extents, facilitating more accurate instance mask extraction via models like SAM2.
conditioner
cabinet
lamp
paper
Figure 13: Visual comparison of 21) bounding box matching. The top row shows matched CA-IM bounding boxes with object labels; the bottom row displays unmatched or ambiguous cases.
B.2.4 3D Object Description Generation and Scene Graph Construction.
The process 3D scene graphs construction largely follows the 2D scene graph pipeline in Appx. B.1.2 and the description generation in Appx. B.1.3. The resulting graphs structurally similar to those from Openlmages in Fig. The key distinction CA-IM's focus on indoor en- vironments and its provision of high-prees nigh-precision geometric data (e.g., ground-truth depth, camera intrinsics/cxtrinsicss 3D oriented bounding boxes). The enhanced 3D scene graph outperforms 2D-based counterparts in object localization and spatial relation accuracy, enabling structurally rich and quantitatively grounded 3D QA data across 28 spatial relation types..
B.2.5 Free Space QA Generation for Object Placement
A key challenge in spatial referring is identifying unoccupied regions suitable for object placement. We propose a multi-step pipeline to address this.
Step 1: Detecting Viable Platforms via Qwen2.5-VL. We first filter out scenes lacking plausible placement surfaces (e.g., only walls or ceilings). To this end, employ the Qwen2.5-VL-7H initialized with system prompt (Listing 9) that specifies its role and procedure. For each image, user prompt (Listing 10) directs the model to detect candidate images with surfaces such as tables. floors, or shelves. Fig. 14 shows the filtering results. The first row shows scenes without platforms, while the second row shows scenes with platforms. This filtering enables subsequent computationally intensive analysis to focus semantically relevant scenes for placement queries.
Listing 9: System Prompt for Qwen2.5-VL identify images with suitable platforms. i image__aaa www_v_eeplaafomm..pm settrrrsssstttttttteeeeee _eextpprompt
You are an expert visual scene understanding assistant.
Your task to analyze an image determine whether contains any obvious flat horizontal surfaces** where physical objects can be placed. These include **floors, tabletops, bed surfaces, or other flat and stable areas**.
IMPORTANT:
If you can see any part of the **loor** **tabletop**, **bed** or **imilarr flat surfaces*e, you MUST assume it can support physical objects (Yeg, books, boxes, pillows).
42

---

Do NOT consider whether the surface is cluttered, partially visible, obstructed. If the platform exists and is horizontal, can hold objects.
Your answer must be based strictly on visible surfaces.
You must provide short reasoning based visual evidence in the image, followed by final conclusion.
Your reaponse MUST strictly follow this format: [Your analysis]. Yes/No"
Examples valid responses:
Example 1:
The image shows wooden floor that is flat and unobstructed. And it could potentially support physical objects. Yes
Example 2:
There is a bed clearly visible in the scene with flat top surface where items like pillows books be placed. Yes
Example
rectangular table is visible in the center of the image, providing a flat surface suitable for placing objects. | Yes
Example
The image contains mostly a wall with a window and no visible floor, table, or other flat surfaces. No
provide any extra commentary or formatting outside this exact format.
Listing 10: User Prompt for Qwen2.5-VL identify images with suitable platforms. sate__ave_pllaffe applatfom__usr___e_
image__aaee Please examine the image and determine whether it contains any **clear horizontal platforms** where physical objects be placed. These platforms include: **floors, tabletops, surfaces, or other flat and stable horizontal areas**.
user_text, beser_text_promp =
Important notes:
As long **any part**
**floor**, **tabletop*., **bed surface **, or similar platform visible the image, you must assume it is capable of supporting physical objects (such as books, boxes pillows, etc.).
Do NOT consider whether the surface is messy, partially blocked, whether there's enough space. the platform exists and horizontal, you must assume it can hold objects.
Your answer must be based entirely on visible visual evidence in the
image.
You must respond the following exact format: [Your analysis]. I Yes/Nlo"
Refer to the following examples guide your response:
Example 1:
The image shovs wooden floor that is flat and unobstructed. And it could potentially support physical objects. I Yes
Example
There is a bed clearly visible in the scene with a flat top surface where items like pillows or books can be placed. I Yes
43

---

Example 3:
rectangular table is visible in the center of the image, providing flat surface suitable for placing objects. Yes
Example 4:
The image contains mostly wall with window and visible floor, table, other flat aurfaces. No
Please strictly follow the format and do not add any extra commentary.
platform
have platform
Figure 14: Visualization of platform filtering results. Row 1: Examples of images filtered out (lacking platforms). Row 2: Examples of images retained (containing platforms).
Step Gravity Alignment and Top-Down Projection. To enable top-down (orthographic) scene projection, we apply gravity alignment matrices from the CA-IM to transform point clouds and object 3D bounding box into a consistent frame where gravity uniformly points downward. This normalization allows for projection onto plane orthogonal to the gravity vector, revealing object layouts and spatial relations more clearly. Fig. 17 shows the point cloud and coordinate axes before and after alignment.
Figure 15:
Figure 16: (b) After gravity alignment
(a) Before gravity alignment
Figure 17: Visualization of gravity alignment. (a) The scene before aligmment. (b) The scene after alignment, with the Y-axis oriented along the gravity vector.
Step 3: Programmatic Platform Association. After gravity alignment, we associate objects with supporting platforms based on their spatial relationships. For relationships such as "front/behind/left.
44

---

/right", we identify the platform supporting a target object by checking the top surfaces of candidate platforms. A platform qualifies if its top surface within 0.05 meters of the object's bottom surface and overlaps at least 70% of the object's bottom area in top-down view. For the "below" relationship, when an object may be suspended in mid-air, we identify the supporting platform as the one located beneath it, whose top surface is nearest to the object's bottom, and whose top-down intersection with the object exceeds 70% of the object's area. For the "above" relationship, the object's own top surface is treated as the reference platform, eliminating the need for additional platform search. For the "between" relationship involving two objects, each object's supporting platform is determined independently using the same procedure as in the stront/back/left/righhh cases. The space between them is considered valid only if both share the same supporting platform, ensuring spatial reasoning occurs within unified physical context.
Step 4: Identifying Other Objects on the Platform. After identifying the target object and its supporting platform, we locate other relevant objects co-occurring on the same platform. For Stronbbeind//efrrighhbeewwee
candidate objects are selected based on the following criteria:
1. Their bottom must not be significantly higher than the top of the target object.
2. Their top must remain above the platform surface.
3. Their XZ-plane footprint must intersect with that of the target object.
4. Their volume must not exceed 4.236 times that of the target object.
To mitigate the over- weer-fitering of solid objects during visible point selection via depth matching, introduce a volumetric constraint (thresholded at approximately 4.236, i.e., (1/0.618)², the cube of the reciprocal of the golden ratio). This prevents large, potentially hollow objects-suce as tables with substantial under-space- ffrm being misclassified as fully occluding based solely on their bounding box volume, which would mislead placement reasoning on the primary platform. Instead, the filter targets small to medium-size -ized objects, where the bounding box volume more accurately reflects actual occupancy. For such objects, even if hollow, the limited under-space is typically negligible for placement purposes. This behavior is shown in the top-view occupancy maps in Fig. 18 and Fig. 26, where large table is excluded due exceeding the volume threshold. Despite being present the scene, its hollow geometry allows for usable space beneath, justifying its omission.
For the "below" relation, object target might be suspended or on a higher tier) if:
platform is considered below the target object (when the
1. Its footprint (XZ-plane projection) intersects with that of the target;
2. Its bottom is no higher than the top of the target;
3. Its top is not below the top surface of its supporting platform.
For the "above" relation, an object is considered above the platform of target object if:
1. Its bottom is within 20 em above the platform's 's top surface;
2. Its top is not below the platform's top surface;
3. Its footprint (projection on the XZ plane) overlaps with that of the platform.
Step 5: Sampling Unoccupied Points in the View Occupancy Map. After identifying the target object, its supporting platform, and adjacent objects, we determine the surrounding free space. This includes regions in front, behind, left, right, above, below the target. To locate free space in the horizontal directions #ffront/behinddeeff/riggge
we define a 90º sector centered on the target and oriented in the respective direction. The sector radius set to the maximum of either the diagonal of the object's 's footprint or fixed 20 cm, ensuring adequate coverage. For vertical directions ("above/below"), we project the object's top or bottom surface onto the supporting platform. To mitigate overestimation from coarse 3D bounding boxes, we shrink the projection to 80% of its original size (centered), reducing overlap with nearby objects and better approximating usable space. To identify free space "between" two target objects, we define the search region as the planar area enclosed by the projections of both objects onto their shared supporting surface. This region is evaluated for occupancy by other objects. Across all spatial contexts ("above/nelow/between we enforce minimum free area constraint: the unoccupied region must exceed 0. 0.036m² (half an A4 sheet) in a top-down view. This threshold filters out trivial cases and ensures the queried space
45

---

can accommodate small objects such as book cup. Free space is computed by analyzing object footprints in the gravity-alignee top-down view. In Fig. 18, 22. 26, blue-shaded regions indicate the final sampling areas after accounting for bounding box scaling and occlusions. These visualizations highlight viable unoceupied regions for subsequent placement analysis.
Step 6: Projection and Visibility Filtering, Given candidate points sampled in the top-down view (XZ-plane) free space target locations, we project them into the original 2D camera image to assess visibility. Each point is assigned the coordinate of the platform's 's top surface, forming full 3D coordinates. Using camera intrinsics, extrinsics, and gravity alignment, we project these 3D points onto the 2D image plane, as shown in Fig. 19, 23, 27. To determine visibility, we compare the z-coordinate of each 3D point with the corresponding depth value the aligned depth image. Points are discarded as occluded if this difference exceeds 2.5 cm. For the front, behind, left, and right directions, we sample 9, 000 points per direction and retain the direction if least 2, 000 points remain visible. For the above, below, and between relations, sample 10, 000 points and require minimum of 000 visible points. We compute the mean position of the remaining visible points as the representative target location. If its depth deviates from the depth image by more than 2.5 cm, we instead choose the nearest point within this threshold. Fig. 20, 24, 28 illustrate this process, highlighting visible points and the final selected target (blue circle).
Bram
Sapa
Figure 18:
Figure 19: (b) Sampled points projected onto the 2D image
Figure 20:
(a) Top-view occupancy map with right-side search area
(c) Visible points with the final placement center
Visualization of right-side free space identification. (a) Toe-view occupancy map with the target's s right-side search area. (b) Projection sampled candidate points into image plane. (c) Non-visible points are removed: the final placement center is marked with blue circle.
Figure 21:
Figure 22:
Figure 23:
Figure 24:
(a) Top-view occupancy map of (b) Sampled points projected the object's 's bottom surface
(c) Visible points with the final placement center
into the 2D image
Visualization of bottom-side free space identification. (a) Top-view occupancy map of the target's 's bottom surface the platform. (b) Projection of candidate points into the image plane. (c) Non-visible points are filtered; the final placement center is indicated.
Figure 25:
B.2.6 Generating Diverse QA via precise 3D annotations
Building on the 2D QA generation pipeline (Appx. B.1.4), we utilize a template-based approach augmented with QwQ-32B to generate diverse QA pairs for 3D scenes. Unlike 2D images, 3D
46

---

Figure 26: *
Figure 27: *
Figure 28:
(a) Top-view occupancy map with between search area
(b) Sampled points projected into the 2D image
(c) Visible points with the final placement center
Figure 29:
Visualization of between-object free space identification. (a) Top-view occupancy map showing the area between two target objects the same platform. (b) Projection of sampled points into the image plane. (c) Non-visible points are filtered; the final placement center is indicated.
datasets offer rich and precise spatial annotations as depth maps, camera poses, and per-object 3D bounding boxes-which enable the construction of more complex QA with spatial referring with reasoning. Additionally, QA generation from 3D data accounts for above/below relations in both world and camera reference frames: the former reflects gravity, while the latter corresponds to vertical image orientation. Leveraging these annotations, we design both qualitative and quantitative QA templates grounded in the following spatial concept categories:
1. Relative Position Relations: Encompasses spatial relations such left/right, above/below, front/behind, inside/outside, touching/separated, and near/far. These queries require accurate 3D positioning and spatial layout to infer inter-object relationships in physical space.
2. Orientation and Rotation Reasoning: Involves reasoning over face/back direction, ori- entation (horizontal/vertical) and relative angles, using 3D object or camera poses (e.g., orientation vectors, rotation matrices) infer facing direction or viewpoint shifts.
3. Geometric Attribute Comparisons:
Covers attributes like size (big/small), height (tal- I/short), and width (wide/thin). These comparisons rely on true 3D dimensions, mitigating distortions from 2D projections.
4. Quantitative Spatial Reasoning: Involves computing depth, distance, relative angles, and spatial betweenness using precise 3D coordinates and metric reasoning.
5. Free Space Reasoning:
Identifies free space above, below, or between objects. As illustrated in Fig. 21, 25, and 29, blue-shaded regions represent unoccupied areas computed from object footprints and platform segmentation. To mitigate overestimation from large bounding boxes, we apply a shrink factor (e.g., 80%) to the projected surfaces for above/below queries.
6. Location and Placement Prediction for Spatial Referring: Involves predicting precise 2D coordinates from language descriptions, e.g., "Point to the second chair from the left" identifying target object, "Indicate free spot to the right of the white box the second shelf" - selecting valid placement location. These tasks require accurate 2D-3D projection and fine-grained spatial understanding, forming a vital bridge between visual perception and physical interaction and execution.
Building on the structured templates, we design a diverse suite 3D QA covering spatial reasoning, geometric comparison, viewpoint inference, environmental understanding, and coordinate-lewe localization. Leveraging QwQ-32B's powerful capabilities, our pipeline also generates complex reasoning QA pairs that are both structurally diverse and semantically rich.
3D training data visualization. For specific examples of 3D training data and their visualizations, please refer to Appx. F, which contains detailed sample presentations.
47

---

B.3 Synthetic Data Generation in the simulator
We want arm our model with multi- step referring capabilities with spatial reasoning. While 2D and 3D data enable single-step spatial understanding, they are less scalable for multi-step spatial referring with reasoning. Therefore, we generate synthetic data the simulator.
B.3.1 Indoor Scene Generation
Initial Scene Generation. We utilize Infinigen [137] to generate a large corpus of indoor scenes. To be specific, we configure the generation process to exclude small objects by setting compose indoors. solve_stepp small=0 to avoid pre-existing clutter on target surfaces. This allows us to reserve space for the subsequent placement of our curated 3D assets. This initial step yields over 3k unique indoor scenes.
Scene Filtering. The generated scenes underwent rigorous filtering process ensure their suitability for our downstream tasks. The primary filtering criteria included:
Selected scenes contain at least one sufficiently large, continuous tabletop surface (e.g., desk, table, counter) suitable for object placement. Scenes with absent or impractically small surfaces are excluded.
Adequate Tabletop Area:
Acceptable Lighting Conditions: Scenes with extreme lighting issues (e.g., darkness, oversaturation, unnatural hues) are discarded to ensure a viable baseline for subsequent lighting adjustments.
Scene Realism and Coherence: Scenes with severe geometric inconsistencies or implausi- ble layouts are removed to maintain physical plausibility.
Camera Accessibility: Scenes are required support feasible camera placement with clear views of the target surfaces. Highly cluttered or confined environments are deprioritized.
Scene Adjustments. To enhance diversity and control experimental variables, filtered scenes underwent automated modifications:
Lighting Randomization:
Light source intensities (e.g., ceiling lights, lamps) are uniformly scaled within [0.67, 1.47], where I denotes the original intensity.
Camera Pose Adjustment: For each tabletop, camera viewpoints are defined with pitch angles randomly sampled from [-600, -30 relative to the tabletop plane, oriented toward the region center.
Camera Height and Distance Variation: Camera height is uniformly sampled from 0.3- 0.8 m above the tabletop. Distance to the target area is adjusted to maintain full visibility, conditioned on surface size and field of view.
Some typical cases of scene filtering are shown in Fig.30.
B.3.2 3D Asset Selection and Preparation
Our 3D assets sourced from the Objaverse [138] LVIS dataset and undergo two-stage filtering process to ensure quality and relevance.
Stage 1: Category Filtering. We select objects based LVIS annotations, following categories:
Are typically placeable on surfaces.
Have maximum dimension under meter, suitable for tabletop scenarios.
Stage 2: Attribute-based Filtering, Next, we apply fine-grained filtering using attributes from the OrienTexx300K [7] dataset. Retained assets satisfy the following criteria:
Axis Alignment: Key features (e.g., edges, handles) align with canonical camera axes.
Single Object: Represents a standalone object, not scene or object collection.
Color Diversity: Contains colors beyond white or gray.
No Ground Plane: Excludes auxiliary visualization ground planes.
48

---

uroalved, isyout
The room lacks harrrmmmlakrssfffhoollligggg
object placerent pletformis big enough.
Figure 30: Cases of Scene Filtering
High Quality: Clean, well-constructed geometry without artifacts.
Canonical views (front, back, top, bottom, left, right) exhibit meaningful visual or semantic differences.
Distinguishable Views:
Reasonable Object: Represents common, identifiable object, not an abstract shape or an unidentifiable entity.
This process yields curated set of over 9k high-quality 3D assets.
Stage Manual Filtering. Due to the suboptimal quality of annotations, we perform manual verification based on the seven defined rules. Since object size estimation relies LVIS-labeled categories with #30% tolerance, we additionally discard instances whose irregular geometry (e.g., entangled cables of wired mouse) distorts the bounding box and hinders reliable scaling. After filtering, we retain over 3k high-quality 3D assets compliant with our criteria.
Fig.31 shows the screening results for some representative 3D assets, which encompass all the criteria mentioned above.
B.3.3 3D Asset Annotation with LLM
To generate diverse annotations, leverage Orion'ex orientation and caption data, processed via GPT-4o with tailored prompts to extract structured textual attributes for each sample.
Generated Attributes:
Orientation Descriptions: Prepositional phrases indicating the object's canonical front, salient parts, or intrinsic orientation (e.g., "on the front of ...", "on the handle side of ..."). suitable for insertion into sentence templates.
Color Labels: A single-word descriptor of the object's dominant color. If multiple colors are prominent, the attribute is marked "none" (e.g., "blue". "none").
Object Labels: Concise noun phrases specifying the object category (e.g.. "coffee mug". "computer mouse"), usable as subjects or objects in templates.
Category Consistency: A boolean flag indicating alignment between the object's visual category and its textual description.
LLM Prompt: The exact prompt used to obtain these annotations is provided in Listing 11.
49

---

The size too large to be placed on the indoor platform.
x It cannot be placed stably on the plane in the illustrated posture.
x it contains an object collection.
Although originaly cake model, is notidentifabll.
x The cord significantly affects normor mouse's bounding box.
x It includes auxiliary viualization ground plane.
x It's not aligned with axes. The mesh is not clean.
x it looks similar from different angles. contains only white/gray.
This a qualified 3D asset.
Figure 31: Cases of 3D assets Filtering
Listing 11: Prompt of generating brief description, color and orientation preposition.
Your objective is to generate four distinct labels derived from the provided asset information. Each is characterized by the following attributes:
Category: {category) Detailed description: (description) Direction hint: (direction_hint)
Based on the information above, are required to perform the following four tasks:
Task 1: Consistency Verification
Evaluate whether the asset's specified category aligns semantically with its detailed description. Output "True" if consistent, and "False" otherwise.
Task 2: Object Description Phrase Generation
ormulatee a concise and unambiguous object description phrase. This phraae must encapsulate the primary characteristics of the object, not contain any articles (Neg, "a", "an", "the"), and be grammatically suitable for use as either the subject or object vithin a template sentence.
Task 3: Simplified Directional Phrase Generation
50

---

Generate concise and clear simplified directional description phrase based the provided direction_hint. This phrase must capable of functioning as a prepositional phrase to describe the relative position of other objects within template sentence. In the generated phrase, the current object should be represented by "OBJECT".
Task 4: Color Extraction
Extract the dominant color the asset based on the detailed description, adhering to the following criteria:
1, The extracted color must be a single English word in lowercase. 2. If the detailed description does not explicitly state color. return the string "none".
Please ensure your output strictly adheres to the following JSON format. The output must be valid JSON object without any supplementary explanations, comments, introductory certtorryycoiingggggg remarks.
"consistent": (True/False),
"imple_desc"" "simplified object description phrase", "simple_dir": "simplified directional description phrase", colorr: "the main color"
Category
8ible
Description
The inages depict 30 nodel of reddish- covers and visthle pages. The covers worn, textured appearance while the sides show pages.
Simplo Description
Bible
Direction Prep
spine side of the 08]ЕСТ
Direction Hint spine
Direction Vec (8, -1, 8)
Figure 32: A Sample of generating object phrases, colors, and orientation descriptions from 3D asset information and orientation hints.
B.3.4 Scene Population and Data Rendering
The curated 3D assets are then programmatically placed into the filtered and adjusted indoor scenes. Asset Placement Strategy:
Assets are placed predominantly on identified tabletop surfaces.
To enhance the model's understanding of object orientation and inter-object relationships,
placement strategies included:
Increasing the proportion of objects with orientation vectors within the X-Y plane (e.g., laptops, teddy bears, mugs) in many scenes.
Increasing the co-occurrence of objects from the same category that have significantly different features. (e.g., ceramic cup with handle and cola cup). Ensuring plausible physical arrangements (e.g., no excessive interpenetration, objects placed upright).
The number of assets per scene varied from to 9.
Rendering Details:
Renderer: High-fidelity, physically- based images are rendered using the Blender Cycles.
Image Resolution: Output images are rendered at 960 x 540 pixels.
Render Quality: The conf igurierenderrr
render reeddr_cycles..... num. sam_sampes parameter is set to
2048 to achieve high-quality renders while maintaining reasonably controlled noise.
51

---

B.3.5 Question-Answer Pair Generation
Based on the rendered scenes and their corresponding ground truth information (scene graphs, object properties, masks), we generate question-ama -answerin (QA) pairs. This process involves two steps:
Step I: Generating Unique Object Referring Expressions. For each object in a scene, we formulate unique identifiers or referring expressions based combination of its attributes and relationships:
Feature Category: The semantic class of the object (e.g.. "the mug", "the laptop").
Color: The primary color of the object (e.g., "the red mug").
Left-Right Rank: Ordinal position from left to right (e.g., "third bottle from left right").
Front-Back Rank: Ordinal position from front to back (e.g., "farthest LEGO minifigure").
Distance Rank from Anchor: Ordinal position based on distance from salient anchor object (e.g., "the closest plate to the blue mug").
Height Rank: Ordinal position based the height (e.g.. "the second tallest teddy bear").
These components are combined to create unambiguous references (e.g., "the second red object from the left"). When filling objects into template structures in the subsequent process, we can randomly select from all referring expressions.
Step 2:
Applying QA Templates. Due to the heavy reliance spatial relationships between objects, we initially developed a template structure to formalize these relationships:
Position: Location of object relative to another (e.g., "on the left of the green bottle").
Orientation: Questions involving an object's intrinsic orientation (e.g., "on the handle side the red mug").
Distance Queries:
The precise distance (e.g., *0.2 meters the left of the plate"). Betweenness: Identifying object located between two other objects (e.g., "between the stapler and the telephone").
Specific Surface Locations: Locating objects relative parts of a surface (e.g., "at the far left corner of the table").
QA Template Types:
Locate from Description: Given a unique referring expression for an object, ask for its location (e.g.. "Give me the position ...""
Identify from Relations: Provide several spatial relationships an object satisfies and ask to identify the object (e.g.. "Please specify an object on the desktop that satisfies the following spatial constraints: ...").
Locate Empty Space: Define point in empty
surface based its spatial
relationships with surrounding objects, and ask to confirm this empty location (e.g.. "Please provide a point in the vacant area the desktop that simultaneously satisfies the following spatial conditions: ...").
Generation of Thought Processes (Reasoning Steps): For each QA pair, structured thought process or chain of reasoning is also generated. This involved selecting the pertinent pieces of information from the complete scene graph and ground truth data that necessary to arrive the answer. This recorded reasoning follows a predefined format.
C Implementation Details and Samples of RefSpatial-Bench
The RefSpatial-Ber -Brchhhl-Beehhhhhe benchmark evaluates spatial referring with reasoning in complex 3D indoor scenes through two tasks: Location Prediction and Placement Prediction, each comprising 100 samples. Each sample includes manually selected image, a referring caption, and precise mask annotations. Moreover, to evaluate the effectiveness of the RFT training strategy, we further select 77 samples from these 200 samples and define it as the Unseen set, which contains novel spatial relation combinations absent from RefSpatial to test the model's generalization.
52

---

Location Task: Given an indoor scene and unique referring expression, the model predicts a 2D point indicating the target object. Expressions may reference color, shape, spatial order (e.g., "the second chair from the left"), or spatial anchors.
Placement Task:
Given a caption specifying free space (e.g., "the vacant area to the right of the white box on the second shelf"), the model predicts 2D point within that region. Queries often involve complex spatial relations, multiple anchors, hierarchical references, or implied placements.
This set comprises 77 samples from the Location/Placement task, specifically designed to evaluate model generalization after SFT/RFT training on RefSpatial, it includes novel spatial relation combinations not present RefSpatial.
Unseen Set:
Notably, we introduce reasoning steps (step) for each benchmark sample, quantifying the number of anchor objects and their associated spatial relations that effectively narrow the search space. Specifically, each step corresponds to either an explicitly mentioned anchor object or a directional phrase linked to an anchor that greatly reduces ambiguity (e.g., "on the left of", "above", "in front of",, "behind", "between"). We exclude the "viewer" as anchor and disregard the spatial relation "on", since typically refers to an implied surface of an identified anchor, offering minimal disambiguation. Intrinsic attributes of the target (e.g, color, shape, size, or image-relative position such as "the orange box" or "on the right of the image") also do not count towards step.
A higher step value indicates increased reasoning complexity, requiring stronger spatial un- derstanding and reasoning about the environments. Empirically, we find that beyond 5 steps, additional qualifiers yield diminishing returns in narrowing the search space. Thus, we cap the step value 5. Instructions with step 2 3 already exhibit substantial spatial complexity. Detailed statistics on step distributions and instruction lengths are provided in the Tab. 11. To further show the diversity and reasoning complexity of RefSpatial-Bench we present representative examples from both the Location and Placement tasks. Fig. 41, 42, 43, and 44 show Location queries with varying reasoning step counts, where RoboRefer accurately localizes the target object (marked by blue dot). Similarly, Fig. 45, 46, 47, and 48 show Placement queries involving the identification of free space based on spatial relations. These examples highlight the step-wise complexity of the queries and the effectiveness of RoboRefer in addressing challenging spatial referring tasks.
Table 11: Statistics of the RefSpatial- across Location/Placemenl tasks and unseen sets.
RefSpatial-
Step
Samples
Avg. Prompt Length
Step
30
11.13
Location
Step
11.97
Step
32
15.28
Avg. (All)
12.78
Step
15.47
Step
28
16.07
Placement
Step
22
22,68
Step 5
7
22.71
Avg. (All)
17.68
Step 2
29
17.41
Step
26
17.46
Unseen
Step
24.71
Step
23.8
Avg. (All)
19.45
To more comprehensive evaluate the spatial referring task, we expand the original RefSpatial-Be -eacchhal-Benchh in terms of difficulty and diversity, producing the manually annotated RefSpatial-Expaie includes more indoor cases (e.g., shops and factories), and also introduces outdoor scenes not present in RefSpptial- Tab. 12 and Tab. 13. The detailed evaluation results of RoboRefer on this expanded benchmark are showed in Tab. 14.
It
(e.g., streets, parking lots, and parks). Statistics of this extension are provided in
53

---

Table 12: Statistics of the BetSpatial-Expand -eechhhnBBenchh
Task Type
Indoor
Outdoor
Total
Location Placement
126
241
120
80
200
Total
235
206
441
Table 13: Statistics of the RefSpatial--xxpan-BBeeee by step and task.
Task Type
Step
Samples
Avg. Prompt Length
Step 1
54
10.61
Location
Step 2
129
12.56
Step 3
58
16.10
Avg. (All)
241
12.98
Step 1
3
15.00
Step 2
86
15.14
Placement
Step 3
75
16.95
Step 4
29
22.24
Step 5
7
22.71
Avg. (All)
200
17.11
D Implementation Details for RoboRefer
D.1 Architecture
We adopt [38] base model, including visual encoder, an LLM, and multimodal projector.
Visual Encoder. We use the same image encoder as siglip-so400mppaee 1a:00000mpptcch14-11 patch14-449 [75] from NVILA [38], supporting 448 x 448 resolution for richer visual details. Rather than simply resizing the image a fixed resolution and producing the same number tokens, this image encoder processes inputs at dynamic resolutions, yielding more visual tokens from higher-resolution images via finer patch division. This enables fine-grained lision-language understanding. crucial for tasks like point prediction that require detailed perception beyond VQA. We further incorporate dedicated depth encoder, structurally mirroring the image encoder and initialized with its weights. It encodes relative depth maps special images, providing spatial cues to enhance 3D understanding.
Large Language Model. We adopt the Qwen2 LLM backbone from NVILA [38], which has been fully fine-tuned with extensive data during supervised training. This endows the model with rich visual knowledge, facilitating downstream 3D spatial understanding and reasoning tasks.
Multi-Modal Projector. To align multi-modal representations (e.g.. image to language, depth to language), use linear connectors, the same as NVILA [38], which is better than Q-Former, allow the LLM to focus on visual understanding and improve generalization. Separate connectors for image and depth embeddings ensure iedality-specifie processing and prevent cross-modal interference.
D.2 Training Data
Here we highlight the training data used at each stage, including the number of samples per dataset and the overall total. See Tab. 15 for details.
SFT stage. Specifically, in the first step of the SFT stage, i.e., depth alignment, we train depth projector to align depth and language space using the RefSpatial (RGB-D) dataset with 2.5M samples. To increase training efficiency, we slice multi-turn conversations (up 15 turns per sample), yielding 3.4M samples post-processing to train our model. In the second step. i.e., spatial understanding enhancement via full-parameter fine-tuning-we use both RefSpatial (RGB) and RefSpatial (RGB-D) datasets, yielding 6.8M samples after slicing. To further improve instruction- -ioruciion-folloinng and referring capabilities, we incorporate auxiliary datasets: 965k samples from instruction-tunet data (L.LaVA- 1.5 [134], LRV [133]), 321k from referring datasets (RefCOCO/+/g [87]), 176k from SAT [4]
54

---

Table 14: Accuracy results of 2B SFT and 8B SFT Models on PrSSpatial-Expand -ennch.
Task
Category
2B SFT
8B SFT
Over
50.21
61.00
Indoor Outdoor
49.57
58.26
Location
50.79
63.49
Step
61.11
72.22
Step
52.71
62.02
Step 3
34.48
48.28
Over all
48.50
60.00
Indoor
50.83
60.00
Outdoor
45.00
60.00
Placement
Step
33.33
33.33
Step 2
41.86
51.16
Step
54.67
70.67
Step 48.28
55.17
Step 5
71.43
85.71
Table 15: Details about the training datasets used in the SFT and RFT stages. D.A. and S.U.E denote the Depth Alignment and Spatial Understanding Enhancement step in the SFT stage, respectively.
Stage
Categorics
Datasets
SFT (D.A)
SSpatial
RefSpatial (RGB-D)
Spatial
RefSpatial (RGB), RefSpatial (RGB-D), SAT EmbSpatial [22] COCO [150], GQA [18], OCR-VQA [151], TexVQQ [152]. [153], LRV [133]
SFT (S.U.B)
General
REC
ReSCOCO/+/g [87]
RFT
Spatial
RefSpatial (RGB-D) w/ Reasoning Processing
benchmark training sets, and 127k from EmbSpatial [22] benchmark training sets. These additions help bridge distribution gaps between RefSpatial and benchmark-style queries. After slicing, the total number of samples used in this stage reaches 8.5M post-slicing.
RFT Stage. In the RFT stage, train the model using RefSpatial data annotated with detailed reasoning processes, including key intermediate steps and final answers. To ensure both training efficiency and effective learning, we moderately difficult samples (typically involving three reasoning steps), resulting in a 100k-sample dataset.
D.3 SFT Training Details
We formulate the SFT training stage as follows: given dataset D consisting of samples in the form of triplets (O, Q. A)), where O a sensor image (either RGB or RGB-D), Q is textual question, and A is the corresponding answer. The answer A may be direct response (e.g., point coordinate) or include intermediate reasoning steps (e.g., perceptual results followed by the final answer). The training objective is maximize the likelihood of generating the answer given the input pair (Q. A):
T
LEFT = - Σ log No ( |0, Q, vee),
where is the model's token distribution. The output model TSFT serves the initialization for the next RFT stage, ensuring a robust foundation for reinforcement learning.
To be specific, our SFT consists of two steps. In the first step. depth alignment, only the depth projector updated by using the RefSpatial (RGB-D). We employ a maximum learning rate le-4, weight decay of 0, and warm-up ratio of 0.03. The 2B variant is trained with batch size of 7 per GPU, and the variant with both for one epoch. In the second step of spatial understanding enhancement, fine-tune all model parameters using the datasets described in Sec. D.2. Training
55

---

conducted for one epoch with maximum learning rate of 5e-5. We use batch size of 6 per GPU for the 2B model and for the 8B model. Other hyperparameters follow those in the first step. For more details, please refer to NVILA [38] settings during alignment and SFT.
D.4 RFT Training Details
During the RFT stage, we refine TSFT via GRPO [114]] a reinforcement learning method designed for efficiency and scalability. Unlike PPO [154], which relies costly value network, GRPO esti- mates relative advantages by comparing intra-group rewards, reducing computation, and simplifying optimization. This makes it well-suited for reasoning inttensive spatial referring tasks. In detail, modify R1-V [122] to support our 3D-aware architecture. Training is conducted for two epochs with batch size of 1 per GPU and outputs in GRPO. For details about hyperparameters, see R1-V [122].
D.4.1 Sampling Action Groups
Given an input state = (0, Q), where O denotes the visual encoding of the RGB or RGB-D obser- vation and Q the textual encoding of the question, GRPO samples a set of actions {a), 02,... (x) from the current policy na, initialized from TSFT. The sampling process is:
a, (a O, Q) for i 1,2,......
This strategy ensures diverse responses, promoting exploration and preventing premature convergence.
D.4.2 Reward Design and Policy Update
Each sampled action a, is assigned a reward R(a;) based on verifiable criteria, yielding a reward set "1. P'y. For spatial referring tasks, R(a:) integrates two outcome- based and proposed two process-based components. The outcome-based reward functions are defined as follows:
Outcome Format Reward RoF. This component ensures structured and interpretable outputs by requiring the model to predefined format: reasoning within "<think> ... </think>" and the final answer in "<answer> ... </ /aasser>". A reward is assigned for strict compliance, 0 otherwise.
Point L1 Reward Rp. This component evaluates the accuracy of the model's final point prediction by comparing it with the ground truth from the annotations of RefSpatial. Following the criterion inspired by Seg-zero [131], stricter reward of is assigned if the L1 distance between the predicted and ground-truth points is within 50 pixels; otherwise, the reward is O.
Notably, most process-based rewards depend on a Process Reward Model (PRM), typically fine- tuned LLM or VLM tasked with providing feedback. However, applying such an approach in our (1) LLMs cannot process images, making it impossible to determine whether predicted coordinates match the target object. (2) Although VLMs integrate visual and textual information. prior work [155] has shown they may lack precise visual understanding when dealing with textual coordinates. Since the correct assessment of predicted coordinates is paramount for reward assignment, additional or specialized methods are needed to ensure reliable feedback.
setting presents main challenges.
To address this issue, we propose a rule-based process reward for spatial referring that obviates the need for a Process Reward Model. Our approach directly evaluates key intermediate perceptual steps using the ground-truth step-wise annotations provided in RefSpatial. This contrasts with most existing methods on process-based rewards [156, 157], which emphasize strictly sequential reasoning and rely on a PRM for evaluation. In contrast, our method employs metric-sensttive rule-based process reward functions to assess intermediate perceptual results in an order-invariant manner. Our key insight lies in two aspects: (1) Metric--ensittvity; Different spatial attributes require distinct metrics due inherent differences in their representations (e.g., points for positions, vectors for orientations). (2) Order-invariance: The reasoning process in spatial referring is not strictly sequential; for instance, identifying the position of the keyboard or the mouse first does not affect the final interpretation of "the free area between the keyboard and the mouse".
In detail, have two process-based reward functions:
Process Format Reward RpF. Similar to the Outcome Format Reward strategy, this compo- nent enforces structured and interpretable reasoning process, thereby facilitating accurate reward
56

---

computation. In particular, the model is required to produce outputs in the following format:
[Perception Type] [Target Object]:
[Value]
(3)
where "Perception Type" must be one of three categories: "Position", "Orientation", or "Size". The "Target Object" corresponds uniquely identifiable entity (e.g.. "the second largest cup" or "the second large cup from large to small"). The "Value" depends on the selected "Perception Type":
For "Position", the value should be normalized 2D coordinate of the form [(x, y)], where both x and y lie in the interval [0, 1]. rounded to three decimal places.
For "Orientation", the value is a 3D unit vector (x, y, 2) representing the object's semantic orientation in the camera coordinate system.
For "Size", the value represents scalar measured in meters.
Below examples illustrate the expected format:
[Position] [the second largest cupl: [(0.245, 0.147)]
[Orientation] [the handle of the second largest cup]: (1.000, 0.000, 0.000)
[Size] [the second largest cup]: 0.12
Accuracy Reward RAce- The reward is computed only for steps annotated as key steps in RefSpatial. In detail, we use regex matching to determine whether the "Target Object" in the current process format appears in the key-step annotations. If not, the step receives reward. Since the model has already undergone a cold-start phase SFT, can interpret instructions and identify relevant target objects. Thus, failed match implies that the model cannot accurately refer to the object linguistically, and no reward assigned. For each perception type, we apply specific metric to compute the reward: (1) "Position": If the L1 distance between the predicted point and the ground truth is below 50 pixels, the reward is 1; otherwise, O. (2)
If the cosine similarity between the predicted and ground-truth vectors exceeds 0.8, the reward 1; otherwise, O. (3) "Size": the predicted value falls within 115% of the ground truth, the reward is 1; otherwise, 0.
We prioritize the correctness of the final outcome over intermediate steps. To prevent reward accumulation from multi-step processes, we scale the process reward by 0.25. The final reward function is defined as:
RoF(a:) + Rp(a.) aRpF(a.) aRAcc(o.)
(4)
where a is set to 0.25. By normalizing the rewards within the sampled group, we obtain the set of relative advantages {A1, Ag,..., AN defined
A; r: mean {r,})
std( (r;))
which measures how each reward compares to the mean units of standard deviation. We then update the policy based these advantages, reinforcing actions with higher relative advantages while reducing the likelihood of those deemed less effective. To ensure stable reinforcement learning, the update is further constrained minimizing the KL divergence between the updated policy and its reference counterpart, thereby promoting incremental and controlled policy adjustments.
E Experimental Setting and Details
E.1 Experiments Compute Resources
We conduct experiments A100 GPU cluster, with each node equipped with 8 GPUs.
2D Web Data Coarse Filtering. We perform the initial coarse filtering 1.7M OpenImages using SigLIP2. The process runs on node and takes 8.5 hours and yields 933k high-quality samples.
57

---

2D Web Data Fine-grained Filtering. We further filter 933K samples using Qwen 2.5- VL 7B to ensure high visual quality and spatially relevant QA pairs. The process is conducted on I node (two models per GPU) over 2.5 days, yielding approximately 845k high-quality samples.
Pseudo-3D Scene Graphs Construction. We construct pseudo-3I) scene graphs for 845k samples using 3 nodes, requiring 10 hours for depth estimation and another 10 hours for camera parameter extraction, segmentation masks, and point cloud generation. Additionally, we generate object-level captions for all instances using 4 nodes over 18 hours by using Qwen 2.5-VL.
Reasoning QA Generation from 2D data source. To enrich factual statements with contextual scenarios, we employ QwQ-32B to construct reasoning QA, utilizing 4 nodes over 3.75 days.
3D Data Filtering and Scene Graphs Construction. Given the limited amount of 3D data (100k) and the availability of precise annotations, only 2D bounding box bidirectional matching required. This process is completed in 3 hours using 1 node.
Reasoning QA Generation from 3D data source. To enrich factual statements with contextual scenarios, we employ QwQ-32B to construct reasoning QA, utilizing 4 nodes over 1.5 days.
Synthetic Data Generation in Simulator. We use 4 RTX 4090 GPUs to generate data for one week.
Depth Alignment in SFT. The process is conducted on 10 nodes over 12 hours for 2B variants and nodes over 40 hours for 8B variants. Both variants training use ZeRO-3.
Spatial Understanding Enhancement in SFT. The process is conducted 10 nodes over 2 days for 2B variants and 10 nodes over nearly 1 week for 8B variants. Both variants training use ZeRO-3.
Spatial Referring in RFT. The process is conducted on I node over 3 days for 2B variants. However, our model over twice as slow as other Qwen 2/2.5-VL-based methods [122, 130], mainly because they process only a single RGB image during training and can leverage vLLM for group inference acceleration. In contrast, our method requires RGB-D inputs and modifies the original NVILA architecture, making it incompatible with vLLM or SGLang acceleration.
E.2 Spatial Understanding Benchmarks
We evaluate several public single-step spatial understanding benchmarks, including CV-Bench [15] (2D Spatial Relation, 3D Depth Order, 3D Distance), the BLINK [16] validation set (Spatial Relation, Relative Depth), RoboSpatial [2] (configuration). SAT [4], and EmbSpatial [22], following their official evaluation protocols. We exclude non-spatial tasks from our evaluation, such as 2D Counting in CV-Bench and Art Style or IQ Test in BLINK. Since all these benchmarks are multiple-choice tasks, we report accuracy as the evaluation metric.
We compare 3 categories of models: (1) proprietary VLMs, such as Gemini-2.5-Pro [9], which show strong spatial perception, as shown in the Gemini-Robotics [107] paper; (2) open-source VLMs trained on general VQA datasets; and (3) spatially specialized models trained on spatially relevant datasets, offering basic spatial understanding.
E.3 Spatial Referring Benchmarks
We evaluate three recent robotic referring Macchnarks-RobRRee [140], Where2Place [5]. and RoboSpatial [2], all limited to 2 reasoning steps. Specifically, RoboReflt concentrates on object loca- tion referring by leveraging object attributes and spatial relations with anchor objects. Where2Place further explores how to place objects relative to anchor objects in the camera's 's coordinate frame. RoboSpatial builds on this idea, considering the same form of placement relative to anchor objects, but many samples in benchmarks consider the object-centric coordinate frame. We also evaluate challenging benchmark based on
more complex multi-step spatial referring RetSpatial-Bench real-world cluttered scenes, as introduced in Appx. C. For all these benchmarks, we use the same evaluation protocol: first compute the proportion of predicted points that fall within the ground truth mask for each sample, and then average the results across all samples to obtain the success rate.
We compare two main categories of models: (1) Proprietary models (i.e., Gemini-2.5-Pro with strong spatial referring capabilities, and (2) spatially specialized models trained on spatially relevant datasets, exhibiting basic spatial referring abilities.
58

---

Foed
Code
Click
Tall
Figure 33: Map Visualization (RViz).
E.4 Simulation Evaluation
We use the same evaluation protocol Open6DOR V2 introduced in SoFar [7], following the official repository. Specifically, we only test the position track as this work focuses on location and placement via spatial referring rather than executing 6DOF manipulation tasks. Notably, we find that our model achieves nearly 100% success in the perception stage (i.e., determining location and placement), with failures primarily attributed to motion planning errors such as IK failures or collision- prone trajectories, We show more demonstrations of simulation evaluation in Appx. F.
E.5 Real-world Evaluation
E.5.1
URS Manipulation
We show two demos for URS Manipulation: human disturbance and voice interruption. In the human disturbance case, RoboRefer runs at 2.5Hz. Significant shifts in predicted 2D coordinates trigger motion interruption and re-planning. In the voice interruption case, incoming speech commands are continuously monitored. Upon detection, the current task is halted. We use the Whisper [158] ASR model to transcribe speech, which RoboRefer processes into new 2D coordinates for task redirection.
For grasping, the 2D coordinates are fed into SAM2 [78] to generate a segmentation mask, which filters the target object's point cloud from the scene captured by third-person Intel RealSense L515 depth camera. The extracted point cloud is input to AnyGrasp [159] to predict grasp pose in the camera coordinate frame. Using an eye-to-hand calibration method, the grasp pose is transformed into the URS robot's 's base frame for execution.
For placement, RoboRefer predicts the 2D placement point, which converted 3D coordinates using the camera's intrinsic parameters and depth data. The 3D point is then transformed into the robot's coordinate system guide the placement action.
E.5.2 G1 Humanoid Mobile Manipulation
For grasping, we employ head-mounted Intel RealSense D435 the Unitree G1 humanoid to capture RGB-D images, which are processed by RoboRefer to extract 2D target coordinates. These coordinates guide SAM2 [78] to generate segmentation mask, which filters the third-person D435 point cloud to isolate the target object. The filtered point cloud is then passed to AnyGrasp [159) to
http://gghhbbccmmm///gggeelol/// DOwp.eeD.......oooOOoooo V2 Execution
https//wwww.mivvrsalllcoo row.toooooooooooooooo http://sssssooooooooooooooooooie c
59

---

Olinu
FD00 11113
Fest Soice
Figure 34: SLAM Navigation (RViz).
Table 16: Ablation on the combination of RGB/RGB-D from RefSpatial for SFT (i.e., spatial understanding enhancement). Top-1/Top-2 accuracies are represented using bold text, and underlines.
CV-Bench [15]
BLINK val [16] 2D-Relation
Method
Input
2D-Relation
3D-Depth 3D-Distance
3D-Depth
Only RGB-D from Re(Spatial
RoboRefer-2B-SFF
RGB
87.69
86.83
82.50
79.02
81.45
Combination of RGB and RGB-D from Re(Spatial
RoboRefer-2BB-35
RGB
96.15 96.31
95.83
90.67
83,92 87.41
88.71
RoboRefer- 2B-FFF
RGB-D
97.17
90.83
91.13
predict a grasp pose in the third-person frame, which transformed to the robot's base frame using known camera-to-robot calibration.
For navigation, the chest-mounted L515 camera continuously captures images used by RoboRefer to detect nearby landmarks (e.g., a table near the robot). The resulting 2D locations, combined with depth and intrinsics, are projected into 3D world coordinates and integrated into global map via FAST LIO__LLO_L____ LICALLLLIIIZTOOON HUMANOID for SLAM-based navigation, as shown in Fig 33, 34.
For placement, the target placement region. The corresponding 3D coordinates, computed from depth and intrinsics, are transformed into the robot's base frame for accurate placement execution.
D435 captures images processed by RoboRefer to localize the
E.6 More Ablation Studies
We conduct additional ablation studies to identify which design choices enhance the performance.
ResSpatial RGB and RGB-D combination for SFT training encourages the image encoder to learn spatial understanding beyond depth cues. Tab. 16, incorporating both RGB and RGB-D data from RefSpatial in the second stage of SFT training effectively enhances the image encoder's spatial understanding. contrast, training solely with RGB-D may lead over-reliance on the depth encoder, limiting the image encoder's s ability to learn spatial cues from RGB images alone.
http://gghhibboomm/eeegggllffffn L0t.A///...............L.L.LAAAO HUMANOID/ree//main ou
60

---

More Demonstrations
Visualization of ResSpatial. We present dataset examples in Fig 36, 38, 39, which cover 31 distinct types of spatial relationships.
Visualization of Simulation Evaluation. We present example rollouts of RoboRefer in Fig. 40.
Visualization of ResSpatial-Bentt
We present examples of location in Fig. 41, 42, 43, 44 and
placement in Fig. 45, 46, 47, 48 with RoboRefer predictions.
Visualization Simulator. We present example rollouts with RoboRefer predictions in Fig. 40.
Visualization of Real-world Evaluation. We present examples in Fig. 49, 50,
G More Discussion on Limitations and Future Work
Despite achieving promising results, our model still has limitations. In particular, it relies on precise textual descriptions to pinpoint specific object locations and placement targets, including accurate references to anchor objects. However, in practical real-world robotics scenarios, human instructions are often ambiguous. While such utterances may still imply unique location, resolving them typically requires sophisticated visual-linguistic reasoning and a process of elimination grounded in human prior knowledge, capabilities that challenge current model.
We show two representative but interesting examples that highlight the need for human intent understanding and visual-linguistic reasoning: (1) Probabilistic Preference. As shown in Fig. 35, humans may refer to sushi plate as "pick the one facing the drink". In the depicted scene, there are four drink bottles, yet only the two in the middle align with the second sushi plate from the left in the farthest row; the leftmost drink aligns with the first plate, and the rightmost with the third. Despite this ambiguity, people often judge the second plate to be the intended reference due to its higher likelihood of being aligned with two out of the four drinks, reflecting a probabilistic bias in interpretation. (2) Spatial Compatibility. As shown in Fig. 35, user might instruct. "Place another sushi between the plate and the soy sauce dish " Although multiple plate-soy sauce pairs exist, only the pair closest to the observer affords sufficient physical space to place another sushi plate. Thus, implicit spatial feasibility guides the correct interpretation, even without explicit constraints.
Our model struggles with these cases because RefSpatial lacks multi-step referring data that embeds human priors and intent understanding. As RefSpatial is procedurally generated, incorporating such characteristics at scale remains challenging. Future work may explore procedural synthesis of intent-aware data or improve model performance via co-training with datasets supporting intent comprehension, such as PixMo-Points [10].
H Broader Impacts
RoboRefer can serve as versatile visual assistant with advanced spatial understanding and reasoning. Due to its integration with large language models (LLMs), inherits both potential benefits and risks, similar to other VI.Ms, such as output hallucinations, biases from base models, and heightened energy consumption associated with model upscaling. Beyond these considerations, RoboRefer can also function as high-level planner with spatial referring abilities, guiding robots in tasks like manipulation and navigation. While such capabilities substantially enhance robotic control, they also pose safety challenges when combined with existing control policies.
Despite these concerns, releasing RoboRefer the broader research community would be highly advantageous. Open access would foster continued development and refinement of spatial referring with reasoning, ultimately benefiting diverse robotics platforms (e.g., robotic arms, humanoids) performing various tasks (e.g., manipulation, navigation).
I Licenses
(1) 2D web image data: OpenImages [135) is released under Apache License 2.0.
(2) 3D embodied video data: CA-IM [136] is released under NNC
61

---

6B
Figure 35: Discussion about the limitation of human prior knowledge and intent understanding.
(3) Procedural scene generation: Infinigen [137] is released under BSD 3-Clause License.
3D digital assets: objaverse-xl [138] released under Apache License 2.0.
62

---

Left Right
a- can you confirm matte purple fabriz belly dancer is postioued to the loft onange sheer fabric belly dancert (A) yes (80) no
(B) Indved, matee purple fabric bally dancer is on the left sice of orange sheer fabric y hoor.
O: Select point located B, 178m on the left of tiigtth Bibble the cleditups
D: the image, point out the rightmost detergent suplly pieel cucrcinate. (89.657, 8. 8.393)
A 08 447, 8.752)
you corfirm grey
Pinpoint point the grean matte plant center, acting as the housopplann startingfros from the left.
is positioned right of light
brown cardboard box on the right side white plastic power outlet from the wiewers perspective?
jeans st centent (A) yes (8))
A: Indeed, cotton shirt
Close & Far
Q: Based on relative distances, which object mesrer to the battle (highlghoedd by red boo?? Choose from barh towe! (highlihtted by blue bex) and clothing thighighttd green bocl (A) beth clothing
A: (R)
Which forther from the hars?? (A) blue denim juars white motte coutty
O: Whicholjett the securd clossst to the sculpt Pleea proviee ia 20 courdinates.
A: (J. 556, 8. 8.3221
one positioned from the stool (righligheed by red box? Select from the kitchen apolarcos (highlyhtedd by slue bod end the oven (ighligheed by green bext (Nlkechen applionces (Bloven
D: Which dkjjett close the dark left among detted right (B) brown docted de right checke
out which onjjett the forthent gul.
(A)
A: [ja. 8.6821)
Depth
Above & Below (World)
9.522)
at compared motte at
real-word blue
cameru?
real perspective,
light gray fabric (B)
A distance of 18 meters exists between point (9.482, 0.522) and the
No, mette black left front matte hatchback center.
8.7791
(AI Correct.
Figure 36: Visualization of RefSpatial.
63

---

Above & Below (Image)
From the image's perspective, would you say gold textured tray center positioned over glossy white cabinet at upper center?
A: Yes. Gold textured tray center appears below white cabinet in the image.
image's perspective, chair
at right below gold brass saxophone center? (A) no (B) yes
A: (B) yes.
image's perspective, wooden chair
O: From the image's perspective, is black iron streetlight at upper left above white glossy car right?
at right below gold brass saxophone center? (A) no (B) yes
Yes. Gold textured tray appears below glossy white cabinet upper center the image.
A: Indeed, light woad/wooden chair at right positioned under gold brass saxophone center.
Tall & Short
O: You organizing display of tobacco tins need to stack them vertically. Given you a space can only accommodate items to the height of the taller object between the red matte tin can at left and the matte plastic peterson university flake center, which tin you choose
Q: Where is the tallest vodka bottle with a red label located? Please provide coordinates.
Q: Give position of the shorter bottle with a blue cap. Please provide its 2D coordinates.
A: (16, 8.258)]
A: [(0. 383, 0.398]]
Which object taller, stone fireplace at center the wooden photo frame at on the side?
fits?
A: The matte plastic peterson university flake at center taller than red matte can left, choosing the matte plastic peterson university flake at center ensures fits within the space designated for the taller object.
A: The stone fireplace at center is taller than wooden photo
Q: transparent glass center, which is the second glass jar from left right, shorter than transparent glass jar at lower left? (A) no (B) yes
Which these plastic recycling bin at right or dark blue quitted mattress at lower left, stands shorter?
height blue quilted mattress lower left.
A: Yes.
Figure 37: Visualization of RefSpatial.
64

---

Between
Rotation (Horizon & Vertical)
Q: A white paper towel found between tronslucent
transparent glass jar conter, glas jar from left to right, in the front- back direction. Please provide 2D cocrdinates.
O: Please mark a point on the desktop that les between the witeoost object and the beer bottle, and is located the with handls
directly face white
look the white should
coffee
should
therighh?
(10,526, 8.6933)
A: (88.636, e. e.n05))
should notate down by 18. 181 degrees.
19. 945 degrets.
to rouate wround
Free Space
Corner & Edge
Please location within
the vacont space on the white mette table center.
Please choose spot from space belowen comera and the right camera.
Q: Select point that pestiioned the free space the deskenp
Q: Please denote the thermos bottle pestiioned at the far left desknnss
539.8.772)
9.985)
8.655)
352, 2.28)]
Angle
Distance
facing when sorted from left to right, and the focing direciin of brushed metal refriguraie elaor at left?
angle
green
center planed as the
brown rough bark
white stone
chenub at center. there noough space between them to ft the statue without moving ether object?
A: The facing directions of green motte carton center, placed as the second bottle when sorted from back to frant, and brushed metal refrigerator at left 93 degrees
The distance between the brown rough bark tree at left and the white stone cheroh center 8.82 feet, which much more than enough accommadate the fest wide statue whou needing move either object. obyert
Front & Behind
O. Which object closer the comena, the white fabric trash bin cencer or the wooden photo frame left?
Q: Is the postion of metallt: chair more distant than that of white radiator right? yAl no
Di Give position the nvonest shue orange Please 20 coordinates.
A: A: 6.523)
wooden photo
(:)
cark light jeans
O: Puint out the which found beiinn! fabric lower left.
Ui Provide the coordmaates location the free desktop fulfills both the following onioria. of rightmase papsicenn Bulind the
center bleck shirt center
buee t-shirt
[18.127, 0.730]]
A: [19 476,0476.... .4223
Figure 38: Visualization of RefSpatial.
65

---

Big & Small
Considering the relatve sizes of shiny metallic left and brushed metal fauces the provided, metalic metal laccet upper (A)
Or golder-brown bread at center higger than black metal watch at leftl (A) yes (8)
(Aries golden- brown laggrr in size block
no 1i0
wetchal upper
a: dering the
doughnut!
laptop
provide coordinates.
checkered paper nocebook on the desk, whic one should place first to ensure they both tably?tably?
0.518)
A: The laptop is langer; you should oce the laptop first ensure both fit comforially the desk.
a: Does the black fabric baby
confirm
rage compared fuffy
upper
smoiller yellow center? (A) yes (B)
(A) yes (B)
Incorrect, neon glass cola sign se upper center smalier in size than yellow motte taxt at center.
A: Yes, the black fabnc baby lower center
achtray.
fluffy white stuffed animal
Please provide its coordinates.
(19.448, 0.346)
Face & Back
Wide & Thin
Q: the white pillow lower the 1th
picture frame
stupper left?
beige upper right nlmedl wwodenn photo frame left? (A) yes Ar/B/Vss.
picture away from white fabriz lampchade at left? (A) yB)
greater compared
atrigh? (A) yR)
Ves, the pilow
frame..
A-Yes.
Qi Is the deer head chinner
no (B)
G: Select point in vacant space lnceted on the focing side of the cap.
C: Locate a point within free spece wit sitweed on the back side the Sporgelobb SquorePants.
cer
(Bi)es The caromic deer head is chinner.
silver (R) no
yes
No.
[8.478, 0.5653)
A: 626, 8.423]]
Touch & Far from
Inside & Outside
Q. white in natte rodiator at lower left inside fora falric curtein left/ (3)
large between the natural woven alacemat at lower center and green leafy plant et upppe
you say at
white
right center?
touching It no yes
A (A) Yes, flral fabrie curtain left surrounding white maste reciittor al lower lnfi.
A: (A) No.
A Ves.
A A-Yos.
Figure 39; Visualization of RefSpatial.
66

---

Place the cup in front of the spatula the table
Location
Placement
Pick-Move Place Execution
Place the cup behind the skillet on the table
Location
Placement
Pick-Move Pocc-eeee Place Execution
Place the tissue box to the left of the wineglass on the table
Location
Placement
Pick-Move-Plade Execution
Place the wineglass the right of the hard drive the table
Location
Placement
Pick-Move-Plasd Execution
Place the skillet between the hard drive and the box on the table
Location
Pick- Maok-MovePPae Place Execution Place the shoe at the center of all the objects the table
Placement
Location
Placement
Pick-Move-Place Execution
Figure 40: Visualization of RoboRefer's prediction (blue point) on Open6DOR V2 [7] benchmark.
67

---

Reasoning Step
Reasoning Step 1
Reasoning Step
Please point out the box with the person logo the picture.
Please point monitor closest the image.
Please point out third chair left to right.
Reasoning Step 2
Reasoning Step =
Reasoning Step = 3
Please point the top piece of paper the white table.
Please point to the farthest white cabinet the picture.
Please out the black framed painting the right of the lamp.
Reasoning Step= Please point out third object from left right on the closest platform.
Reasoning Step =
Reasoning Step =1
Please point out sofa the right side of the picture that closest the viewer.
Please point to the wooden plate on the left of the picture.
Reasoning Step
Reasoning Step
Reasoning Step
Please point to the rightmost box in the picture.
Please point out the second silver from left right the picture.
Please point out the painting hanging on wall.
Figure 41: Some Location Examples. The model asked identify the object referred by prompt. The blue point shows the RoboRefer's prediction (all correct).
68

---

Reasoning Step
Reasoning Step 2
Reasoning Step 2
Please point out the blue toothbrush forthest the faucet.
Please point out the card closest wooden door.
Please point to third card from right left on the cabinet.
Reasoning Step 2
Reasoning Step =
Reasoning Step = 2
Please point out the object between the white box and the farthest black pot picture.
Please point to the pillow closest to the remote controller.
Please point out the closest the TV.
Reasoning Step=
Reasoning Step
Reasoning Step
Please point the leftmost black object the same platform the micro-wave
Please point out the orange box on the white table on the left.
Please point out the yellow five- pointed star in the picture.
Figure 42: Some Location Examples. The model asked identify the object referred by prompt. The blue point shows the RoboRefer's prediction (all correct).
69

---

Reasoning Step
Reasoning Step
Reasoning Step 2
Please point out the tallest bottle on the black table.
Please point out the farthest chair from viewer.
Please point out the silver bottle on right edge the sink.
Reasoning Step 3
Reasoning Step =
Reasoning Step = 2
Please point out the object under the table.
Please point to the rightmost blue box on the refrigerator.
Please point the pillow closest the right nightstand.
Reasoning Stepp
Reasoning Step
Reasoning Step
Please point the cardboard box under the bed which closest box the viewer.
Please point out the blue object on the table in the picture.
Please point to the white box closest to the remote control.
Figure 43: Some Location Examples. The model asked identify the object referred by prompt. The blue point shows the RoboRefer's prediction (all correct).
70

---

Reasoning Step Please point out the second object from right left the platform with the banana.
Reasoning Step 1
Reasoning Step 1
Please point out the farthest chair from viewer.
Please point out the orange box picture.
YORK
Reasoning Step 1
Reasoning Step =
Reasoning Step 3
Please point the remote control the right side of the picture.
Please point the stool which the third stoolfrom the front.
Please point out the white object the first shelf of the cabinet.
Reasoning Step
Reasoning Step
Reasoning
Please point out the plant the left of the second shelf of the cabinet.
Please point out the blue object on the third level the wooden shelf.
Please point out the paper tube closest to the viewer.
Figure 44: Some Location Examples. The model asked identify the object referred by prompt. The blue point shows the RoboRefer's prediction (all correct).
71

---

Reasoning Step
Reasoning Step
Reasoning Step =
Please point out the free space between the black cloth to bottom-right of the monitor the keyboard.
Please point out the free spot halfway between the blue cup and green cup place another identical
Please point out the free space between the black water bottle, the pot tid, and the scissors.
Reasoning Step
Reasoning Step 4
Reasoning Step =
Please point out the free spot to the left of the can, where an object of the same size can be placed at an equal distance between the can the bag.
Please point out the free space mid midway between the first and second green from the left.
Please point out the free space the direction the logo of the closest white bottle to the viewer.
Reasoning Step
Reasoning Step
Reasoning Step
Please point the free spot, equidistant from both the blue bowl
Please point out the free space
and the red bowl, and between them, where similar-sized bowl can be placed.
between the red box on left black box.
Please point out the free space the facing direction the purple bag.
Figure 45; Some Placement Examples. The model is asked to identify a valid free space based on spatial reference. The blue point shows the RoboRefer's prediction (all correct).
72

---

Reasoning Step
Reasoning Step
Reasoning Step = 3
Please point out the free area in the direction of the handle of the rightmost green
Please point out the free space in front of the brown object on the shelf.
Please point out the free space facing direction of orange box.
Reasoning Step 4
Reasoning Step =
Reasoning Step
Please point out the free space on the table between the pillow and the brown bowl.
Please point out the free space below the white table.
Please point the free space below the black shelf.
Reasoning Step=
Reasoning Step=
Reasoning Step
Please point the free space between the white chair and the brown shelf.
Please point out the free space inside the closest pot.
Please point out the free space below the table.
Figure 46: Some Placement Examples. The model is asked to identify a valid free space based on spatial reference. The blue point shows the RoboRefer's prediction (all correct).
73

---

Reasoning Step
Reasoning Step Please point out the free space between the scissors and the microwave.
Reasoning Step 3
Please point out the free space in direction of white side of the can.
Please point out the free space between the bathtub the toilet.
Reasoning Step 4
Reasoning Step =
Reasoning Step
Please point out the free space in front of the blue box which is on top level the shelf.
Please paint out the free space front of the white vase which on the top level of the shelf.
Please point out the free space the right the farthest pot.
Reasoning Step
Reasoning Step
Reasoning Step =
Please point out the free area on the table in facing direction of the second chair from the front on the right side.
Please point out the free space between the black plate, blue and closest water glass.
Please point out the top corner of the table.
the free space in the
Figure 47: Some Placement Examples. The model is asked to identify a valid free space based on spatial reference. The blue point shows the RoboRefer's prediction (all correct).
74

---

Reasoning Step Please point out the free spot between the blue water kettle and the orange.
Reasoning Step = Please point out the free space between the mouse and the green
Reasoning Step 3
Please point out the free space between the pot and the black box.
Reasoning Step 2
Reasoning Step =
Reasoning Step
Please point out the free space on the third level from the top of the cat tree.
Please out the free area in the top-left corner of the table.
Please point out the free space the lowest shelf of the cat tree.
Reasoning Step=
Reasoning Step
Reasoning Step
the/free Please point out the free area on the
Please point the free space on the table between the speaker right of the monitor and
Please point the free space right part the table between mouse and the picture frame.
table that the first chair from the front on the left side is directly facing.
Figure 48: Some Placement Examples. The model is asked to identify a valid free space based on spatial reference. The blue point shows the RoboRefer's prediction (all correct).
75

---

Reasoning Step
Reasoning Step
Reasoning Step
Find loccatio blank space located 0. meters to the left of the table.
Where should additional panda be placed in line? Please mark the location the desktop.
Identify the panda that is the closest to the cup.
Reasoning Step
Reasoning Step
Reasoning Step
Please choose spot located the free space between the car and the flashlight.
Locate point within vacant space located 0, e5 meters the front- facing side of the teddy bear.
Locate point within vacant space located 13 meters the right of the cup.
Reasoning Step 2
Reasoning Step
Reasoning Step 3
Locate point within the vacant space located on the handle side the cup.
Please indicate the cup located at the left edge the desktop.
Point to the free space between the green and pink plates.
Reasoning Step=
Reasoning Step = 2
Reasoning Step =
Please indicate the cup located the near-left corner on the desktop.
Please indicate the hamburger located at the far-left corner on the desktop.
Point to the hamburger between the car and the mug.
Figure 49: Visualization of RoboRefer's prediction (blue point) in the real-world evaluation.
76

---

Reasoning Step
Reasoning Step=
Reasoning Step = 2
Please select point located in the vacant area at the far edge of desktop.
Please select point located in the vacant area at the far-left corner of the desktop.
Please select point the vacant area between the cup and the car.
Reasoning Step
Reasoning Step
Reasoning Step
Please provide point in the vacant area desktop that simultaneously satisfies the following two spatial conditions:1.000 On the side of the car;2.In front the brown toy.
Please select point located the vacant area at the far-right the desktop the green plate.
Point to the highest hamburger.
Reasoning Step
Reasoning Step
Reasoning Step
Point to the hamburger closest to the cup.
Point to the hamburger on the green plate.
Point to the rightmost blue plate.
Reasoning Step 2
Reasoning Step
Reasoning Step
Point to the plate that the flashlight facing.
Point to the biggest panda.
Point to the closest plate.
Figure 50: Visualization of RoboRefer's prediction (blue point) in the real-world evaluation.
77

---

Reasoning Step Point the farthest panda.
Reasoning Step=1
Reasoning Step Point closest plate.
Point to biggest panda.
Reasoning Step
Reasoning Step 3
Reasoning Step 1
Point to the hamburger that is first close the nearest mug to the camera.
Point to the free space
Point to the hamburger that the flashlight is illuminating.
front-facing side of the teddy bear.
Reasoning Step 1
Reasoning Step= 2
Reasoning Step
Point the plate illuminated by the flashlight.
Point the nearest panda.
Point to the leftmost panda.
Reasoning Step 2
Reasoning Step
Reasoning Step 1
Pick the apple in front of the logo side the leftmost cup.
Place the apple on the table, aligned with the apple row.
Navigate to the nearest table.
Figure 51: Visualization RoboRefer's prediction (blue point) in the real-world evaluation.
78


Reviews:
[{"review_id": "5HuOKrKHpy", "reviewer": "Reviewer_ksPZ", "summary": "RoboRefer is a 3-D-aware vision-language model built for spatial referring. Given one- or multi-step spatial instructions, it picks the exact pixel that can be lifted into a 3-D coordinate for a robot. A depth encoder—aligned through supervised tuning—adds depth cues without disturbing the pretrained RGB branch. The model then gets a reinforcement pass with Group Relative Policy Optimization and a mix of outcome- and process-level rewards, which tightens its multi-step reasoning and point accuracy. To train and test the system, the authors introduce RefSpatial and RefSpatial-Bench dataset.", "questions": "1. Demonstrate RFT at scale:  Please provide 8 B-RFT results and training stability logs.  If the larger model fails to converge or offers no gain, the claimed contribution of RFT is weakened.\n\n2. Expand and diversify the evaluation benchmark. RefSpatial-Bench contains only 200 images—100 samples for Location and Placement task, plus 77 for Unseen set. And all the scenes are indoor. Therefore, the statistical confidence is low and the setting fails to test outdoor spatial reasoning.  To substantiate the claim of general spatial understanding (Line 8), the benchmark must grow to at least 2000 images, and diversify its domains to include outdoor environments, such as street, warehouse and construction sites, so that depth noise, lighting variation and larger spatial scales are represented.", "limitations": "No.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n[+] The paper is generally well-written; figures concisely convey the pipeline and dataset composition.\n\n[+] The additional depth branch is straightforward and works.\n\nWeaknesses:\n\n[-]  Benchmark scale & diversity – RefSpatial-Bench comprises only 200 indoor tabletop images, yet the paper claims “general 3-D reasoning. Thirty-five examples per reasoning length (≤ 5 steps) are insufficient for statistically robust conclusions; the setting omits mobile-robot/outdoor scenarios altogether.\n\n[-] Depth-fusion idea mirrors MM-Spatial[1] and Spatial RGPT[2], novelty is incremental and not rigorously compared.\n\n[-] RFT is a straightforward RLHF method (GRPO) without major modification.\n\n[-] RFT is demonstrated only for the 2 B model, the stronger 8 B version has no RFT results. Without 8 B-RFT evidence, the main claims that \"RFT stage fosters better reasoning ability\" (Line 258) & \"RFT stage provides powerful generalization ability\" (Line 264) are unsubstantiated at the scale likely to interest practitioners.\n\n[-] Even after “speed-ups” the model needs ~29 s per command on robot hardware, far from real-time.\n\n\n[1] MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs\n[2] SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models", "was_revised": true, "final_justification": "The authors addressed most of my concerns. The paper is generally well-written, with clear figures that effectively illustrate the pipeline and dataset. The rebuttal strengthened the submission in two important ways: \n1. Expanded Benchmark: RefSpatial-Bench was extended from 200 to 2,000 images, now covering a wider range of scenes and improving statistical reliability.\n2. 8B-RFT Results: The authors provided RFT results for the 8B model, addressing the earlier concern that the main claims were not validated at larger scale.\n\nNevertheless, the dataset, while larger, is still modest compared to existing multimodal reasoning benchmarks and lacks mobile-robot/outdoor scenarios.\n\nOverall, the authors have addressed key concerns on scale and validation. I increase my score.", "rebuttal": "> **[`Weakness 1 & Question 2`]**: *Benchmark scale & diversity*\n\n**A**: Thank you for the suggestion. Actually, spatial referring tasks demand precise mask annotations and complex textual instructions with spatial constraints, which are costly and challenging to obtain. Therefore, **existing benchmarks are typically on the order of hundreds of images (e.g., Where2Place: 100; RoboSpatial: 122). Our RefSpatial-Bench (200 images) follows this common scale**. Nevertheless, to further validate our approach, we expand it to RefSpatial-Bench-Large (2,000 images), covering diverse indoor and outdoor scenes, annotated by external annotators. Data sources and statistics are summarized in Rebuttal Table 1 below. Results in Rebuttal Table 3 below on this extended benchmark consistently support our claims: the RFT stage improves reasoning (see L258) and generalization (see L264), enabling generalized multi-step spatial referring with reasoning (see L8). We will release the extended RefSpatial-Bench publicly.\n\n**Rebuttal Table 1: Data sources and statistics of RefSpatial-Bench-Large. 'Old' refers to the original scene categories in RefSpatial-Bench, while 'New' indicates the newly introduced ones.** Each part contains 600 images from the OpenImages test split, 200 from nuScenes, 100 from SUNRGB-D, and 100 from the original RefSpatial-Bench. Numbers indicate the number of benchmark samples for each scene category.\n\n\n|Scene Category|Location (1000)|||Placement (1000)|||\n|-|-|-|-|-|-|-|\n||Indoor Num.|Outdoor Num.|Total Num.|Indoor Num.|Outdoor Num.|Total Num.|\n|Street View (New)|0|239|239|0|253|253|\n|Sport Field (New)|0|83|83|0|76|76|\n|Park (New)|0|123|123|0|112|112|\n|Office (Old)|76|0|76|83|0|83|\n|Living Room (Old)|75|0|75|85|0|85|\n|Kitchen (Old)|76|0|76|76|0|76|\n|Bedroom (New)|75|0|75|62|0|62|\n|Industrial (New)|31|41|72|45|18|63|\n|Commercial (New)|60|0|60|55|0|55|\n|Bathroom (Old)|63|0|63|66|0|66|\n|Dining Room (Old)|58|0|58|69|0|69|\n|**Total Num.**|**514**|**486**|**1000**|**541**|**459**|**1000**|\n\n---\n\n> **[`Weakness 2`]**: *Depth-fusion idea mirrors MM-Spatial and Spatial RGPT, novelty is incremental and not rigorously compared.*\n\n**A**: \n- We have discussed the limitations of MM-Spatial and SpatialRGPT (see L43, L136) and highlighted key differences (see L315).  Our method introduces an **additional depth branch** that enables effective co-training on limited RGB-only data with minimal performance drop in general VQA (see Table 4 in the main paper), while significantly reducing training costs (only requires 1/20 QA pairs of RefSpatial). In contrast, **MM-Spatial and SpatialRGPT require over twice as much RGB-only data relative to spatial data to maintain general VQA performance**. Moreover, SpatialRGPT relies on the region-level depth of given masks or bounding boxes, which oversimplifies the depth perception of the specific object compared to our full-image depth approach. \n\n- We report SpatialRGPT’s results on spatial understanding benchmarks in the Rebuttal Table 2 below. We find that its RGB images and Depth maps as inputs rely on the same shared image encoder and imprecise data construction from 2D images, leading to inferior performance compared to RoboRefer. MM-Spatial is not open-sourced, precluding quantitative comparisons across these benchmarks.\n\n**Rebuttal Table 2: Comparison with SpatialRGPT on spatial understanding benchmarks.** Numbers represent success rates ( $\\uparrow$ ).\n\n| Model | CV-Bench           |               |                | BLINK             |                | RoboSpatial | SAT   | EmbSpatal |\n|--------------------------|:-----------------:|:-------------:|:--------------:|:-----------------:|:--------------:|:-----------:|:-----:|:---------:|\n|    | 2D-relation  | 3D-Depth  | 3D-Distance | 2D-Relation   | 3D-Depth   |    |     |    |\n| SpatialRGPT-VILA-1.5-8B  | 91.00        | 89.83 | 88.50  | 81.12    | 89.51   | 66.67       | 64.00 | 59.62     |\n| RoboRefer-2B-SFT | 96.31  | 97.17 | 90.83| 87.41| 91.13 | 82.93| 82.00 | 71.10 |\n| RoboRefer-8B-SFT | 96.90  | 98.33 | 93.50| 91.61 | 92.74| 84.55 | 86.67 | 72.53 |\n\n\n---\n\n> **[`Weakness 3`]**: *RFT is a straightforward RLHF method (GRPO) without major modification.*\n\n**A**: **Our RFT introduces novel metric-sensitive, process-based rewards tailored for spatial referring tasks** (see L164) and is significantly different from vanilla GRPO, which relies solely on outcome-based rewards. This design is motivated by the nature of spatial referring tasks, whose intermediate outputs are not inherently sequential but instead require metric sensitivity and order invariance (see Appx. L1090–L1100)—key properties that guided our modification of vanilla GRPO into process-based RFT.\n\n---\n\n> **[`Weakness 4 & Question 1`]**: *Demonstrate RFT at scale.*\n\n**A**: Thank you for the suggestion. \n\n- As noted in L231, we excluded 8B-RFT due to its high computational cost. Our TRL (Transformer Reinforcement Learning)-based training framework lacks multi-node support, and vLLM acceleration is not applicable since our model requires RGB-D input. Consequently, **training 8B-RFT would take over 10 days on 8×A100 GPUs, which exceeds the rebuttal timeframe.** We will include 8B-RFT results in the next version. \n\n- Section 4.2 and Table 2 in the main paper have already provided detailed support for our claims about RFT. Results in Rebuttal Table 3 below on extended RefSpatial-Bench-Large remain consistent: the RFT stage improves reasoning (see L258) and generalization (see L264), enabling generalized multi-step spatial referring with reasoning (see L8). Recent work [1][2] has also confirmed that 2B-RFT is effective, and 7B/8B-RFT will also be effective as expected.\n\n\n\n**Rebuttal Table 3: Results on RefSpatial-Bench and RefSpatial-Bench-Large.** L. and P.\ndenote the benchmark’s Location and Placement parts. Numbers represent success rates ( $\\uparrow$ ).\n\n|Benchmark|2B-SFT|||2B-RFT|||8B-SFT|||\n|-----------------------------------|------------------------|------------------------|------------------------|------------------------|------------------------|------------------------|------------------------|------------------------|------------------------|\n||Old Category|New Category|Total|Old Category|New Category|Total|Old Category|New Category|Total|\n|RefSpatial-Bench-L.|44.00|-|44.00|49.00|-|49.00|46.00|-|46.00|\n|RefSpatial-Bench-P.|45.00|-|45.00|47.00|-|47.00|47.00|-|47.00|\n|RefSpatial-Bench-Large-L.|44.68|42.11|43.20|49.64|45.75|47.40|46.34|44.89|45.50|\n|RefSpatial-Bench-Large-P.|45.35|42.75|43.90|47.61|44.90|46.10|47.39|45.97|46.60|\n\n---\n\n> **[`Weakness 5`]**: *Even after “speed-ups”, the model needs ~29 s per command on robot hardware, far from real-time.*\n\n**A**: We apologize for the confusion. The reported execution time in the Open6DOR v2 simulation benchmark covers the entire pick-and-place pipeline: (1) perception (e.g., object localization/placement prediction, mask prediction, object point cloud extraction), (2) motion planning (e.g., IK solving), and (3) robotic execution. **RoboRefer is responsible only for object localization/placement of step (1) perception and requires just 0.4s per inference on RTX4090**. A detailed runtime breakdown is provided in Rebuttal Table 4 below. Notably, **we do not claim real-time performance**. Section 4.4 merely demonstrates RoboRefer's robotic application in both simulated and real-world environments.\n\n\n**Rebuttal Table 4: Detailed runtime on Open6DOR-v2 benchmarks (simulation).** Numbers represent time cost ( $\\downarrow$ ).\n\n| Method | Perception(s)  | Motion Planning(s) | Robotic Execution(s) | Total Execution Time(s) |\n|--------------------------|:-----------------:|:-------------:|:--------------:|:--------------:|\n| SoFar (Florence-2 for location, GPT-4o for placement)  | 14.5  |  14.6 | 10.8  | 39.9 |\n| Ours (RoboRefer for location and placement)  | 3.8   | 14.6  | 10.8  | 29.2 |\n\n---\nReferences: \n\n[1] Visual-RFT: Visual Reinforcement Fine-Tuning, ICCV'2025\n\n[2] Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning, Arxiv'2025"}, {"review_id": "NQ6QiKnbE8", "reviewer": "Reviewer_Wvkt", "summary": "This paper introduces a new dataset, RefSpatial, which is twice as large as previous datasets and includes a greater variety of spatial relationships. It also presents a model called RoboRefer, designed for precise 3D spatial understanding by incorporating depth inputs.\nThe authors propose using Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT) to post-train vision-language models (VLMs), specifically focusing on NVILA. Experimental results show that RoboRefer improves performance on spatial reasoning benchmarks while preserving accuracy on other multimodal tasks.", "questions": "# Questions\n\n- In Table 2, is depth information included when evaluating RoboRefer?\n- For the 2D and 3D data, do the answers require explicit reasoning steps, or are they primarily spatial recognition tasks?\n- Fig. 4 says \"RoboRefer-RFT excels in unseen and multi-step cases.\", how about \"RoboRefer-SFT\"? any quantitative analysis to justify the RFT training?\n\n\n# Suggestions\n\n- Given the complexity of the data synthesis pipelines across domains, a full-page diagram illustrating each pipeline would be helpful—consider adding this to the appendix.\n- Consider adding additional spacing between Tables 3 and 4 to improve readability and visual separation.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n- The paper presents a scalable data synthesis pipeline that integrates 2D, 3D, and simulation-based data generation.\n- Table 7 provides a well-executed ablation study that offers meaningful insights into the contribution of different data sources.\n\n# Weaknesses\n\n- Distinction from SpatialRGPT:\n    - The main difference appears to be the inclusion of multi-step reasoning data generated from simulation. However, the simulation data seems critical (as shown in Table 7), yet the details necessary to reproduce it (e.g., L221–L226) are lacking.\n    - Additionally, the paper does not include a direct comparison with a VLM trained on SpatialRGPT data, which would help clarify the improvements gained from their proposed approach.\n- Justification for RFT:\n    - The rationale for applying Reinforcement Fine-Tuning (RFT) is unclear, especially given the already substantial Supervised Fine-Tuning (SFT) dataset. What specific benefits does RFT bring in this context?", "was_revised": true, "final_justification": "Overall, the paper presents two main novelties: (1) the dataset and (2) the exploration of different RL rewards. I initially found the details regarding the dataset unclear, but this was adequately resolved in the rebuttal.\n\nI strongly suggest that the authors further streamline the main paper and move detailed explanations to the appendix, ensuring that readers are clearly directed to the appendix whenever necessary.\n\nI moved my score from 4 to 5.", "rebuttal": "> **[`Weakness 1`]**: *Distinction from SpatialRGPT.*\n\n**A**: Our approach differs from SpatialRGPT in 4 key aspects:\n- [Task Setting]: **Our model addresses a more challenging spatial referring task,** where take a spatially constrained textual instruction as input. It requires multi-step spatial reasoning with learned spatial knowledge to precisely localize the referred object as a 2D point step-by-step. In contrast, SpatialRGPT addresses a simpler VQA task and relies on externally provided region information as input for specific object referring.\n\n- [Model Usage]: Unlike SpatialRGPT, which needs additional masks or detection tools to generate masks or 2D boxes as inputs for object reference and simplify referring tasks, **our model can use textual descriptions for object referencing** (see L283), which better aligns with real-world robotic applications.\n\n- [Data Pipeline]: **Our data pipeline adopts a more structured, progressive design than SpatialRGPT.** It first uses 2D image data to teach core spatial concepts and general depth perception across diverse indoor and outdoor scenes. Next, accurate 3D data enhances fine-grained spatial understanding in indoor settings for robotics. Finally, simulation data introduces multi-step spatial referring with reasoning. This staged approach yields stronger spatial understanding and reasoning than SpatialRGPT, which relies solely on data generated from 2D images and lacks precise spatial perception for more complex spatial referring tasks.\n\n- [Training Pipeline]: **Our training pipeline includes process-based RFT after SFT to further improve multi-step reasoning and generalization** for spatial referring tasks, whereas SpatialRGPT is trained with SFT only.\n---\n> **[`Weakness 1`]**: *Lack the details necessary to reproduce simulation data.*\n\n**A**: Please see Appx. A.3 for details on simulation scene generation, asset cleaning, QA generation, and reasoning process labeling. We will release all cleaned assets, scenes, and the full pipeline to help reproduce it.\n\n---\n> **[`Weakness 2`]**: *Not include a direct comparison with a VLM trained on SpatialRGPT data.*\n\n**A**: Following the reviewer's suggestion, we add SpatialRGPT’s performance on spatial understanding benchmarks in Rebuttal Table 1 below. RoboRefer outperforms SpatialRGPT, benefiting from our RefSpatial dataset, which is large-scale and offers more precise and diverse spatial relations (31 vs. 17). Moreover, SpatialRGPT’s reliance on masks or boxes oversimplifies object references, limiting performance.\n\n**Rebuttal Table 1: Comparison with SpatialRGPT on spatial understanding benchmarks.** Numbers represent success rates ( $\\uparrow$ ).\n\n| Model | CV-Bench           |               |                | BLINK             |                | RoboSpatial | SAT   | EmbSpatal |\n|--------------------------|:-----------------:|:-------------:|:--------------:|:-----------------:|:--------------:|:-----------:|:-----:|:---------:|\n|                          | 2D-relation        | 3D-Depth      | 3D-Distance    | 2D-Relation        | 3D-Depth       |             |       |           |\n| SpatialRGPT-VILA-1.5-8B  | 91.00        | 89.83 | 88.50  | 81.12    | 89.51   | 66.67       | 64.00 | 59.62     |\n| RoboRefer-2B-SFT             | 96.31              | 97.17         | 90.83          | 87.41              | 91.13          | 82.93       | 82.00 | 71.10     |\n| RoboRefer-8B -SFT            | 96.90              | 98.33         | 93.50          | 91.61              | 92.74          | 84.55       | 86.67 | 72.53     |\n\n---\n> **[`Weakness 3 & Question 3`]**: *Justification for RFT.*\n\n**A**: RFT brings two main benefits:\n-  [Generalization to Unseen Cases]: In Table 2 in the main paper (RefSpatial-Bench Unseen raw), which features novel combinations of spatial relations absent from RefSpatial dataset, our 2B-RFT model surpasses 2B-SFT by 9.1% in accuracy, showing **the strong generalization enabled by the RFT stage** (see L264).\n-  [Enhance Multi-Step Reasoning Ability]: In the Rebuttal Table 2 below, the RFT-based model consistently outperforms the SFT-based model across varying reasoning steps, especially at larger steps, showing the RFT stage's **effectiveness in enhancing multi-step reasoning** (see L258).\n\n**Rebuttal Table 2: Results on RefSpatial-Bench of different reasoning steps**. Numbers represent success rates ( $\\uparrow$ ).\n\n| Reasoning Step Num.|     2B-SFT     |     2B-RFT     |    \n|:---------------|:--------------|:--------------|\n| **RefSpatial-Bench-Location** |||||\n| Step 1 |     63.33      | 66.67 (+3.34) |    \n| Step 2 |     39.58      | 43.75 (+4.17) |   \n| Step 3 |     27.27      | 36.36 (+9.09) |    \n| **Total**      |     44.00       |   49.00 (+5.00) |  \n| **RefSpatial-Bench-Placement** ||\n| Step 2 |     55.56      |  55.56 (+0.00) |   \n| Step 3 |     41.67      |   41.67 (+0.00) |     \n| Step 4 |     41.67      | 45.83 (+4.16) |   \n| Step 5 |       0.00        | 25.00 (+25.00)  |   \n| **Total**      |     45.00       |   47.00 (+2.00) |   \n\n---\n> **[`Question 1`]**: *Is depth information included when evaluating RoboRefer*\n\n**A**: Yes. RoboRefer is evaluated with RGB-D inputs by default, with depth maps generated from RGB images via DepthAnything V2.\n\n---\n> **[`Question 2`]**: *For the 2D and 3D data, do the answers require explicit reasoning steps, or are they primarily spatial recognition tasks?*\n\n**A**: Yes, the 2D and 3D datasets are primarily spatial recognition tasks. They are not annotated with explicit reasoning steps (see L190) and are mainly used to learn diverse spatial concepts and improve spatial understanding.\n\n---\n> **[`Suggestion 1`]**: *A full-page diagram illustrating each pipeline would be helpful.*\n\n**A**: Thanks for your advice. We will polish the pipeline part to make it easier to understand.\n\n---\n> **[`Suggestion 2`]**: *Adding additional spacing between Tables 3 and 4.*\n\n**A**: Thank you for the suggestion. We will revise the paper accordingly.."}, {"review_id": "sfjgFBUmIT", "reviewer": "Reviewer_Z1zy", "summary": "This paper presents RoboRefer, a VLM pipeline for spatial referring in robotics. The task is framed as predicting a single 2D point in image space given RGB(D) input and a textual instruction. The authors propose a multi-stage training process, SFT for spatial understanding using a dedicated depth encoder, followed by RFT to improve multi-step spatial reasoning. The paper also introduces RefSpatial, a new large-scale dataset of 2.5M samples with 20M QA pairs and detailed spatial reasoning steps. Evaluation spans VQA, spatial benchmarks, and simulated/real-world robot experiments.", "questions": "1.\tLine 127, “Compared to the 2D bbox, point can naturally map to 3D coordinates via depth, providing accurate spatial anchors”. I have a concern on this one, see limitations.\n2.\tIs there any specific reason why NVILA is chosen as the backbone?", "limitations": "1.\tWhile the problem setting is different and somewhat more complex than previous works, the method and training pipeline developed by the paper are not entirely new and fall within expectation. For instance, training with RGBD images improve the spatial reasoning capabilities of VLMs [2]. RFT alleviates the memorization issue of the models and improves reasoning [5]. Joint training preserves commonsense knowledge [6], which also applies to data of other modalities.\n2.\tLine 288, “By predicting a single target point, it mitigates 2D detection ambiguity under occlusion”. Why does it help with occlusion?\n3.\tThe depth-to-3D mapping assumption is central but under-specified. How is 3D accuracy guaranteed by predicting a 2D point, especially in real-world robotic use where depth noise or partial views are common?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: see below; Weaknesses: see Questions and Limitations\n\n1. The formulation of the problem, spatial referring, which involves predicting a single 2D \npoint (x, y) in image space to specify a target location or destination given RGB/RGBD and textual inputs, while also reasoning about sizes, position, orientations, spatial relations with more fine-grained questions (e.g., place the object between the pen holder and keyboard, lined up with the cup’s logo) is more comprehensive and complex than the problem setup in previous works [1, 2, 3, 4].\n\n2. The paper introduces multi-stage training recipe (depth alignment + spatial understanding enhancement + RFT) to first align depth and spatial understanding and then avoid the VLM to “memorize” the QA answers.\n\n3. A 2.5M dataset is introduced for the problem setting with carefully designed data curation pipeline. The dataset can be helpful for the community. The overall workload is high.\n\n4. Experiments are comprehensive, comprising both VQA and simulated/real robot experiments.", "was_revised": true, "final_justification": "My concerns are mostly addressed, though I have slight remaining concern regarding depth-to-3D mapping with depth noise in read-world application. Yet, I think paper strengths overweigh the weakness and is helpful for the community. My initial rating of 5: Accept is maintained.", "rebuttal": "> **[`Question 2`]**: *Is there any specific reason why NVILA is chosen as the backbone?*\n\n**A**: Please check Table 1 in the main paper. NVILA outperforms other open-source VLMs under comparable model scales, such as Qwen 2.5-VL (even 72B), in spatial understanding. **Enhancing a strong baseline with our dataset and training strategy further validates their effectiveness**. Notably, our dataset is model-agnostic and transferable to other backbones. Despite partial training on the RefSpatial dataset, Qwen2.5-VL-7B still shows notable improvements on spatial understanding benchmarks in the Rebuttal Table 1 below.\n\n**Rebuttal Table 1: Results on spatial understanding benchmarks.** Numbers represent success rates ( $\\uparrow$ ).\n\n| Model                    | CV-Bench           |               |                | BLINK             |                | RoboSpatial | SAT   | EmbSpatal |\n|--------------------------|:-----------------:|:-------------:|:--------------:|:-----------------:|:--------------:|:-----------:|:-----:|:---------:|\n|                          | 2D-relation        | 3D-Depth      | 3D-Distance    | 2D-Relation        | 3D-Depth       |             |       |           |\n| Qwen-2.5-VL-7B (base)       |         82.15           |        60.17         |         69.00           |        64.34        |      60.98       |    49.59    | 30.00 |   40.20    |\n| Qwen-2.5-VL-7B (finetuned)  |         95.85           |        95.00         |         90.83           |        83.22        |      84.68       |    69.92    | 85.75 |   76.32    |\n| NVILA-8B (base)             |         91.54           |        91.83         |         90.67           |        76.92        |      76.61       |    59.35    | 63.33 |   67.72    |\n| RoboRefer-8B-SFT (finetuned)             |         96.90           |        98.33         |         93.50           |        91.61        |       92.74        |    84.55    | 86.67 |   72.53    |\n\n\n---\n\n> **[`Limitation 1`]**: *The method and training pipeline are not entirely new and fall within expectations.*\n\n**A**: **RoboRefer introduces a novel process-based RFT pipeline tailored for multi-step spatial referring**. Unlike prior RFT works that rely solely on outcome-based rewards or struggle with process rewards in sequential decision-making tasks (see Appx. L1083–L1089), **we identify a key insight: the metric sensitivity and order invariance of intermediate outputs in spatial referring** (see Appx. L1090–L1100), which are uniquely suited to this task rather than general sequential decision-making tasks. Based on this, we design significantly new metric-sensitive process rewards (see L164) and special annotations of reasoning process in RefSpatial dataset, leading to good performance gains (see L326).\n\n---\n\n> **[`Limitation 2`]**: *By predicting a single target point, it mitigates 2D detection ambiguity under occlusion”. Why does it help with occlusion?*\n\n**A**: We apologize for the confusion. We clarify that masks generated from a single target point are more accurate than those from 2D boxes under occlusion. In cluttered scenes, occlusions often cause 2D boxes to include irrelevant objects, resulting in ambiguous visual prompts for SAM2 and imprecise masks. This degrades grasp accuracy and success rates in the open-loop AnyGrasp pipeline, a known limitation of 2D boxes also noted in prior work [1].\n\n---\n\n> **[`Question 1 & Limitation 3`]**: *The depth-to-3D mapping assumption is central but under-specified. How is 3D accuracy guaranteed by predicting a 2D point, especially in real-world robotic use where depth noise or partial views are common?*\n\n**A**: Thank you for the suggestion. We will clarify this assumption in the revision. While depth noise and partial observations are important real-world challenges, our setting follows prior work [1][2][3], which assumes that accurate depth-to-3D mapping is feasible given known camera intrinsics and extrinsics—sufficient for common manipulation and navigation tasks. Moreover, these challenges can be effectively mitigated via the following strategies:\n\n- Depth noise can be mitigated by recent advances in monocular depth estimation [4], monocular geometry prediction [5], and stereo methods [6]. In cases of severe noise, we employ [6] in real-world settings to mitigate this issue.\n\n- Partial views are mitigated in our method by leveraging pixel-level target points. Further improvement is possible by incorporating RoboRefer as a spatially-aware planner for active perception, which we leave for future work.\n\n---\n\nReferences:\n\n[1] RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics, CoRL'2024\n\n[2] RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete, CVPR'2025\n\n[3] A0: An Affordance-Aware Hierarchical Model for General Robotic Manipulation, ICCV'2025\n\n[4] UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler, CVPR'2024\n\n[5] MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details, Arxiv'2025\n\n[6] FoundationStereo: Zero-Shot Stereo Matching, CVPR'2025"}, {"review_id": "QvH8GJNvHa", "reviewer": "Reviewer_2QkV", "summary": "The paper introduces RoboRefer, a 3D-aware vision-language model designed to enhance spatial referring capabilities for robotics. The authors propose a novel VLM that integrates a dedicated depth encoder to improve spatial understanding and employs reinforcement fine-tuning to enable multi-step spatial reasoning. The authors introduce RefSpatial, a large-scale dataset containing 2.5M high-quality examples and 20M QA pairs, covering 31 spatial relations and supporting complex reasoning processes. This paper also propose RefSpatial-Bench, a challenging benchmark to evaluate spatial referring with multi-step reasoning.", "questions": "1. The authors refine fine-grained spatial understanding of indoor scenes for robotics using 3D embodied videos. However, it is unclear how the authors segment each subtask within a video stream. Could you please elaborate on the methodology used to divide the video stream into individual subtasks? Additionally, does this segmentation process introduce any potential errors or biases, and if so, how are these addressed?\n2.  In Section 4.4 and the supplemental material, the authors suggest that RoboRefer can be integrated into the system as a useful tool. However, it is unclear how this integration is achieved for a Vision-Language-Action (VLA) model. Specifically, is RoboRefer integrated by incorporating its results as additional training data during the training process, or is it used as a planner tool to segment tasks into subtasks? Given the limited gains observed in subtask segmentation for small tasks due to poor prompt following by existing VLA models, how effective is this integration strategy? Additionally, how does the system determine when a subtask is completed by the VLA model to switch to the next task?\n3. The method proposed in Figure 2 relies on depth map inputs, which can significantly affect the model's inference speed and practical application. In real-world scenarios, depth sensor outputs are often noisy. How does the presence of depth noise impact the performance of the model, and what steps are being taken to mitigate this issue? Specifically, have the authors conducted experiments to quantify the effect of depth noise on the model's accuracy and robustness? If so, what are the findings, and what measures are being considered to improve the model's resilience to noisy depth inputs?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "> **Strengths**\n> \n\nThe authors introduce a dedicated depth encoder and reinforcement fine-tuning (RFT) to improve spatial understanding and reasoning. The claims are well-supported by both theoretical analysis and experimental results. The methods used are appropriate for the task, and the authors provide a thorough evaluation of both single-step spatial understanding and multi-step spatial reasoning.\n\n> **Weaknesses**\n> \n\nThe limitations of the model presented in the paper are primarily related to its reliance on precise textual descriptions and its inability to handle ambiguous human instructions effectively. Specifically, the model struggles with:\n\n- Probabilistic Preference: The model has difficulty interpreting instructions that involve probabilistic reasoning based on human biases. For example, when an instruction refers to an object with some ambiguity (e.g., \"pick the one facing the drink\"), humans often use probabilistic biases to infer the intended object, a capability that the current model lacks.\n- Spatial Compatibility: The model also struggles with instructions that require understanding implicit spatial feasibility. For instance, when an instruction suggests placing an object between two others without explicit spatial constraints, humans use their understanding of spatial compatibility to determine the most feasible placement, which the model cannot currently do.", "was_revised": true, "final_justification": "My questions are sufficiently addressed by the rebuttal and I’m happy to raise my score.", "rebuttal": "> **[`Weakness 1`]**: *Struggle with ambiguous human instructions, such as probabilistic preference and spatial compatibility.*\n\n**A**: Thank you for the suggestion. We have discussed limitations such as probabilistic preference and spatial compatibility in Appx. F, along with potential solutions including procedural synthesis of intent-aware data and co-training with intent-focused datasets. As our work mainly focuses on multi-step spatial referring with explicit instructions, handling ambiguous human intents is beyond the current scope and will be explored in future work.\n\n---\n> **[`Question 1`]**: *Segment each subtask within a video stream.*\n\n**A**: In fact, we do not divide the video stream into subtasks. Instead, **we utilize the CA-1M dataset at frame level rather than clip level** (see L215 and Appx. A.2). With (1) dense per-frame 2D/3D annotations and (2) complete camera and depth information, we construct fine-grained, spatially-aware template and reasoning QA pairs. Moreover, the videos in CA-1M were collected by individuals walking indoors without specific goals, making it difficult to segment the video stream into subtasks.\n\n---\n> **[`Question 2`]**: *Is RoboRefer integrated by incorporating its results as additional training data during the training process?*\n\n**A**: Yes, RoboRefer can serve as a perception tool to localize and place interactive objects (see L284). Integrated with open-loop control, it generates action data with spatially constrained instructions, enriching VLA training. In Rebuttal Table 1 below, fine-tuning OpenVLA with RoboRefer-collected data on Open6DOR-v2 enhances spatial ability, enabling the successful completion of previously unsolvable Level-3 tasks involving complex spatial relations.\n\n**Rebuttal Table 1: Results on Open6DOR-v2 benchmarks (simulation).** Numbers represent success rates ( $\\uparrow$ ).\n\n\n| Model                    | Level-1 |      Level-2  |      Level-3          | Avg. |\n|--------------------------|:-----------------:|:-------------:|:--------------:|:-----------------:|\n| OpenVLA  | 51.6       | 13.1 | 0.0  | 43.6  |\n| OpenVLA (finetuned)             | 69.9              | 30.4         | 30.0          | 61.4    |\n\n---\n> **[`Question 3`]**: *Is RoboRefer used as a planner tool to segment tasks into subtasks?*\n\n**A**: No, RoboRefer isn't used to decompose tasks in our experiments, but it can provide predicted points as visual prompts to guide VLA in generating actions if combined with the VLA model, improving success under spatially constrained instructions.\n\n---\n> **[`Question 4`]**: *Given the limited gains observed in subtask segmentation for small tasks due to poor prompt following by existing VLA models, how effective is this integration strategy?*\n\n**A**: We do not integrate RoboRefer as a planner with VLA due to marginal benefits in our experiments. Instead, RoboRefer’s predicted points can serve as effective visual guidance to enhance VLA’s instruction-following compared to directly following textual instructions, as shown in previous VLA works [1][2].\n\n---\n> **[`Question 5`]**: *How does the system determine when a subtask is completed by the VLA model to switch to the next task?*\n\n**A**: With 0.4s inference (2.5Hz; see L295), RoboRefer can efficiently serve as a checker to monitor sub-goal completion under spatial constraints (e.g., verifying if the hamburger is placed in front of the teddy bear) using spatial knowledge acquired during SFT if combined with the VLA model.\n\n---\n> **[`Question 6`]**: *How does the presence of depth noise impact the performance of the model, and what steps are being taken to mitigate this issue?*\n\n**A**: In real-world experiments, we use a relative depth estimation model, DepthAnything v2, to obtain relative depth as the model's depth input (see L236), which effectively reduces depth noise from a real camera.\n\n---\n> **[`Question 7`]**: *Have the authors conducted experiments to quantify the effect of depth noise on the model's accuracy and robustness?*\n\n**A**: We evaluate success rates under depth noise in real-world settings (see Rebuttal Table 2 below). Depth maps generated from the strong monocular relative depth estimation model (i.e., DepthAnything V2) offer the highest robustness and success. Despite depth noise from a real camera, RoboRefer maintains great performance by leveraging RGB priors due to mixed RGB and RGB-D training during SFT stage.\n\n**Rebuttal Table 2: Results on real-world experiments.** Numbers represent success rates ( $\\uparrow$ ).\n\n\n| Real-world Task | Depth from DepthAnything V2         |      Depth from a real camera       |\n|--------------------------|:-----------------:|:-------------:|\n| Pick the specific hamburger closest to the mug nearest to the camera.  | 80       | 70  |\n| Place the hamburger in front of the teddy bear.  | 90| 90|\n| Pick the apple in front of the leftmost cup’s logo side.  | 80       | 80|\n| Place the apple aligned with the existing apple row.  | 60| 40|\n\n\n---\nReferences:\n\n[1] Improving Vision-Language-Action Models via Chain-of-Affordance, ICCV'2025\n\n[2] DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping, Arxiv'2025"}]

Author rebuttal:

Meta-review: All reviewers found the studied problem to be important, the proposed method and benchmark to be effective, and the results promising. The rebuttal successfully addressed most reviewer comments and all reviewers recommend acceptance. The authors are encouraged to improve the final paper version by following reviewer recommendations.
Decision: Accept (poster)