Title: RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics
Venue: NeurIPS 2025
Accepted: yes
Keywords: Spatial Referring

Spatial Reasoning

Vision-Language Model

Robotics
Abstract: Spatial referring is a fundamental capability of embodied robots to interact with the 3D physical world. However, even with the powerful pretrained VLMs, recent approaches are still not qualified to accurately understand the complex 3D scenes and dynamically reason about the instruction-indicated locations for interaction. To this end, we propose RoboRefer, a 3D-aware vision language model (VLM) that can first achieve precise spatial understanding by integrating a disentangled but dedicated depth encoder via supervised fine-tuning (SFT). Moreover, RoboRefer advances generalized multi-step spatial reasoning via reinforcement fine-tuning (RFT), with metric-sensitive process reward functions tailored for spatial referring tasks. To support SFT and RFT training, we introduce RefSpatial, a large-scale dataset of 20M QA pairs (2x prior), covering 31 spatial relations (vs. 15 prior) and supporting complex reasoning processes (up to 5 steps). In addition, we introduce RefSpatial-Bench, a challenging benchmark filling the gap in evaluating spatial referring with multi-step reasoning. Experiments show that SFT-trained RoboRefer achieves state-of-the-art spatial understanding, with an average success rate of 89.6%. RFT-trained RoboRefer further outperforms all other baselines by a large margin, even surpassing Gemini-2.5-Pro by 12.4% in average accuracy on RefSpatial-Bench. Notably, RoboRefer can be integrated with various control policies to execute long-horizon, dynamic tasks across diverse robots (e,g., UR5, G1 humanoid) in cluttered real-world scenes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces RoboRefer, a 3D-aware vision-language model designed to enhance spatial referring capabilities for robotics. The authors propose a novel VLM that integrates a dedicated depth encoder to improve spatial understanding and employs reinforcement fine-tuning to enable multi-step spatial reasoning. The authors introduce RefSpatial, a large-scale dataset containing 2.5M high-quality examples and 20M QA pairs, covering 31 spatial relations and supporting complex reasoning processes. This paper also propose RefSpatial-Bench, a challenging benchmark to evaluate spatial referring with multi-step reasoning.\n\nSTRENGTHS AND WEAKNESSES:\n> **Strengths**\n> \n\nThe authors introduce a dedicated depth encoder and reinforcement fine-tuning (RFT) to improve spatial understanding and reasoning. The claims are well-supported by both theoretical analysis and experimental results. The methods used are appropriate for the task, and the authors provide a thorough evaluation of both single-step spatial understanding and multi-step spatial reasoning.\n\n> **Weaknesses**\n> \n\nThe limitations of the model presented in the paper are primarily related to its reliance on precise textual descriptions and its inability to handle ambiguous human instructions effectively. Specifically, the model struggles with:\n\n- Probabilistic Preference: The model has difficulty interpreting instructions that involve probabilistic reasoning based on human biases. For example, when an instruction refers to an object with some ambiguity (e.g., \"pick the one facing the drink\"), humans often use probabilistic biases to infer the intended object, a capability that the current model lacks.\n- Spatial Compatibility: The model also struggles with instructions that require understanding implicit spatial feasibility. For instance, when an instruction suggests placing an object between two others without explicit spatial constraints, humans use their understanding of spatial compatibility to determine the most feasible placement, which the model cannot currently do.\n\nQUESTIONS:\n1. The authors refine fine-grained spatial understanding of indoor scenes for robotics using 3D embodied videos. However, it is unclear how the authors segment each subtask within a video stream. Could you please elaborate on the methodology used to divide the video stream into individual subtasks? Additionally, does this segmentation process introduce any potential errors or biases, and if so, how are these addressed?\n2.  In Section 4.4 and the supplemental material, the authors suggest that RoboRefer can be integrated into the system as a useful tool. However, it is unclear how this integration is achieved for a Vision-Language-Action (VLA) model. Specifically, is RoboRefer integrated by incorporating its results as additional training data during the training process, or is it used as a planner tool to segment tasks into subtasks? Given the limited gains observed in subtask segmentation for small tasks due to poor prompt following by existing VLA models, how effective is this integration strategy? Additionally, how does the system determine when a subtask is completed by the VLA model to switch to the next task?\n3. The method proposed in Figure 2 relies on depth map inputs, which can significantly affect the model's inference speed and practical application. In real-world scenarios, depth sensor outputs are often noisy. How does the presence of depth noise impact the performance of the model, and what steps are being taken to mitigate this issue? Specifically, have the authors conducted experiments to quantify the effect of depth noise on the model's accuracy and robustness? If so, what are the findings, and what measures are being considered to improve the model's resilience to noisy depth inputs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents RoboRefer, a VLM pipeline for spatial referring in robotics. The task is framed as predicting a single 2D point in image space given RGB(D) input and a textual instruction. The authors propose a multi-stage training process, SFT for spatial understanding using a dedicated depth encoder, followed by RFT to improve multi-step spatial reasoning. The paper also introduces RefSpatial, a new large-scale dataset of 2.5M samples with 20M QA pairs and detailed spatial reasoning steps. Evaluation spans VQA, spatial benchmarks, and simulated/real-world robot experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: see below; Weaknesses: see Questions and Limitations\n\n1. The formulation of the problem, spatial referring, which involves predicting a single 2D \npoint (x, y) in image space to specify a target location or destination given RGB/RGBD and textual inputs, while also reasoning about sizes, position, orientations, spatial relations with more fine-grained questions (e.g., place the object between the pen holder and keyboard, lined up with the cup’s logo) is more comprehensive and complex than the problem setup in previous works [1, 2, 3, 4].\n\n2. The paper introduces multi-stage training recipe (depth alignment + spatial understanding enhancement + RFT) to first align depth and spatial understanding and then avoid the VLM to “memorize” the QA answers.\n\n3. A 2.5M dataset is introduced for the problem setting with carefully designed data curation pipeline. The dataset can be helpful for the community. The overall workload is high.\n\n4. Experiments are comprehensive, comprising both VQA and simulated/real robot experiments.\n\nQUESTIONS:\n1.\tLine 127, “Compared to the 2D bbox, point can naturally map to 3D coordinates via depth, providing accurate spatial anchors”. I have a concern on this one, see limitations.\n2.\tIs there any specific reason why NVILA is chosen as the backbone?\n\nLIMITATIONS:\n1.\tWhile the problem setting is different and somewhat more complex than previous works, the method and training pipeline developed by the paper are not entirely new and fall within expectation. For instance, training with RGBD images improve the spatial reasoning capabilities of VLMs [2]. RFT alleviates the memorization issue of the models and improves reasoning [5]. Joint training preserves commonsense knowledge [6], which also applies to data of other modalities.\n2.\tLine 288, “By predicting a single target point, it mitigates 2D detection ambiguity under occlusion”. Why does it help with occlusion?\n3.\tThe depth-to-3D mapping assumption is central but under-specified. How is 3D accuracy guaranteed by predicting a 2D point, especially in real-world robotic use where depth noise or partial views are common?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new dataset, RefSpatial, which is twice as large as previous datasets and includes a greater variety of spatial relationships. It also presents a model called RoboRefer, designed for precise 3D spatial understanding by incorporating depth inputs.\nThe authors propose using Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT) to post-train vision-language models (VLMs), specifically focusing on NVILA. Experimental results show that RoboRefer improves performance on spatial reasoning benchmarks while preserving accuracy on other multimodal tasks.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n- The paper presents a scalable data synthesis pipeline that integrates 2D, 3D, and simulation-based data generation.\n- Table 7 provides a well-executed ablation study that offers meaningful insights into the contribution of different data sources.\n\n# Weaknesses\n\n- Distinction from SpatialRGPT:\n    - The main difference appears to be the inclusion of multi-step reasoning data generated from simulation. However, the simulation data seems critical (as shown in Table 7), yet the details necessary to reproduce it (e.g., L221–L226) are lacking.\n    - Additionally, the paper does not include a direct comparison with a VLM trained on SpatialRGPT data, which would help clarify the improvements gained from their proposed approach.\n- Justification for RFT:\n    - The rationale for applying Reinforcement Fine-Tuning (RFT) is unclear, especially given the already substantial Supervised Fine-Tuning (SFT) dataset. What specific benefits does RFT bring in this context?\n\nQUESTIONS:\n# Questions\n\n- In Table 2, is depth information included when evaluating RoboRefer?\n- For the 2D and 3D data, do the answers require explicit reasoning steps, or are they primarily spatial recognition tasks?\n- Fig. 4 says \"RoboRefer-RFT excels in unseen and multi-step cases.\", how about \"RoboRefer-SFT\"? any quantitative analysis to justify the RFT training?\n\n\n# Suggestions\n\n- Given the complexity of the data synthesis pipelines across domains, a full-page diagram illustrating each pipeline would be helpful—consider adding this to the appendix.\n- Consider adding additional spacing between Tables 3 and 4 to improve readability and visual separation.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRoboRefer is a 3-D-aware vision-language model built for spatial referring. Given one- or multi-step spatial instructions, it picks the exact pixel that can be lifted into a 3-D coordinate for a robot. A depth encoder—aligned through supervised tuning—adds depth cues without disturbing the pretrained RGB branch. The model then gets a reinforcement pass with Group Relative Policy Optimization and a mix of outcome- and process-level rewards, which tightens its multi-step reasoning and point accuracy. To train and test the system, the authors introduce RefSpatial and RefSpatial-Bench dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n[+] The paper is generally well-written; figures concisely convey the pipeline and dataset composition.\n\n[+] The additional depth branch is straightforward and works.\n\nWeaknesses:\n\n[-]  Benchmark scale & diversity – RefSpatial-Bench comprises only 200 indoor tabletop images, yet the paper claims “general 3-D reasoning. Thirty-five examples per reasoning length (≤ 5 steps) are insufficient for statistically robust conclusions; the setting omits mobile-robot/outdoor scenarios altogether.\n\n[-] Depth-fusion idea mirrors MM-Spatial[1] and Spatial RGPT[2], novelty is incremental and not rigorously compared.\n\n[-] RFT is a straightforward RLHF method (GRPO) without major modification.\n\n[-] RFT is demonstrated only for the 2 B model, the stronger 8 B version has no RFT results. Without 8 B-RFT evidence, the main claims that \"RFT stage fosters better reasoning ability\" (Line 258) & \"RFT stage provides powerful generalization ability\" (Line 264) are unsubstantiated at the scale likely to interest practitioners.\n\n[-] Even after “speed-ups” the model needs ~29 s per command on robot hardware, far from real-time.\n\n\n[1] MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs\n[2] SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models\n\nQUESTIONS:\n1. Demonstrate RFT at scale:  Please provide 8 B-RFT results and training stability logs.  If the larger model fails to converge or offers no gain, the claimed contribution of RFT is weakened.\n\n2. Expand and diversify the evaluation benchmark. RefSpatial-Bench contains only 200 images—100 samples for Location and Placement task, plus 77 for Unseen set. And all the scenes are indoor. Therefore, the statistical confidence is low and the setting fails to test outdoor spatial reasoning.  To substantiate the claim of general spatial understanding (Line 8), the benchmark must grow to at least 2000 images, and diversify its domains to include outdoor environments, such as street, warehouse and construction sites, so that depth noise, lighting variation and larger spatial scales are represented.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}