Title: SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Spatial Reasoning; Object Orientation; Robotic Manipulation
Abstract: While spatial reasoning has made progress in object localization relationships, it often overlooks object orientation—a key factor in 6-DoF fine-grained manipulation. Traditional pose representations rely on pre-defined frames or templates, limiting generalization and semantic grounding. In this paper, we introduce the concept of semantic orientation, which defines object orientations using natural language in a reference-frame-free manner (e.g., the ''plug-in'' direction of a USB or the ''handle'' direction of a cup). To support this, we construct OrienText300K, a large-scale dataset of 3D objects annotated with semantic orientations, and develop PointSO, a general model for zero-shot semantic orientation prediction. By integrating semantic orientation into VLM agents, our SoFar framework enables 6-DoF spatial reasoning and generates robotic actions. Extensive experiments demonstrated the effectiveness and generalization of our SoFar, e.g., zero-shot 48.7\% successful rate on Open6DOR and zero-shot 74.9\% successful rate on SIMPLER-Env.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSoFar proposed orientation is the missing piece in today’s spatial-reasoning VLMs. It proposes semantic orientation (language-grounded directions such as the “plug-in” side of a USB), learns it at scale with the new OrienText300K dataset, and deploys a 3-D transformer (PointSO) plus a VLM reasoning layer to plan 6-DoF robot actions that respect both where and which way. The system lifts zero-shot success on representative 6-DoF manipulation benchmarks (48.7 % on Open6DOR-v2, 74.9 % on SIMPLER) and wins 60 diversified real-world tasks without task-specific finetuning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\nThe paper pinpoints orientation as the missing half of spatial reasoning, then formalises it as language-grounded directions (e.g., “plug-in side of a USB”). To learn this at scale, the authors create OrienText300K, a 350 k-object dataset annotated via GPT-4o prompts, giving the community the first large-coverage supervision for orientation understanding. \n\nOn standard 6-DoF manipulation benchmarks, SoFar lifts accuracy to 48.7 % on Open6DOR-v2 and 74.9 % on SIMPLER, clearly ahead of prior work. In the real world it succeeds on 60 diverse tabletop tasks spanning position-only, orientation-only, and full 6-DoF tracks, beating baselines across the board—demonstrating impressive generalisation from purely synthetic training. \n\nWeakness:\n-Heavy reliance on high-quality RGB-D; robustness to noisy or occluded depth is only briefly touched.\n\n-Task diversity: while 60 real-world tasks is commendable, many describe pick-and-place style motions; articulated objects and continuous-rotation tasks remain sparse.\n\n-PointSO + SAM + VLM reasoning may be slow for real-time reactions—latency numbers are not reported.\n\n-It will be good to compare with PRISM dataset from graspmolmo, to see how that fairs.\n\nQUESTIONS:\nRefer to the weakness\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper presents SOFAR, a framework incorporating semantic orientation into spatial reasoning and robotic manipulation. It introduces the concept of semantic orientation, linking object orientations to natural language descriptions. To support this, the authors construct OrienText300K, a dataset of 3D objects with semantic orientation annotations, and develop PointSO, a model for semantic orientation prediction. SOFAR integrates PointSO with foundation models like SAM to enable 6-DoF spatial reasoning and generate robotic actions. Experiments demonstrate SOFAR's effectiveness and generalization in tasks such as object manipulation and navigation.

STRENGTHS AND WEAKNESSES:
The paper introduces the novel concept of semantic orientation, linking object orientation with natural language to bridge geometry and function. It proposes PointSO, a cross-modal 3D Transformer that predicts semantic orientation from point clouds and text. Extensive experiments show strong generalization, with zero-shot success rates surpassing existing VLM and VLA methods in both simulation and real-world tasks.

The authors do not provide detailed descriptions of the CUDA memory and time costs for training and inference of the methods listed in Table 1. This lack of information makes it difficult to fully assess the computational efficiency and resource requirements of the proposed method and baseline methods. Could the authors provide a detailed explanation for the significant performance gap of SOFAR in the drawer opening task in Table 4.

QUESTIONS:
See Weaknesses

LIMITATIONS:
See Weaknesses
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main innovation of this paper is the introduction of a new representation named Semantic Orientation (SO), for spatial reasoning and object manipulation. This representation binds natural language with orientation of objects, enabling the understanding of semantic information associated with different object orientations. \n\nCompared to existing representation such as pose estimation, SO incorporates semantic meaning for each orientation, making it more suitable for functional understanding and affordance reasoning. In contrast to Affordance Map,which highlight possible interactions but ignore orientation, SO explicitly encodes directional information. Thus SO addressing some of the limitations present in prior representations for object manipulation.\n\nIn addition, the paper presents the PointSO model for Semantic Orientation prediction, along with a new large-scale dataset, OrienText300K, used to train this model. Finally, the authors propose the SOFAR system, which integrates PointSO with GPT, Florence2, and SAM to enable comprehensive spatial reasoning and object manipulation.\n\nSTRENGTHS AND WEAKNESSES:\nThe overall writing quality of the paper is high, and the exposition is clear and well-structured. The proposed representation, offers a novel perspective by associating natural language with object orientation. The method is relatively simple yet well-designed, and its effectiveness is supported by empirical results.\n\nQUESTIONS:\n1.For the line 170-171, the description of the \"edges\" in the 6DoF scene graph is unclear. \nAuthors should introduce more detail of the edge of this scene graph. \nIn addition, I suggest author provide at least one complete JSON-format Scene Graph as an extension to the one in Figure 5 to help readers better understand how to represent the node and edge of the graph in practical.\n2.How does the proposed SO representation or the SOFAR system determine which part of an object to grasp during manipulation? For instance, without relying on a affordance map, how does the model infer that the handle of a knife should be grasped rather than the blade?\n\n3.Currently, each object is only roughly divided into six directions, representing the whole object rather than specific parts. This limits the method to coarse manipulation. For example,task like in \"ReKep\", pour the tea to the cup,the robot needs to align the spout of the kettle not just the general orientation. The current approach overlooks such practical requirements for fine-grained manipulation.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose SoFar that integrates object orientation with spatial understanding and manipulation. The authors studies the \"semantic orientation\" that extends standard 6D pose estimation with language conditions, e.g., \"pick up\" or \"back\". Moreover, SoFar integrates a range of visual foundation models and parsing modules and builds a complete parsing of the scene graph. With a strong visual parsing (PointSO and previous visual foundation models) and language reasoning (ChatGPT), SoFar achieves improved results on a range of tasks, especially on ones that require rotation understanding.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. Integrating 6D orientation for spatial understanding and manipulation is an interesting and important research direction.\n2. Experiments a thorough and well-designed. Most claims are supported by experimental results and benchmark evaluations demonstrate the advantages of the presented system.\n\n**Weaknesses:**\n\n1. SoFar claims to be an object manipulation model that advances spatial reasoning and orientation understanding. Although explicit 6D orientations as inputs can assist 6D spatial reasoning (as also studied in [A]), it is unclear if the orientation/pose knowledge can truly benefit direct object manipulation, e.g., in a VLA system.\n2. The novelty of semantic orientation is limited. It has been studied in prior works: (1) how to align orientations of open-vocabulary objects [A,B], and (2) how to align canonical poses of objaverse objects [B]. Semantic orientation seems to be a trivial extension--e.g., once the 6D orientation of an electric fan is estimated, the \"pick up\" or \"back\" direction can be inferred from a simple transformation (with the help of a language model).\n3. Is semantic orientation representation necessarily a better option than standard 6D orientation in the SoFar system? Although current PointSO is finetuned with language-orientation paired data that can directly predict \"take picture\" orientation, using standard 6D orientations in the JSON scene graph should work well too with sufficient finetuning.\n4. The spatial reasoning evaluation in Table 6 seems unfair and biased. For models in \"VLMs with Spatial Awareness\", SoFar seems to be the only model finetuned with 6DoF-related data. Also, besides depth, SoFar also exploits segmentation and pose expert models.\n5. Some key technical details are missing and ablation study results are needed, e.g., **Q3** and **Q4**.\n\n[A] 3D-Aware Visual Question Answering about Parts, Poses and Occlusions.\n[B] ImageNet3D: Towards General-Purpose Object-Level 3D Understanding.\n[C] Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models.\n\nQUESTIONS:\n1. Is the PointSO trained on the same set of \"semantic orientations\" as used in the downstream spatial understanding and manipulation? Does PointSO general to unseen and novel \"semantic orientations\"?\n2. How is the \"time cost\" in Table 6 estimated?\n3. For various modules in Figure 5, how frequent are they executed (e.g., FPS)? What's the running time of various systems?\n4. What is the VLM used in SoFar in Table 1?\n\nLIMITATIONS:\nThe author mentioned decoupled design of SoFar as a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}