SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors
Authors: Chenyang Ma
Kai Lu
Ta-Ying Cheng
Niki Trigoni
Andrew Markham
Venue: NeurIPS 2024 poster (2024)
Keywords: VLM Spatial Reasoning
Zero-Shot
Primary area: machine_vision
TL;DR: We present SpatialPIN, a framework designed to enhance the spatial reasoning capabilities of VLMs through prompting and interacting with priors from multiple 3D foundation models in a zero-shot, training-free manner.

Abstract: Current state-of-the-art spatial reasoning-enhanced VLMs are trained to excel at spatial visual question answering (VQA). However, we believe that higher-level 3D-aware tasks, such as articulating dynamic scene changes and motion planning, require a fundamental and explicit 3D understanding beyond current spatial VQA datasets. In this work, we present SpatialPIN, a framework designed to enhance the spatial reasoning capabilities of VLMs through prompting and interacting with priors from multiple 3D foundation models in a zero-shot, training-free manner. Extensive experiments demonstrate that our spatial reasoning-imbued VLM performs well on various forms of spatial VQA and can extend to help in various downstream robotics tasks such as pick and stack and trajectory planning.

SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors
Chenyang Ma Niki Trigoni
Kai Lu
Ta-Ying Cheng
Andrew Markham
University of Oxford
chenyang ma@cs. or.ac.uk
Abstract
Current state-of-the-ar spatial reasoning-enhancee VLMs are trained to excel at spatial visual question answering (VQA). However, believe that higher-level 3D-aware tasks, such as articulating dynamic scene changes and motion planning, require fundamental and explicit 3D understanding beyond current spatial VQA datasets. this work, we present SpatialPIN, a framework designed to enhance the spatial reasoning capabilities of VLMs through prompting and interacting with priors from multiple 3D foundation models in a zero-shot, training-free manner. Extensive experiments demonstrate that spatial reasoning-imbor VLM performs well on various forms of spatial VQA and can extend to help in various downstream robotics tasks such as pick and stack and trajectory planning.
1 Introduction
Equipping vision-language models (VLMs) the capacities of spatial reasoning unlocks exciting applications. such as general-purpose reward annotation [52) robotic data generation [61]. and grounding 3D object affordances [26] 38). However, the spatial reasoning capabilities VLMs on fine-grained spatial understanding tasks are somewhat limited. Current state-of- state-of-the-ant ttt-hhhertt (SOTA) spatial reasoning-enhanced enhanced VLM [12 mostly tested on spatial visual question answering (VQA), such determining objects' relative positions and orientations; experiments higher-level tasks, such as scene comparisons and trajectory planning, which require more nuanced comprehension, are underexplored.
Many works enhance the spatial reasoning capabilities of VLMs by training/fine-tuning them on standard spatial VQA datasets [12]. As a result, VI.Ms primarily learn surface-level associations between image-text-data triplets. Given the scarcity and difficulty obtaining spatially rich embodied data or high-quality human annotations for 3D-aware queries, we hypothesize that these VLMs may not generalize to questions outside their dataset distribution or adapt to more challenging tasks that require advanced level of spatial understanding.
Recent studies 773 7 65| 69 in image space understanding show that VLMs, equipped with internet- scale language knowledge, and multimodal foundation models capture complementary knowledge that can combined to conduct new tasks spanning both modalities without additional training. Given the recent advancements in 3D foundation models 41 22), this work explores whether there exists an alternative approach to enhance VLMs with higher-level apatial-awareness by incorporating 3D priors from these models.
To this end, we propose SpatialPIN, a framework that utilizes progressive prompting and interactions between VLMs and 2D/3D foundation models as "free lunch" to enhance spatial reasoning capabilities shro,shot, training-free manner. By using these foundation models to decompose, comprehend,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

and reconstruct an explicit 3D representation, SpatialPIN grasps the core understanding of the 3D space presented by the 2D image. This allows generalizations various 3D-aware tasks, from VQAs to 3D trajectory planning.
We provide an extensive empirical study com- bining multiple off-the-shelf and handcrafted datasets, ranging from fundamental spatial ques- tions regarding relative positions and orienta- tions to providing fine-grained 3D information on objects' locations, sizes, inclinations, and dynamic changes, and plan for robotics tasks with full 3D trajectories. Results show that this straightforward approach significantly outper- forms SOTA VLMs trained from extensive spa- tial VQAs (see SpatialPIN examples in Figure D. consolidating our belief that truly 3D-aware VLM can actually be imbued by simply inject- ing explicit, fundamental knowledge of the 3D scene. With the entire framework being fully modularized, each component can be easily re- placed with the latest improvements within its specific domain.
Uier
mie dee pares
mernn oie gru en divel
ereen grko degaiie ched le of 30 5306
enall grop.
User
plylic boddy and
SparDIPIN
Iru sige the platic budioe the trah can Hitling butto aapeyda kami sudsi
Quaed GAIR,, (5S115 Hedk waee Nue colate tn Setteeeeeennn cin
We present SpatialPIN, framework to enhance the spatial reasoning capabilities of VLMs through prompting and interacting with 3D priors in a zero-shot, training-free manner.
Figure 1:
In summary, our main contributions are threefold:
We investigate the problem of equipping VLMs with 3D reasoning capabilities without fine-tuning on large spatial VQA datasets.
We propose SpatialPIN, a modular plug-and-play framework that progressively enhances VLM's 3D reasoning capabilities by prompting and interacting with 3D foundational models.
We show that SpatialPIN unlocks 3D-aware applications including spatial VQA and both classic and novel robotics tasks, supported by extensive experiments.
2
Related Work
VLM Grounding With the recent birth of powerful LLMs and VLMs 8 40 2]. the task of VLM grounding, combining generative language models with real-world data to adapt to specific cases, has gained significant popularity. Several recent works focused on fine-tuning these LLMS for a wide range of downstream applications, such as interactive decision making (34. multi-task agents [62), or even tasks in interactive environments (67 [T]. close work to ours is Socratic Model (73). a framework of combining multiple foundation models to unleash LLMs in downstream tasks. However, this work still focuses on tasks in 2D pixel space understanding of images. There remains many challenges in the 3D world to combine information for full scene understanding, which we hope to tackle in our paper.
VLM Spatial Understanding Many VLMs encompass the ability of image- reasoning and efforts in incorporating these understandings into image space manipulations and editting 7 65). However, the current ability of VLMs to fully understand a 3D scene and the potential interactions within this scene is still rather limited. Several works build from this foundation and establish datasets to help with spatial
understanding [3] 33 40). There
[30 39 45)
Recently, SpatialVL.M [12] proposed fine-tuning VL.M on 3D-VQA datasets enhance the precision of VLMs on 3D understanding tasks. Nevertheless, using a 3D-VQA dataset only provides a partial picture the complete 3D understanding of an image, and could lead to suboptimal performances under aatoo--distributin tasks. In this work, we hope to introduce holistic 3D information from multiple 3D foundation models via prompting and interactions as a way to enhance VLMs with comprehensive 3D understanding given RGB inputs.
3 Method
Given an RGB image / H a scene with K unknown objects and a spatial task Q, our goal is to inspire VLMs with spatial reasoning capabilities and solve Q with fine-grained 3D understanding.

---

Aj Understaddnng 20 Scene
B) Understanding 3D Soone
tehing with Roosniin
31 Seetan Hig and the careonn 2038 me
Lurge Soyrictlation
Coranol Dopth
Zooo Shot Sconee
Ac To stack the da the bowt
Surra Lie
Cluect sa-dial
o Whet the cissrica betwwen en
Underttanding Constraning ning
Coune Understndinng
Fine- 20 Undersanaing
How lmacc Tove
Noatpen snlone
Nuxel
Figure 2: SpatialPIN. Our plug-an audg-and--lay framework is fully modularized and designed for zero-shot deployment. Each module be easily replaced with the latest updates. Exact prompts for VLMs are in Appendix.
To prevent the models from overfitting to the standard problems from spatial VQA datasets [12, we hope to derive a method that utilizes fundamental 3D foundation models to provide explicit scene understandings, then leverage the generalization capabilities of VL.Ms to tackle unforeseen tasks all within a zero-shot, training-free manner.
Our modular pipeline, SpatialPIN, enhances VLMs' spatial understanding of image through progressive interactions with the scene decomposition, comprehension, and reconstruction processes with prompting. For image scene understanding (Sec. 3.1 we use VLM describe objects by segmentation and repainting models object masks. Elevating 2D understanding to coarse 3D (Sec. 3.2). we use
appearance and 2D location, complemented by language-guide to obtain occlusion metric depth estimation and perspective fields to estimate the 3D scene size and conduct perspective canonicalization with VLM. For fine-grained 3D understanding (Sec. (33), we partially reconstruct the 3D scene, with the full 3D representation of foreground objects and the background as plane. With the reconstructed 3D scene, we summarize spatial information and prompt to the VL.M for various downstream tasks.
3.1 2D Image Scene Understanding
Objects Understanding by Constraining We start with querying VLM to identify and understand objects given 7. We explicitly ask VLM to describe the objects by precise color, texture, and 2D spatial locations. This step is vital for two reasons: 1) enhance VI.M's understanding of the objects, 2) differentiate between items of similar or identical categories and appearances.
Prompting:
As a concrete example, given the left image of Fig. 2 VLM outputs: "
object 0: Laptop of color rose gold, texture metallic location left-center. object 1: canera of color black, texture smooth at location center-right. ...""
2D Representations Refinement The concise descriptions of identified objects are used as input text prompts for language-guided segmentation model, enabling the acquisition of K segmentation masks [Moce]& Jk=1, with each mask corresponding to a unique object.
However, an object i E [], may be occluded by other object(s), leading to an incomplete mask M which may be burdensome when we elevate the image to 3D representation the later stage.
To resolve this, we create an inpainting mask, M,"", for each object, in which all objects except the one itself are removed and replaced with white pixels. The inpainted masks again fed to the lunguage-guidee segmentation model along with input text prompts such that occlusion -ree object masks, (M K [k=1> are obtained. This two-step segmentation process for object is formulated as:
(1) where seg denotes the language-guides segmentation and r; denotes the description of object i. In practice, to cleanly remove objects without residual fragments for inpainting, we apply dilation to and expand the white areas. Inpainted background 4ng is acquired by removing and replacing all objects with dilated white pixels.
M = seg(I, 7,), Mo seg(inaann((MM" TT),
3

---

3.2 Coarse 3D Scene Understanding
Scene Size Estimation Using the estimated metric depth [29] and estimated camera intrinsic matrix by finding field of view (FOV) through perspective fields 22 backproject to determine the dimensions of the 3D spatial scene.
Perspective Canonicalization 3D information without any knowledge regarding the camera perspectives lead ambiguities [[] Consider question "What is the orientation of the bowl relative to the laptop?" with the input scene in Fig. 2 but taken from top-down perspective. VLMs may oulput "downward to the left", but the correct answer should be "front-left" because humans perceive orientation from a horizontal angle. To address this, we provide the VLM with I, estimated scene size, and maximum and minimum dimensions, allowing it to reason about the camera Scene size information helps differentiate shot angles
Prompting:
shot angle htti//ont///oo-doww/oooottee by providing clues about spatial layout and object proportions. For instance, if the depth variation small, the VIM can infer top-down or bottom-up angle along with visual cues.
As concrete example, given the left image of Fig. 2 VLM outputs:
'Visual cues reasoning: Objocts are vieved from the side, indicating the camera is positioned horizontally with a slight elevation.
Spatial data reasoning: The depth varies significantly from 57.50 en to 115.00 indicating the camara captures the acena acroas different distances, supporting horizantal perapactive. Conclusion: horizental."
3.3 Fine-Grained 3D Scene Understanding
We partially reconstruct the 3D scene with full representation of foreground objects while sim- plifying the inpainted background as a plane, 3
as shown in Fig. a). We summarize spatial information from the reconstructed scene and prompt it to the VLM. Please see our Appendix for implementation details about reconstruction.
Cor
Figure 3: Our method of partial 3D scene reconstruc- tion (a). The reconstructed scene (b) and the input image (c) show high alignment.
Scene Initialization Given the occlusion-free
single-view 3D reconstruction model to acquire object 3D models, {O*} k= with canonical poses determined during reconstruction. Pinhole camera is set at the origin, looking at positive depth-axis. With the estimated background plane size (Sec. (3.2). move the background plane, Obg (visually identical to Ibg), along the depth-axis to fit precisely within the camera.
object masks, {M 1
Ik=1,
Scene Reconstruction To resolve the imprecision of backprojection, our goal is to position object 3D models into the reconstructed 3D scene without visual discrepancies and ensure accurate depth. Instead of using naive backprojection, for an object i E [1, K], we perform raycasting from object 3D center & on the camera plane to object 3D center dh on the background plane with metric depth d, The 3D coordinate t; of object is:
d2
d, Idep (ce t, =t=
bg (t
- te).
by
t
The rotation R3 of the 6D pose of object P: RR 4.] is explained previously. After integrating all 3D object models into the 3D scene, we refine each object's scale to accurately reflect depth variations by rendering binary masks and evaluate the length of their contour lines relative their occlusion-free masks, through the lens of the pinhole camera, tc.
We determine the principal axes (x-axis, y-axis, and z-axis) of each object using the minimal oriented bounding box (OBB), which is essential for unlock novel applications.
Objects and Spatial Context Understanding The reconstructed 3D scene from I with accurate object poses and scales is denoted as Vo. As the final step of progressive prompting, we feed VLM the fine-grained 3D information derived from Vo. grounding on the canonicalized perspective (Sec. 3.2). For example, with the input image in Fig. and horizontal camera shot angle, depth corresponds to the positive y-axis (similarly, in top-down/bottom-uu view, depth is
Prompting:

---

leptop forther to reparet
75.4 makkin Can Japlop Coctlats
Vey, lephop horkeer from the
PeArEe teight didfecusos betwer
The weggh aittrasan betrene boar
The dule leads taile tallec
GLEfaees uctac.
farees Jaben
vertanmal lowaling angle
that the estation segle betones Use
Saitg tie peodut Aouiada
the bor port tloeed diagonal angle
bottle appcoeinceely laacely degines.
kride and the mile nocate acound chdi
accose the inuce while the box apples
las misk loe destases
the slowe, Lodiesting Duis leung
Figure 4: Qualitative examples of spatial VQA. SpatialPIN outputs answers with fine-grained 3D reasoning. Zoom in for better view.
the negative/positive z-axis) in a right-handed coordinate system. The width and height axes can be determined accordingly, aligning each axis's orientation with human perception.
We feed VLM paragraph describing the objects' poses, sizes, and principal axes in physical units, alongside their spatial relationships. To augment VLM's understanding, we also feed Vo with visualized object axes (see Fig. 2B). Visualizing 3D spatial information pivotal in improving VLMs' understanding of 3D spatial contexts derived from 2D images, validated by 3DAxiesPrompts (37). Yet, we want to emphasize that we do not feed hardcoded information, such as objects' relative distances and inclinations, to VLMs. Instead, aim for the summarized 3D information to enhance VLMs' general spatial understanding.
As concrete example for the left image on Fig. 2
<bbj spatial context: 3D center: [7.0, 100.0, 9.0] I-axis (right): [0.9529, -0.2456, 0.1779];
Y-axis (back): [-0.3528, 0.8746, 0.3327]; ; Z-axis (up): [-0.1761, -0.3285, 0.9279]
ūbj 1 size: 13.54 9.37 9.50 (WxDa))
übj closest par direction:
left: Obj 0; right: Obj 2
3.4 Combining External Tools for Downstream Tasks
By partially reconstructing the 3D scene with visual alignments, our framework enables VL.Ms to use tools like rapidly-explorin random tree star (RRT*) [3]] to generate accurate, collision-fiee free paths based on task specifications (more details in Appendix). This capability unlocks novel and interesting applications when combined with task-specific prompting techniques, shown in Experiments (Sec
Experiments
We conduct experiments to answer the following questions: 1) Does our framework enhance the general spatial reasoning capabilities of VLMs, and how well does it perform? 2) What novel applications does our framework unlock for VLMs, and how well do we perform in these applications? 3) How effective is each module in framework?
Since we evaluate our approach on wide range of tasks test VLMs' higher-level spatial awareness, some tasks are novel and lack existing/open source datasets. Therefore, for all our experiments, we use a combination of 4 existing datasets and 2 hand-crafted datasets.
Implementations The language-guicee guided segmentation model is Language Segment-Anything 44 and the repainting model LaMa [55]. We use One-2-3-45++ [4] for single-view 3D reconstruction, perspective fields 22 for camera intrinsic estimation, and ZeeDepth [4] for depth estimation. For partial 3D scene reconstruction, we use Blender [17] as the 3D software. All inference is run on
NVIDIA A10 GPU with 24GB RAM.
4.1 Spatial Visual Question Answering
We experiment on the basic form of spatial VQA introduced by SpatialVIM (laOR-VQA), and two new forms introducted by us (laAD- & I-SD-VQA). For IaOR-VQA, please check SpatialVI.M [2 for details. For laAD- & InSD-VOL VQ-, please Appendix.
5

---

Intra-Image Object Relations VQA (IaOR-VQA) As the basic form of spatial VQA, it involves spatial reasoning about object relative orientations and sizes. This is divided into qualitative (e.g., "is [A] in front of [B]". "is [A] smaller than [B]") and quantitative (e.g., "how far apart are [A] and [B]", "measure the width of [A]") questions.
We follow the evaluation method of SpatialVLM [2]. Since Spatial VLM did not release their evaluation dataset, reproduce one using RGBD images from NOCS [57] (object dataset), RT-1 (6) and BridgeData V2 [56] (robotics manipulation datasets). We sample 13, 20, and 20 distinct scenes from each. We generate QA pairs using the SpatialVLM data generation pipeline [5]]. followed by manual refinement. We check correctness for qualitative questions and calculate distances for quantitative questions. We annotate 300 qualitative and 200 quantitative spatial VQA pairs [SpatialVLM, has 331 and 215 for each).
Intra-Image Angular Discrepancies VQA (IaAD-VQA) We propose a new form of Spatial VQA that needs spatial reasoning about objects' inclinations. It includes qualitative (e.g.. "is [A] tilted", "is [A] more tilted than [B]") and quantitative questions (e.g., "how many degrees is [A] tilted vertically", "measure the angle between [A] and [B]").
Since this form of Spatial VQA involves out-of -ft-neee rotations, YCBInEOAT 64 (object tracking dataset) is a suitable choice. We sample 30 scenes from it and annotate 50 questions each for qualitative and quantitative spatial VQA pairs.
Inter-Image Spatial Dynamics VQA (IrSD-VQA) We further propose a more challenging form of Spatial VQA. Given two images with multiple objects, the objects in the second image may move, rotate, incline, or the image may have a change camera angle. The VLM needs to reason about these changes. Example qualitative questions include "does [A] move, rotate, or incline", "does [A] incline along the y-axis" while quantitative questions include "how far does [A] move", "how many degrees does [A] rotate horizontally".
As it is difficult to find a dataset that meets these requirements, we craft our own. We capture 20 image pairs using an iPhone 12 Pro Max, with each image containing - 5 objects, and annotate 50 questions each for qualitative and quantitative spatial VQA pairs.
The results in Tables D and on qualitative and quantitative -OORVQQA demonstrate that providing various VL.Ms fnneggriied 3D information enhances their spatial reasoning capacities by a large margin. Surprisingly, VLMs with math and geometry reasoning capacities (c.g., GPT-4V, GPT-4o) show substantial improvements with this information.
Results
Table 1:
Qualitative IQOR-VQA. We exclude comparisons to PaLl [.] PaLM-E 20 and PaLM 2-E 3 as they not open source, and include experiments with GPT-4o ① and InstructBLIP [18). We use the HF version of SpatialVL.M
in addition GPT-4V
LLaVA- 1.5 40)
GPT-VV
GPT-4o
LLaVA-1.5
InstructBLIP
SpatialVLM
w/o
w/o
ours
w/o ours
ours
w/o ours
ours
Accuracy
70.7
86.3
69.0
87.3
70.0
83.0
62.3
79.3
76.7
Table 2: Quantitative IaOR- VQA. Sp measures the accuracy by the percentage of answers that fall within 0.5x to 2.0x of the ground truth value. We also evaluate within narrower ranges of 0.75x to 1.33x and 0.9x 1x. "Output number" means VLMs produce number the response instcad of vague descriptions.
GPT-4V
GPT-4o
LLaVA-1.5
InstructBL.IP
SpatialVLM
w/o ours
w ours
w/o ours
W ours
w/o ours
W ours
w/o ours
w ours
Output numbers
0.8
98.5
31.5
99.5
23.5
97.0
28.5
98.5
91.0
range [50, 200] %
0.0
73.5
14.0
74.5
16.5
43.0
8.0
31.5
33.5
range [75, 133] %
0.0
69.5
70.5
6.0
29.0
2.5
22.0
20.5
range (90, 111]
0.0
54.5
3.0
55.0
2.0
14.5
0.0
11.5
7.5
The results in Tables and demonstrate the effectiveness of our approach both qualitative and quantitative IaOR- VQA and IrSD-VQA tasks. Notably, the performance on quantitative IaOR-VQA suboptimal compared to quantitative IrSD-VQA, despite the latter being more challenging. We
4

---

Table 3: Qualitative IAAD-VQA & IOSD-VQA. Since SpatialPIN VLM backbone for proposed spatial VQA, for fair comparison. we should use SpatialVLM backbone (PaLM 2-E B. However, since it is not open source, we use GPT-4o as our backbone, as it shows the most improvement with our framework.
IaAD-VQA
IrSD-VQA
GPT-4o
GPT-4o + ours
SpatialVLM
GPT-40
GPT-4o ours
SpatialVLM
Accuracy %
68
84
62
64
82
54
Table 4: Quantitative IAAD-VQA & IrSD-VQA.
IaAD-VQA
IrSD-VQA
GPT--4
GPT-40 + ours
SputiaVVL
GPT-4o
GPT-4o + ours
SpatialVL
Oulput numbers
38
100
66
100
In range [500, 200] %
8
64
12
10
68
26
range [75, 133]
In range [90. 111] %
0
30
2
2
38
believe this is because, for quantitative IrSD-VQA, the VLM sometimes confuses the camera and world coordinate frames, comparing the object's principal axes with the world axes to reason about changes in angles.
Fig. 4 presents qualitative examples on all forms of spatial VLM.
4.2 Robotics Pick and Stack
Pick and stack is a classic robotics task. Given a robot's egocentric observation of scene with multiple objects and task description, our pipeline uses traditional planning to solve the problem. This task demands advanced spatial reasoning, the model must comprehend 3D locations, sizes, and physical properties of the objects (i.e., how much to grasp and how high to drop? the object deformable or articulated so the robotic grasper needs to grasp more firmly?). For instance, grasping and stacking a soft toy bear on a cube significantly different from stacking a solid apple on a mug. The model reasons about grasping and stacking policies, directly outputting 3D trajectories for the robot's end effector using traditional path planning algorithm as external tool.
Set-Up We set up the pick-and-stack problem in the ManiSkill 22 simulator, applying real-world physics properties. Rigid and articulated objects are chosen from the YCB dataset [10] and are randomly allocated on the table within the robotic arm's reach, with observations from different perspectives. We create 50 scenes. Since robot observations from simulated scenes suffer from sim2real gap and consider that most real-world robots have depth sensors, we use ground truth camera matrix and depth.
We compare our method to the following baselines: 1) direct 3D information output from our framework without GPT-4o [1] reasoning about physics and object properties and 2) SpatialVLM with our RRT* trajectory generation module.
Results 5 the results, with a qualitative example demonstrated in Fig. 5 The results indicate that using precise 3D information from our framework significantly improves the success rate, and incorporating VLM reasoning further enhances performance.
4.3 Discovering and Planning for Robotics Tasks from Single Image
We present novel task that requires advanced spatial reasoning capacities of VLMs. Given single RGB image of any scene comprising unknown environments and objects, the VLM discovers potential tasks and plans their execution with full 3D trajectories, with the motivation that it can be used for robot learning in future research. To solve this complex task and visualize the execution using our framework, we introduce: 1) task proposal approach using VLM, planning approach that enables spatial reasoning-imie imbued VLM to plan the object motion based on the proposed tasks by specifying waypoints. Please Appendix for the pipeline and details.
novel axes- ches-constr 3D
Dataset We create a diverse evaluation dataset by combining self-captured photos (38) using an iPhone 12 Pro Max and scenes (13) from NOCS [57]. Our dataset covers diverse scenes (e.g., office,

---

est stack the cracher boe Detemine grosper' cleatin distance the optinal beight using lmuge and iedrematiion
graap
"Oe zdout should grapp the besasa at ita Dalassst sade with sezs sf apperainateiy 1.5 as Gust lesa than the
banans haight of 3,05 oa plce a neegial and deop 11 from 22.36 to stact seczenly on the cracker bor. Vadre
"The knide ie used to slice the orange into halvez or vedgus on the cutting beard.
Planned Task Trijctooes
Figure 5: Qualitative examples of pick and stack (top) and task trajectory planning (hottom). SpatialPIN successfully outputs picking and stacking policies using spatial reasoning plans 3D trajectories with geometric awareness to align with task descriptions.
Table 5: Pick and stack. We classify the success rates into: 1) successfully picked, 2) successfully picked and contacted the target object but slipped/collided, and 3) successfully picked and stacked.
GPT-40 ours
Our Direct 3D Info
SpatialVLM RRT
Picked %
44
Picked & contacted %
8
12
Picked stacked %
36
kitchen, bathroom), and features a rich diversity of object categories (116) and quantities (185), with each image containing 1 7 objects and trajectories in total). The dataset's s diversity is further enhanced by the variety of perspectives (e.g., frontal, top-down, side views). This deliberate choice of diverse angles, both in capturing process and through the random extraction frames from NOCS, aims to simulate a realistic and challenging array of scenes for evaluation. See Appendix for statistics and visuals.
tasks proposed for each object (278 tasks/planned
image
Qualitative Demonstration We present qualitative example in Fig. 5 Additional examples in Appendix shows our framework's capability to produce diverse and accurate task trajectories spanning various scenes and tasks.
Human Evaluation: User Study We rely human preference evaluation as one of our quantitative metrics. We ask 25 users to rate 5 translation and 5 rotation task executions in terms of task description alignment. For these complex context-dependent manipulation tasks, we instead ask users to judge 10 executions relative human action, and encapsulate their perception of the action in our with a single sentence. These sentence description will be used test human understanding our planned trajectories (please Ap- pendix). Note that our user study size is similar to those representative works such ControlNet [74 and Trompt-to-Promp
Table 6: User study. Ratings (scale 5) are averaged.
Rating
Rotation Translation Manipulation
4.58
4.43
4.29
Results Table. 6
Machine Understanding We assess the interpretability of our generated task executions from machine's perspective using SOTA video understanding model, Video-LLaVA-7B [35]. We use two approaches: binary classification and descriptive generation. For classification, with the task descriptions generated by VLM and ask question (is the video doing... .?). In generation, we prompt Video-LLaVA-78 articulate its interpretation our task executions. To quantify the correspondence between the model's perception and the tasks, we use OpenCLIP cosine similarity score [.5]
feed the model

---

Table 8: Ablation study. For quantitative IaOR-VQA, the accuracy is measured by the answers that fall within 0.75x to 1.33x of the ground iruth value.
Overall Design
2D Understanding
3D Understanding
ShAPO
6AM-6D 3D models
Spa
w/o objects
w/o coarse
wio fine grained
w/o both
Qualitative 36.7
48.0
81.3
68.3
76.0
63.3
61.7
Quamitativv
29.5
37.0
62.5
54.5
64.5
50.5
58.0
70.5
However, we find that even SOTA video understanding model shows limited performance. To assess false positive rate in classification, we deliberately misalign the sequence of generated task executions with their corresponding task descriptions, expecting a theoretical accuracy of 0%. Contrary to expec- lations, Vidco-LLaVA-7B reports a false positive rate of 36.3%. To adjust for this anomaly, we subtract this rate from the model's raw accuracy for correctly aligned video-task pairs. This method, while unconventional, pro- vides more fair and reasonable evaluation of machine video understanding, underscoring the current challenges faced by video understanding models in accurately interpreting complex video content. Results in Table.
Table 7: Results for machine understanding (classification and gen- eration) on 278 task ex- ecutions.
Machine
Raw Ace f
0,974
Fal-Pos Rate
0.363
True Aoc
0.611
OpenCLIP
0.636
4.4
Ablation Study
We evaluate the effectiveness of each module in our framework on IaOR-VOS by 1) seeking alternative designs of the overall pipeline and 2) removing each component in ablations.
Overall Design To demonstrate our framework's s generalization across wide range of objects, We replace our 2D + 3D pipeline with: 1) SOTA mesh-free single image object pose and size estimation model, ShAPO (27). SOTA mesh-based single image object pose and size estimation model, SAM-6D 36. and feeds it with the object 3D model reconstructed by One-2-3-45++ 41., and 3) the data generation backbone of SpatialVLM [2]. Since models 1) and 2) do not provide language annotations for their outputs, we first summarize the numerical outputs using our approach in Sec. 3.3 Then, GPT-4V identifies QA pairs.
Removing 2D Understanding Module In this case, the VLM no longer examines the objects through prompting, and only the object name is input into the language-guided segmentation model.
Removing 3D Understanding Modules This means there is no scene size estimation, and the VLM does not conduct perspective canonicalization. During 3D scene reconstruction, we assume the image plane width to be meter.
To validate the fine- ane-grained 3D scene understanding module, we replace object mask raycasting with backprojection using the object's 's 2D center and remove the object scale calibration.
To demonstrate the overall effectiveness of our 3D understanding modules, we simply backproject the input image with the estimated metric depth.
Results Table BableBdemoosttatte the effectiveness of each module in our framework. The results also
highlight the limitations of using aff-the-shee SOTA mesh-free and mesh-based angle-image object pose and size estimation methods as our backbone. These methods are not language-driven and may struggle to generalize to novel objects in diverse input scenes.
5 Discussion and Conclusion
We present SpatialPIN, a framework designed to enhance the spatial reasoning capabilities of VLMs through prompting and interacting with 3D priors in a zero-shot, training-free manner. We see our work as a step towards equipping VLMs with more generalized spatial reasoning capacities, demonstrated through applications in various forms of spatial VQA and both traditional and novel robotics tasks.
Limitations Readers may be curious about the inference speed of our framework. The bottleneck is the 3D object reconstruction process and the API call to closed-source VLMs (~ 20 seconds per image). However, want highlight that this process runs only once per image, and the speed is expected to improve with future versions of 3D foundation models.

---

References
[1] Open AI. Hello gpt-4o.
http://eii...mmmmeeeeeeeelll---o wapphee-looppppo/o/ppo 2024.
[2] Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonogiou, Julian Schrittwieser, Amelia Glaese. Jilin Chen, Emily Pitler, Timothy P. Lillicrap. Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Is- ard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Syg- nowski, and al. Gemini: family of highly capable multimodal models. arXiv preprini arXiv:2312. / 2023.
[3] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, and et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
[4] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. 2023.
[5] Reiner BirkI, Diana Wofk, and Matthias Müller, Midas v3.1 A model zoo for robust monocular relative depth estimation. arXiv preprint 1xXiv:2307.144600 2023.
[6] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, and et al. RT-1: robotics transformer for real-world control at scale. In Robotics: Science and Systems, 2023.
[7] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Conference on Computer Vision and Pattern Recognition, pages 18392-18402. IEEE, 2023.
[8] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[9] Dingding Cai, Janne Heikkilä, and Esa Rahtu. OVE6D:
object viewpoint encoding for depth- based 6d object pose estimation. In Conference on Computer Vision and Pattern Recognition, pages 6793-6803. IEEE, 2022.
[10] Berk Çalli, Arjun Singh, Aaron Walsman, Siddhartha S. Srinivasa, Pieter Abbeel, and Aaron M. Dollar. The YCB object and model set: Towards common benchmarks for manipulation research. In International Conference on Advanced Robotics, pages 510-517. IEEE, 2015.
[11] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference Machine Learning, pages 3676-3713, 2023.
[12] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialvIm: Endowing vision-language models with spatial reasoning capabilities. arXiv preprint arXiv:2401.12166 2024.
[13] Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, and Geoffrey E. Hinton. Pix2seq: A language modeling framework for object detection. International Conference on Learning Representations, 2022.
10

---

[14] Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong. Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V. Thapliyal, James Bradbury, and Weicheng Kuo. Pali: A jointly-scaled multilingual language- image model. In International Conference Learning Representations, 2023.
[15] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Conference on Computer Vision and Pattern Recognition, pages 2818 2829. IEEE, 2023.
[16] Jaehoon Cho, Dongbo Min, Youngjung Kim, and Kwanghoon Sohn. DIML/CVL RGB- dataset: 2m RGB-D images of natural indoor and outdoor scenes. arXiv preprint 1rXiv:2110.11590, 2021.
[17] Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
[18] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems, 2023.
[19] Murtaza Dalal, Tarun Chiruvolu, Devendra Singh Chaplot, and Ruslan Salakhutdinov. Plan-seq- learn: Language model guided RL for solving long horizon robotics tasks. In CoRL. Workshop on Learning Effective Abstractions for Planning, 2023.
[20] Danny Driess, Fei Xia, Mchdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. International Conference on Machine Learning. pages 8469-8488, 2023.
[21] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research, pages 1231-1237, 2013.
[22] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunehao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills. In International Conference on Learning Representations, 2023.
[23] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou MU, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskil12: unified benchmark for generalizable manipulation skills. In Conference on Learning Representations, 2023.
[24] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kńr Aberman, Yael Pritch, and Daniel Cohen-Or. image editing with cross- cooss-attention control. In International Conference on Learning Representations, 2023.
[25] Ming-Kuei Hu. Visual pattern recognition by moment invariants. IRE Transactions Informa- tion Theory, pages 177, 1962.
[26] Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. arXiv preprint arXiv:2403.08244. 2024.
[27] Muhammad Zubair Irshad, Sergey Zakharov, Rares Ambrus, Thomas Kollar, Zsolt Kira, and Adrien Gaidon. Shapo: Implicit representations for multi-object shape, appearance, and pose optimization. In European Conference Computer Vision, pages 275-292, 2022.
11

---

[28] Stephen James, Zicong Ma. David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark learning environment. Robotics and Automation Letters, pages 3019-30260 2020.
[29] Linyi Jin, Jianming Zhang. Yannick Hold-Geoofrro, Oliver Wang, Kevin Blackburn-Matzent Matthew Sticha, and David F. Fouhey. Perspective fields for single image camera calibration. In Conference on Computer Vision and Puttern Recognition, pages 17307-17316. IEFE, 2023.
[30] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Conference Computer Vision and Pattern Recognition, pages 1988-1997. IEEE, 2017.
[31] Sertac Karaman and Emilio Frazzoli. Sampling- based algorithms for optimal motion planning. The International Journal of Robotics Research, pages 846-894, 2011.
[32] Kourosh Khoshelham and Sander Oude Elberink. Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors, pages 1437-1454, 2012.
[33] Alexander Kirillov, Eric Mintun. Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv preprint (xXxx2304.02643 2023.
[34] Shuang Li, Xavier Puig, Chris Paxton, Yilun Du Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, and Yuke Zhu. Pre-trained language models for interactive decision- decision-making. In Advances in Neural Information Processing Systems, 2022.
[35] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122. 2023.
[36] Jiehong Lin, Lihua Liu, Dekun Lu, and Kui Jia. SAM-6D: segment anything model meets zero-shot 6d object pose estimation. arXiv preprint arXiv:2311. 2023.
[37] Dingning Liu, Xiaomeng Dong, Renrui Zhang, Xu Luo, Peng Gao, Xiaoshui Huang, Yongshun Gong, and Zhihui Wang. 3daxiesprompts: Unleashing the 3d spatial task capabilities of GPT-4V. arXiv preprint arXiv:312.0977 2023.
[38] Fangchen Liu, Kuan Fang. Pieter Abbeel, and Sergey Levine. MOKA: open-vocabulary robotic manipulation through mark-based visual prompting. arXiv preprint 4rXiv:2403.03174, 2024.
[39] Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 2023.
[40] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023.
[41] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint arXiv:2311.0788) 2023,
[42] Yuan Liu, Yilin Wen, Sida Peng, Cheng Lin, Xiaoxiao Long, Taku Komura, and Wenping Wang. Gen6d: Generalizable model-free 6-dof object pose estimation from RGB images. In European Conference on Computer Vision, pages 298-315, 2022.
[43] Chenyang Ma, Xinchi Qiu, Daniel Beutel, and Nicholas Lane. Gradient-less federated gradient boosting tree with learnable learning rates. In Proceedings of the 3rd Workshop on Machine Learning and Systems, pages 56-63, 2023.
[44] Luca Medeiros. Language segment-anything.
httpe///gthuub.com//uaaaaaaaa
Long-segment.o wwgment--ttttiinnngnggggnneneget anything 2023.
12

---

[45] Oscar Michel, Anand Bhattad, Eli VanderBilt, Ranjay Krishna, Aniruddha Kembhavi, and Tanmay Gupta. OBJECT 3DIT: Language-guide guided 3d-aware image editing, In Advances in Neural Information Processing Systems, 2023.
[46] Van Nguyen Nguyen, Thibault Groueix, Georgy Ponimatkin, Vincent Lepetit, and Tomas Hodan. CNOS: strong baseline for cad-based novel object segmentation. International Conference on Computer Vision Workshops, pages 21 2126-2132. IEEE, 2023.
[47] OpenAl. GPT-4 technical report. arXiv preprint 2r33v:3303.08774, 2023.
[48] Xinehi Qiu, Heng Hen, Wanru Zhao, Pedro Porto Buarque de Gusmao, and Nicholas Donald Lane. Efficient vertical federated learning with secure aggregation. In Federated Learning Systems (FLSys) @orkshop@ MIsSys 2023, 2023.
[49] Xinchi Qiu, Heng Pan, Wanru Zhao, Chenyang Ma, Pedro PB Gusmao, and Nicholas D Lane. vfedsec: Efficient secure aggregation for vertical federated learning via secure layer. arXiv preprint arXiv:2305.16674, 2023.
[50] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions Pattern Analysis and Machine Intelligence, pages 1623-1637, 2022.
[51] remyxai. Vqasynth, 2024. GitHub repository.
[52] Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- language models are zero-shot reward models for reinforcement learning. arXiv preprint arXiv:2310.1222 122221,,, 2023.
[53] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from RGBD images. In European Conference on Computer Vision, pages 746-760, 2012.
[54] Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto
Fei Xia, Kent El- liott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. BEHAVIOR: benchmark for everyday household activities in virtual, interactive, and ecological environments. In Conference on Robot Learning, pages 477-490, 2021.
[55] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In Winter Conference on Applications of Computer Vision, pages 3172-3182. IEEE, 2022.
[56] Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
[57] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Conference on Computer Vision and Pattern Recognition, pages 2642-2651. IEEE, 2019.
[58] Qiang Wang, Shizhen Zheng, Qingsong Yan, Fei Deng, Kaiyong Zhao, and Xiaowen Chu. IRS: A large naturalistic indoor robotics stereo dataset to train deep models for disparity and surface normal estimation. In International Conference on Multimedia and Expo, pages 1-6. IEEE, 2021.
[59] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian A. Scherer. Tartanair: dataset to push the limits of visual SLAM. In International Conference on Intelligent Robots and Systems, pages 4909 IEEE, 2020.
(60] Yen-Jen Wang, Bike Zhang, Jianyu Chen, and Koushil Sreenath. Prompt a robot walk with large language models. arXiv preprint 23Xiv2230.009969 2023.
13

---

[61] Yufei Wang, Zhou Xian, Feng Chen, Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:231 1.01455, 2023.
Wang, Yian Wang, Zackory Erickson, David
[62] Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain. plan and select: Interactive planning with Ilms enables world multi-task agents. In Advances in Neural Information Processing Systems, 2023.
[63] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems, 2022.
[64] Bowen Wen, Chaitanya Mitash, Baozhang Ren, and Kostas E. Bekris. se(3)-tracknet: Data- driven 6d pose tracking by calibrating image residuals in synthetic domains. In International Conference on Intelligent Robots and Systems, pages 10367-10373. IEEE, 2020.
[65] Tsung-Han Han Wu, Long Lian, Joseph E Gonzalez, Boyi Li, and Trevor Darrell. Self-correcting llm-controlled diffusion models. arXiv preprint arXiv:2311 16090, 2023.
[66] Ke Xian, Jianming Zhang, Oliver Wang. Long Mai, Zhe Lin, and Zhiguo Cao. Structure guided ranking loss for single image depth prediction. In Conference on Computer Vision and Pattern Recognition, pages 608-617. IEEE, 2020.
[67] Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. Language models meet world models: Embodied experiences enhance language models. In Advances in Neural Information Processing Systems, 2023.
[68] Fengyu Yang and Chenyang Ma. Sparse and complete latent organization for geospatial semantic segmentation. In Conference on Computer Vision and Puttern Recognition, pages 1809-1818, 2022.
[69] Fengyu Yang, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens. Touch and go: Learning from human-collected vision and touch. Advances in Neural Informa- tion Processing Systems, pages 8081-811 8803, 2022.
[70] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. arXiv preprint arXiv:2401. 10891, 2024.
[71] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Conference on Computer Vision and Pattern Recognition, pages 1787-1796. IEEE, 2020.
[72] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1100, 2019.
[73] Andy Zeng, Maria Attarian, Brian lchter, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S. Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. In International Conference on Learning Representations, 2023.
[74] Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffuusin models. In International Conference on Computer Vision. IEEE, 2023.
[75] Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In SIGIR, pages 1097-1100. ACM, 2018
14

---

Appendix for SpatialPIN
A Overview
This Appendix includes: 1) more technical details about our partial 3D scene reconstruction, 2) additional details, templates, and visualizations of our proposed two forms of SpatialVQA, 3) implementation details on our proposed application: discovering and planning for robotics tasks from single image (task proposal, axes-constrained motion planning through waypoints, and trajectory generation and smoothing), 4) more experiments our proposed application (dataset statistics, additional qualitative demonstrations, human understanding, and task diversity), and 5) prompt details for VLMs.
B Partial 3D Scene Reconstruction Details
Metric Depth Estimation Because significant portion of depth estimation model 70
50
68
is trained on depth datasets with depth data determined by sensors [21 53 and stereo matching 71
16 58
59 66), we assume that the predicted normalized depth is the perpendicular distance the camera plane, instead of straight line from the object to the camera lens 32]
Camera Intrinsic Estimation Given an RGB image, With the estimated vertical field of view (FOV), 0, through perspective fields (29), the camera focal length f can be found by:
H
2 tan
On T
where is the image height in pixels.
Object 6D Pose Estimation Single-view 3D reconstruction model reconstructs mesh O, of object i at the 3D origin, in the coordinate frame set by the input mask M, and captures O by a pinhole camera. This camera, with 6D pose P. = [Re jorigin], captures O 's canonical pose within the image. Thus, we can restore all objects' canonical poses across all images by identifying Po.
We use One-2-3-45++
which provides the camera pose. For models without this information. we develop an efficient method to determine the pose by comparing object masks with rendered 3D
model templates, inspired from matching- based 6D pose estimation works 33 46 42
We generate a set of object templates, denoted as (Ti each rendered from the object's 3D model O, These templates are created by positioning the camera various locations on icosphere surrounding the object in (3) space, which simulates spherical coverage around the object to capture its geometry from all angles uniformly. For each template 7J. we compute matching score against the occlusion-free object mask M.
We propose simple yet effective score matching method. We draw bounding rectangle around the segmented object inside M and across all (T) and crop the bounding rectangle. We then calculate the shape similarity between the contour line of cropped M and that of each cropped Tj using Hu moments 255 Additionally, we crop and resize the bounding rectangle to the same I dimension, and evaluate the similarity based on the pixel area of the cropped and resized masks. Our score matching method can be formulated as:
Hu HuffinCContou((crop(/ M ))),
pa sum(resizeecroppMA) (M. ))),
m
Hu LuffinCCottur(cropp (croooour(croo())))))
pat, sum(resize(crop((T)))
m.j
min(pa prf)
C(Met, TT) = a
+
max(paç) (po(aapppa))))
sm (g log(mm (.))
sgn(m B.h
B.
This dual approach allows for a comprehensive comparison that incorporates both the geometric configuration and the scale of the object representations. The best- matched template can be found by arg argmin L(M, ).).
15

---

Intra-Image Relations
Intra-Image Angular Discrepancies VQA
Inter-Image Spatial Dynamics VQA
Figure 6: Example input images all forms spatial VQA.
Object Scale Calibration After integrating all 3D object models {Ox) into the 3D scene, each with pose {P& [Rg tx]) and an initial scale {St Jk=1 set by the single-view 3D reconstruction model, we refine their scales to accurately reflect depth variations (e.g. moving an apple from close to the camera to distant corner reduces its apparent size). Through the lens of the pinhole camera with pose Pe. we render binary masks for each object and evaluate the length of their contour lines relative to their occlusion-free masks. The adjusted, final scale of object can be expressed as:
arcLength (ucLennghhfindConnouur (iindContour(A))) M arcLength (onngoonffnn (Mrend)) where Mrend is the rendered mask of object i.
squdl = Sinit
(5)
C Additional Experiments and Details
C.1 Spatial Visual Question Answering
Intra-Image Angular Discrepancies VQA (IaAD-VQA) For annotation, since YCBInEOAT [64]
offers ground truth object 6D poses, we first determine the table/ground plane using the principal axes of objects resting on (if present). Then, calculate the angles between the principal axes of different objects to annotate a list of qualitative and quantitative QA pairs. We provide a subset of the question template below.
Qualitative questions:
[A] tilted?
[A] tilted the left?
[A] inclined the back?
Is [A] more tilted than [B]?
[A] more tilted than [B] to the back?
Is [A] more inclined than [B] the right?
[A] leaning tovards [B] vartically?
In [A] straighter than [B]?
Along which axis
B) is [A] tilted?
Which (sject(s) are not upright?
nany (e)
upright?
Quantitative questions:
What inclination angle along vertical axia?
How many degrees in [A] tilted horizentally?
Calculate angle
16

---

Neasure the tilt of [A] relative the front.
What the relative angle between [A] and [B]?
Measure the angle between [A] and [R]
What in the angle betseen the horizontal axia of [A] and [B]?
How much Neasure the inclination difference slong the vertical axis for [A] and (B).
inclined along dapth axis compared [B]?
Neaeure angular deviation Deternine the angular difference betseen the depth azee of [A] and (6].
slong vertical
Deternine difference Compare the angles of tilt for [A] and [B] along the vertical axis.
along
Inter-Image Spatial Dynamics VQA (IrSD-VQA) For annotation, manually measure the changes in objects' locations and angles between two photos. To measure the change in camera shot angle, we record the change in the angle of the tripod to which the iPhone 12 Pro Max is attached. provide subset of the question template below.
Qualitative questions:
[A] nove?
Doea [A] rotate?
(A] zotate clockwise?
Does (A] incline?
(A]
right?
Does [A] nove closer to [B]?
[A]
more upright?
Does [A] incline more to the back?
Does the angle betseen [A] and becone snaller?
Along which direction does [A] nove?
Along which axis (M, D, H) doea rotate?
Which (bject(() move?
How many object(e) rotate?
How nany (b) becone tilted to the back?
Does the camera shot angle change?
which axis H) does shot angle change?
Quantitative questions:
How far does [A] vertically?
How far does [A] move horizontally?
How far does (A] tovards the back?
How nany degrees does [A] rotate clockuise?
How nany degrees does
comterciockvisee
What in the total distance [A] moves from ita original position?
Calculate the angle inclination [A] the sacond inaga.
Neasure the tilt of [A] relative to the first inage.
What change height
inage?
How much doee the distance betseen [A] and [B] change?
(A] incline
compared
inage?
What is the angular displacement of (A] tovards the right?
Neasure
angle [A] about
What in the neu distance batusen [A] and [B] in the second image?
angle [A] Deternine the change in angle of [A] relative to the ground plans.
tuo images.
movement
respect [B]?
Neseure the angular deviation [A] and [B] slong the vertical axis.
By hov nany degrees has the camera shot angle changed?
Along which axis/azes has the canera shot
changed?
How does (A] appear to nove if we do not account for the camera shot angle change?
What the perceived change in orientation of [A] due the angle change?
17

---

Al Unddrssandding 2D Scate
Understandngg 30 Soane
G) Harning & Execoling Tasks
S
Consuct voochet
K26,
12
Segmennrtin
Reconstuutionn
Alipiaal Reconeractio
& Shaostayy
Spate
Peckrranding Dejece Conrssiing
Irpet BG0 mage
Task Freporal
Uecermaediig mg
MulAnn Plannn
Figure 7: Pipeline for discovering and planning for robotics tasks from single image. It incorporate the task proposal and motion planning modules based on SpatialPIN.
How does the position of [A] change rolative to the camara angle diffarenca?
What the actual distanca of [A] when accounting for the shot angle change?
Dataset Examples We illustrate example input images of all forms of spatial VQA in Fig. 6
C.2 Discovering and Planning for Robotics Tasks from a Single Image
Given a single RGB image of scene with unknown environments and objects, the VLM identifies potential tasks and plans their execution using full 3D trajectories, complete with visualization. Figure 7 shows the pipeline, which incorporates task proposal and axes-constrained motion planning modules into SpatialPIN's pipeline from our main paper (Fig. 2).
We query VLM to propose meaningful, diverse tasks, each with a one-sentence task description. Instead of directly querying VLM for task proposal, we employ hybrid approach that integrates role-play and object-based initialization. In the role-play scenario, we prompt VLM to envision itself as robotic/human hand working in the scene perform household tasks. For the object-based initialization, we guide VI.M to sequentially focus on each identified object within the scene. When the scene contains more than one identified objects, VLM is instructed to suggest two tasks emphasizing interactions between the manipulating object and any of the detected objects, and an additional task focused solely on the manipulating object. If only one object is detected, VLM is directed to propose a task involving just that object. This strategy guarantees a broad spectrum of task suggestions, ensuring comprehensive object engagement.
Task Proposal
To further tailor the task proposals, we impose specific constraints, directing VLM to consider the practical affordances of objects while encouraging creative assumptions (e.g., a bowl's capacity to hold water) and potential interactions (e.g.. transferring water from a cup into a bowl). Additionally, we delineate clear boundaries by excluding tasks that entail the disassembly of objects, functionality tests, or the involvement of imaginary objects, thereby focusing on feasible and meaningful tasks.
As concrete example, given the image on the left of Fig. 2 with the manipulating object to be the red can, VLM will propose the following tasks:
"Task nane: Can Boul Transfer
Description: Pick up the can and pour its contents into the howl.
Task Can Ralocation
Description: Pick up the can and place it inside the boul.
Taek nane: Rotation
Description: Rotate the can 90 degrees on its vertical axis.
Cies-Constrained Motion Planning through Waypoint We introduce a novel method to guide VIM to conduct motion planning within a 3D scene based on proposed task by planning motion waypoints along the manipulating object's principal axes. More specifically, we define four types of manipulations that VLM can use:
18

---

Bed
Regincom
NOC3
Lounge
Pontry
Coffee
Station
Clasaroon
Kitchen
diversity cetegories quantities per
FORR BR
High variety perspectives
Figure 8: Dataset statistics, Our dataset presents 51 scenes- 13 from NOCS and 38 captured from varied perspectives featuring wide range of object categories, quantities, and diverse of tasks and planned trajectories.
Rotation type 1: Axial rotation. The object rotates around its principle axes.
Rotation rype 2: Rotation relative the target object.
pitch: Tilt similar to pouring water, around horizontal axis formed by the cross product of the connecting directional vector and the target's vertical axis.
yaw: Horizontal rotation, like a camera panning, around a vertical axis formed by the cross product of the connecting directional vector and the pitch axis.
roll: Rotation like drilling surface, around the connecting directional vector.
Translation type 1: Defines the goal relative to the target object's principle axes, with translation values for its [2, y, axes in centimeters. [0, 0, 0] cm indicates the goal the center of the target object.
Translation type 2: Sets the goal relative to directional vector between two reference objects, specifying how far (in cm) object should move towards or away from object 2 along this vector.
Since VLM inherently lacks the capability provide 3D coordinates and low-level actions di- rectly [9) our method offers practical workaround by translating natural language instructions into precise motion waypoints. This approach significantly enhances VLM's utility in spatial rea- soning and manipulation tasks without requiring direct 3D coordinate generation capabilities. Also, four types of manipulations we defined are both simple and comprehensive, covering a broad spectrum of manipulation tasks.
As concrete example, given the image on the left Fig. with 'Task name: Transfer', VLM will plan as follows:
Can to Bowl
"Task Nane:
Boul Tranafer
Manipulating obj idx:
Interacting ebj idx:
1. Hove Manipulating übj [3] to [6, o, 7] en relative to Target 0bj [4] ts local Dr, 2] axes.
rotate_Nref:
Rotate Minipulating Obj relative Target 0bj [4] around [pitch] azie by (75)
degrees."
In practice, the quality of motion planning by VLMs can be enhanced using various prompting techniques. One such technique chain-of-thougll (CoT) [3], where another LLM guides the VLM to plan each axes-constrained sparse waypoint step by step.
Dataset Statisties See Fig. for statistics and visuals. We standardize all image dimensions by resizing all to 640 x 480.
More Qualitative Demonstration We present more qualitative examples spanning various scenes and tasks, shown in Fig.
19

---

"Hove the zed eraser to the left of the blue eraser and erase lettars..
"Lift the milk container and deposit it into the trash bin."
"The cup is repositionad under the coffee dispenser prepare for coffee pouring.
"Cover the pot with the pot lid. .M
"Insert
the and cleaning.
Input
Planned Task Trajectories
Figure 9: More qualitative examples. With diverse input scenes and proposed tasks, our framework produces 3D trajectories with geometric awareness that aligns with the task descriptions. Zoom in for better view.
Table 10: Comparison of task diversity. We sample 106 proposed tasks for fair comparison with RoboGen and
previous RI. henchmarks.
Ours
RoboGen
Behavior-100 [54
RLbench 28
MetaWorid
Maniskill2 23
Number of Tasks
106
106
100
106
50
20
ScIf-BLEU 4.
0.269
0.284
0.299
0.317
0.322
0.674
Embedding Similarity
0.154
0.165
0,210
0,200
0.263
0.194
Experiment on Human Understanding
We aim understand human perception of our generated task executions by asking participants to provide one-sentence descrip- tion. We then evaluate the alignment between these descriptions and the ground truth task descriptions proposed by VLM, using OpenCLIP 15 43). Table. reveals high degree of aligmment. Intriguingly, it appears humans understand our task executions more accurately than machines do (Table. [7]. We hypothesize this dis- crepancy stems from the limitations of current video understanding models, whereas humans draw on their prior experiences for a deeper comprehension.
Table 9: Results for machine un- derstanding (generation) on 278 task exccutions and human under- standing, where users write de- scriptions for 10 tasks.
Machine
Human
OpenCIIP
0.636
0.823
Experiment on Task Diversity We evaluate the diversity of the proposed tasks in terms of semantic
meaning using Self-BLEU and the embedding similarity [75 48 following RoboGen [61] where lower scores mean higher diversity. We also compare with previous reinforcement learning (RL) benchmarks. From Table. TO ours method generates most diverse tasks as it is open all scenes with constraint.
20

---

C.3 Trajectory Generation Implementation Details
Trajectory Generation With the waypoints planned by VLM, we generate the manipulating object's trajectory using path planning algorithm, specifically rapidly-exploring random tree star (RRT*) [31] To generate accurate collision-free path, we perform K-means clustering on the point clouds of object 3D model with high number of clusters, segmenting the object mesh into discrete voxels and treating each voxel as an obstacle. Then, to accurately consider the manipulating object's 's dimensions, we grow the size of each voxel by its dimensions.
Handling VLM Planning Discrepancies The waypoints generated by VLM are typically accurate and practical. Nonetheless, there are instances where the waypoints suggested by VLM lead to collisions determined by the RRT planner. This discrepancy is less about VLM's misunderstandinn of the objects' sizes and their spatial relationship and more about the precision level of the waypoints, which may not match the exacting standards of the RRT* planner's outcomes. To resolve this, implement Gaussian sampling around the initially planned waypoints whenever a collision is detected. The sampling strategy is guided by predefined set of geometric rules. In our 3D coordinate system, positive x-axis [1, 0, 0] points right, positive y-axis [0, 1, 0] is away from viewer, positive z-axis [0, O, is up. For translation type 1, denote the goal pose relative the target object's principle axes as [dx, dy, dz]. For translation type 2, we denote the distance that object moves towards object 2 as follows:
dD. The of geometric rules
if type 1 & [dx, dy, dz] [0, 0, 0]: sample along [x. y, Z] axes elif type dx = & dy = =0 & dz 0: sample along [z] axis, Zsampled dz elif type & dz 0: sample along [x, y] axes
elif type & dz 1= 0: sample along [x, z] axes, Zsampled dz >
type 2: sample along the connecting directional vector, sampled dD 1
Trajectory Smoothing Finally, to ensure our trajectory is natural and smooth, we linearly interpolate rotation and interpolate translation using cubic spline.
D Prompt Details
We show exact prompts for VLMs for proposed application: discovering and planning for robotics tasks from single image.
Understanding Objects by Constraining Prompt.
Input:
BGB inage (640, 480) (vidth, height) with multiple objects.
Your task to identify and objecta by preciae color, texture, and 20 spatial locationa (in worda).
Do not vague phrase like multi-colored.
Please vrite in the following format. Do output anything else:
Object idx (actual integer, start from 0): x of color y, texture z at location v.
Task Proposal Prompt.
Input:
BGB inage (640, 480) (vidth, height) with multiple objects.
Detected objects with index.
You are single robot hand vorking in this inage scene to perforn simple household tasks. Tacks must be discovered from the inage. Consider objects' affordances and feel free make assumptions (e.g., bovl can contain vater) and interactions with other objecta (e.g., pouring vater from a cup into a bowl).
Task typas:
Interaction menipulating object the detected objects (involve translation,
21

---

translation rotation).
Rotate manipuaating object (involve rotation).
Strictly follaz constraints:
Exclude tasks involving disansenbly objects.
2. Exclude taske involving cleaning or functionality testing.
Exclude involving inaginary objects.
4. Manipulating object noves; interacting object static.
Assume objects rigid, transform).. This applies even to objects that are typically articulated (e.g., laptop).
joints
parta (i.e., cannot disassenble,
Propose tasks (2 interaction, rotation) for maniplatting Objact 5. Weite in the folloving fornat.
not output anything
Task Name: XXx
Hanipolating obj
Interacting idx: obj_idx (actual integer, manipulating 1dx)
Description: basic descriptions.
Coarse 3D Understanding Prompt.
Inputs:
RGB image (640, 480) (vidth, height) with multiple objects
Detected objecta with index.
Inage scene size.
Maximum and minimun vidth, depth, and height.
Your taek is to identify the camers shot angle (horizental, top-down, botton-up). Reaeon with respect to the vieual the
size, and naximune and minimune along each dimension. Choose horizontal
if not saverely angled.
Please vrite in the following format. Be concise. Do not output anything else:
Visual reasoning: ... Spatial data reasoning:
Conclusion: horrsantal/to--oom/bbottu-uu
Image and Spatial Context Understanding Prompt.
Inputs:
1. RGB image (640, 480) (vidth, height) with mitipple objecta and their visualized local axes (x rad, grean, blue).
Dotected objects with index.
For detected object, center, xyz-azes, size, spstial relationehip reletive other objecte.
The 3D coordinate systen of the inage is in centineters and follovs Blender. Positire x-axis [1, 0, right, positive y-axis 0] asay viever, positive [0, 0, up. clatter-loccvise around all axes.
Your task 1a to learn the apatial context. Do not output.
Motion Planning Prompt.
Inputs:
RCB inage (640, (vidth, height) multiple objects and visualized local red,
green, blue).
Detacted objects
3. Simple household tasks and descriptions to be perforned by single robot hand.
22

---

Your plan fine-graiee fine-grained notions for the nanipulating object complete the tacks four nenipolatione, explained as follovs:
Rotation:
rotate_calf: Axtal rotation. object rotates around local [x/y/x] axis [degrees].
rotate_Ntef: Botation relative to the larget object:
pitch: Tilt sinilar pouring water,
horizontal
by product
counecting directional vector and the target's 2-axis.
vertical formed by crose product the connecting directional vector and the pitch axis.
yas:
rotation,
camera panning,
drill entering The dagrees can be specified in two [dagreea].
roll:
connecting
manipulating object tosarda target object.
Fixed_towaree saredd_toarrdsttxedd.eea "fixed_torards orients the object towards the target, nimicking actions pouring (pitch), , facing (yau), drilling (yas-roll) target. Mixod back' reverses aligmnent.
Translation:
jaasslatetaa__bj- Defines the goal relative to the target object's local axes, with translation values for its [local_x, local_y, local_z] centineters. [o, 0] indicates the is the center of the target object.
translata_dtrec_xxle Seta the goal relative directional specifying how far (in an) abject should nove towards or away fron abject 2 along this vector (ponitive closer, negative asay). Objact indices differ, and if referance object the manipulating object, its current location is used.
between referance objects,
Strictly follow caveate:
Apply rotate_aret thoughtfully sequentially around different needed.
2. Use the provided spatial information and image effectively for understanding and planning vithin the
3. Combine comnon physical understanding with the scene's spatial details (like relative positions and objacts) strategic planning.
4. Remember that objects' local axes' pasitive diractions might require using negative values in rotation
motion planning.
Plan below. Fill in obj_idx based on the tasks.
potate_self: Botate Manipulating Object [obj_idx] around Ita local axis [x/y/2] [degrees]. rotate_wref: Rotate Manipulating Object [ob]_idx] relative Target Object [target_obj_1dd around [pitch/yss/rell] axis by aiggres//ixed__toaarrttttte
translate_
baresllae_tar_bbpp tar__bj:: Hove Manipulating Object [obj_idx] to [s, b, c] en relative to Target Object
[target_obj..c [ob_et)=bjj_dx]++ ts local [x, 2]
translate, branslata_direc.nre dirrs_dddre___ai Nove Manipolating Object [obj_idx] [a] en along the directional vector from Reference Object [rof_obj_1_idx] Reforence Object [raf_obj___idx]..
output anything Description: Rotate toy degrees vertical axis.
exenples.
following
Teak Category: Bear rotation
Notion Planning:
Manipulating obj idx: bear_idx (actual integer)
Interacting obj idx: vex (actual integer)
rotate_eelf;
Manipulating Object [bear_idx] around
(90) degreee.
Cup
Description: Pick the and its contents into the boul.
Notion Planning:
Manipulating obj idx: cup_idx (actual integer)
Interacting ebj idx: bool_idx
integer) tranalate_tae taroe__tr__ojj-- Move Manipulating Object [cup_idx] to [6, -7, 6] on relative to Target Object
[bowl_idx] 's local [x, y, axes.
23

---

rotate_vref: Rotate Manipulating Object [obj_idx] relative Target Object [boxl_obj_idx] around [pitch] axis by [Fixed_towards)..
Task Nama: Screadriver panatration
Description: Use scraudriver panetrate avocado.
Notion Planning:
Manipulsting obj idx: screv_idx (actual integer)
Interacting ebj idx: avocado_idx (actual integer)
tramelate_tar. bar_aattttar_aj:
Manipulating Object [screx_idx] -5,
Target Object
[avocado_idee 's local [x, =] azes.
Hamipulating Object (screx_idx] Target Object (avocado, _idx] around [yau] axia by [rixad_tovarda].
rotate_wref:
rotate_vraf: Manipulating Objact [screx_idx] Target: Object [avocado_ [rol]] axia by [380] degrees.
24

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification:
We conducted extensive empirical experiments to demonstrate it.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed limitations the Discussion and Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
We did not have theoretical assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have thoroughly discussed implementations details in the main paper and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
We have created our project page and Github repo. These will be updated
soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //nips.cc/pil wwcppbblc/guuiicuccccccc
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We discussed these in the Experiments and Appendix with lot of details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
In our experiments, large time cost is needed to do this.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discussed in the Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///waauu...ccppppaaattaaaaa
Answer: [Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work does not have broader social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No such concern.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not have such.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not have such.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "zAtsSNppwY", "reviewer": "Reviewer_gF9G", "summary": "This paper presents a pipeline designed to equip 2D Vision Language Models  with the capability to understand 3D spatial relationships. The key benefits of this framework are its zero-shot, training-free nature. The effectiveness of this approach has been validated through experiments on spatial Visual Question Answering (VQA) and various robotics tasks.", "questions": "See Weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Prompting VLM model for spatial awareness is an important field. The author provides a feasible solution.", "weaknesses": "1. The motivation behind this paper appears ambiguous. The authors assert that \"high-level 3D-aware tasks are underexplored,\" yet there is notable prior work such as \"Spatial VLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities\" and a series of studies involving GPT-4-V for robotics that address high-level 3D-aware tasks. It may be beneficial for the authors to refine the principal motivation behind SpatialPIN to distinguish it more clearly from existing research.\n2. The distinction between SpatialPIN and other forms of spatial prompting is not clearly articulated in the introduction. This section could benefit from a more detailed comparison to enhance clarity and strengthen the justification for SpatialPIN's unique contributions.\n3. The claim that operating without fine-tuning represents a contribution seems questionable, as existing works involving LLM agents and LLM prompting also employ techniques that do not require tuning. Clarification on why this aspect is particularly innovative or advantageous in the context of SpatialPIN would be helpful.\n4. There is room for improvement in the writing, including the presentation in Figure 1, the text , and the sections detailing the main contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Ambiguity in motivation. The authors assert that \"high-level 3D-aware tasks are underexplored,\" yet there is notable prior work such as SpatialVLM and a series of studies involving GPT-4V for robotics that address high-level 3D-aware tasks. \n\n**A1**: We thank the reviewer for highlighting this issue and hope the following response addresses the ambiguity.\n\nTo clarify, we define high-level 3D-aware tasks as tasks that consider the semantics of the objects to provide more accurate predictions in the spatial relationships (e.g., measurements between two objects) and potential plans for executing robotic tasks (e.g., pick and stack, task trajectory planning).\n\nSpatialVLM [1] is a powerful and recent model that enhances the spatial capabilities of VLMs by finetuning them on spatial VQA datasets. However, since most of the spatial VQA questions are on surface-level distance relationships (how far apart are object A and B), we argue that the semantics of objects are not well considered in the VLM compared to explicit 3D information, which SpatialPIN leverages. This is supported by several of our experiments, including Fig. 4 in our paper which shows that SpatialPIN provides much more accurate answers on more diverse and challenging forms of VQAs (Tables. 1-4) as it explicitly considers the semantic/geometry of the objects (knowing how big/tilted a laptop and a mug is by reconstruction first), as well as Table. 5 where explicit 3D information enables VLMs to act as a strong prior for tasks like pick and stack which SpatialVLM was less successful at. We will refine these descriptions and add a detailed motivation in our Introduction.\n\nMeanwhile, we agree with the reviewer that addressing several robotics 3D-aware tasks utilizing GPT-4V is beneficial to our work and we list their differences as below:\n\n1. VLM as Success Detectors [2]. This work uses VLMs to output yes/no answers to determine if the robotics pick-and-place task is completed based on RGB images but cannot perform other downstream tasks.\n\n2. VLMs are Zero-Shot Reward Models for RL [3]. This work uses VLMs' trained encoder as a CLIP model to output rewards given observations in simulation/gaming environments on tasks that do not require 3D spatial understanding.\n\n3. RoboGen [4]. This work uses VLMs to set up scenes in the simulator using its common sense. For example, VLMs output that the oven should be put on the table. These understandings do not require detailed spatial relationships nor task planning.\n\n4. VoxPoser [5]. The VLM is not used for spatial reasoning but used to detect object affordances to generate 3D value maps, which serve as objective functions/dense rewards for motion planners. However, the generated 3D value maps are coarse as a smoothing operation is applied to them, making it challenging for the motion planner to handle scenes/tasks involving a large number of objects or spatial tasks that require high precision regarding objects' shapes and poses (e.g., peg insertion). We provide an additional experiment on peg insertion in **attached pdf Table. T5** and demo in **Fig. F3**, comparing our performance with VoxPoser. We set up the peg insertion problem in 30 scenes using the ManiSkill simulator applying real-world physics properties. More details can be found in our reply to Reviewer C3yx.\n\n---\n\n**Q2**: The distinction between SpatialPIN and other forms of spatial prompting.\n\n**A2**: \nThank you for pointing this out! There are various prompting methods to improve the output quality of LLMs on general VQA, such as Chain-of-Thought (CoT), Tree-of-Thought (ToT), etc. However, enhancing VLM's 3D spatial reasoning capabilities through prompting without training is still an underexplored area. To the best of our knowledge, we are the first to investigate how to inject explicit 3D scene understanding into VLMs through progressive prompting and interactions between VLMs and 2D/3D foundation models. As supported by Reviewer K1EW, our meticulously designed progressive prompting and interactions between VLMs and 2D/3D foundation models is a novel approach.\n\nThere are some existing works that also utilize the idea of prompting 3D information into VLMs, such as Chat-3D [6], ShapeLLM [7], and Agent3D-zero [8]. However, they all require 3D scans as inputs and have no way to interpret the semantic meanings of single images. SpatialVLM is the closest in terms of the inputs and tasks that we can perform but is empirically shown by our results to perform less well on a variety of tasks.\n\nFinally, to further provide a more comprehensive study against other formats of spatial prompting, we also amend SpatialPIN to take in 2D screenshots of 3D scans directly and compare it on ScanQA dataset with Chat3D and Chat3D-v2 [8], as shown in **attached PDF Table. T1**. Our results indicate superior performance even with the visual imperfections of 2D screenshots of 3D scans.\n\n---\n\n**Q3**: The claim that operating without finetuning represents a contribution seems questionable.\n\n**A3**: While many LLM prompting techniques are training-free, SpatialPIN's plug-and-play module, as supported by Reviewer C3yx, has the advantage that future improvements on individual foundation models (e.g., 3D reconstruction) can directly replace certain components in this framework for better performance. As there currently exists no comprehensive 3D-VQA dataset that tackles all the tasks SpatialPIN can currently perform (spatial relationships, pick and stack, coming up with potential tasks and trajectories), we argue that a training-free approach is a contribution to the community in chaining 3D foundation models for a diverse set of tasks.\n\nOverall, given the strong results and being the first to combine explicit 3D predictions with VLM prompting for various downstream tasks, we strongly believe SpatialPIN's acceptance would be beneficial to the vision community.\n\n---\n**References:**\n\nWe put it in the Official Comment due to word limit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kyhm27Jyys", "reviewer": "Reviewer_K1EW", "summary": "This paper presents SpatialPIN, a framework designed to enhance the spatial reasoning capabilities of Vision-Language Models (VLMs) through prompting and interacting with 3D priors from multiple foundation models in a zero-shot, training-free manner. The authors argue that current state-of-the-art spatial reasoning-enhanced VLMs, which are trained on spatial visual question answering (VQA) datasets, may not generalize well to more complex 3D-aware tasks. SpatialPIN aims to address this by incorporating explicit 3D scene understanding through progressive prompting and interactions between VLMs and 2D/3D foundation models. The framework is evaluated on various spatial reasoning tasks, including different forms of spatial VQA and robotics applications like pick-and-stack and trajectory planning.", "questions": "1. Can the authors provide more details on the inference time for the full pipeline? How might this be optimized for real-time applications?\n\n2. How sensitive is the performance to the quality of the 3D reconstruction proposed in Section 3.3.? What happens when the reconstruction is imperfect or fails and how do the authors handle this issue?\n\n3. What do the authors mean by \"the VLM discovers potential tasks\"? Is it to allow the VLM to come up with feasible tasks given the objects in the scene and then to solve the self-proposed tasks? If so", "limitations": "The paper acknowledges the limitations related to inference speed and dependency on the quality of 3D foundation models. However, it would be constructive to discuss potential solutions or future directions to address these limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The approach of combining VLMs with 3D foundation models for spatial reasoning is novel and addresses the limitations of current methods that rely solely on training on spatial VQA datasets.\n2. The paper is generally well-written and organized.\n3. The work addresses an important gap in current VLMs' spatial reasoning capabilities, which has implications for various applications, particularly in robotics.", "weaknesses": "1. While the integration of 3D priors is novel, the combination of techniques used in the framework may not be entirely new, as it builds on existing 3D and VLM methodologies.\n2. The reliance on multiple 3D foundation models might introduce complexities and dependencies that are not fully addressed in terms of scalability and robustness.\n3. The use of proprietary VLMs as the core renders the proposed pipeline less insightful as the proprietary VLMs' capability of utilizing 3D information is a mystery and the performances are hard to analyze.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The combination of techniques used in SpatialPIN may not be entirely new, as it builds on existing 3D and VLM methodologies.\n\n**A1**: While using existing 3D and VLM methods, composing the framework of SpatialPIN isn't trivial and requires many thoughtful designs to first understand 2D objects, incorporate coarse 3D understanding, and inject fine-grained 3D info. Additional details such as background inpainting before depth estimation, and perspective canonicalization also help in determining many ambiguities during inference.\n\n---\n\n**Q2**: The reliance on multiple 3D foundation models may introduce complexities and dependencies that are not fully addressed in terms of scalability and robustness.\n\n**A2**: We agree that accumulated errors occur when each foundational model depends on previous output. In particular, we notice cases where imperfect 3D reconstruction leads to suboptimal results. Additional explanations and statistics are provided in response to Q5.\n\nMeanwhile, we would like to highlight that the current robustness of 6D pose and size estimation (detailed in Section 3.3 & Appendix B) is greatly helpful in the recovery of the 3D spatial space even with suboptimal object reconstruction (examples in **Fig. F2**). Based on all experiments in the paper, the accumulated errors on multiple datasets are significantly lower than our baselines such as SpatialVLM. We will discuss this further in the Limitations of our paper.\n\n---\n\n**Q3**: The use of proprietary VLMs as the core renders the proposed pipeline less insightful as the proprietary VLMs' capability of utilizing 3D information is a mystery and the performances are hard to analyze.\n\n**A3**: Thank you for the insight! We definitely agree that using proprietary VLMs may cause difficulties in analyzing some of the performances. We follow numerous prior notable works (FoundationPose [1], VoxPoser [2]) and use GPT-4 as it brings the best performances.\n\nTo better evaluate SpatialPIN, we provide additional experiments with the newly released LLaMa 3.1-70B as our backbone on the task of IaOR-VQA as shown in **Table. T2** & **T3**.\n\n---\n\n**Q4**: More details on the inference time for the full pipeline and optimization for real-time applications.\n\n**A4**: We provide detailed inference speeds for 30 input images with varying object categories and quantities (1-7) on the task of IaOR-VQA. For the 3D object reconstruction process, we vary the resolution of the input images (higher resolution generally leads to better quality but longer inference time). As detailed in Appendix C.2, we standardize all input images to 640 × 480 for an optimal trade-off between quality and inference speed. For VLMs, we compare API calls to GPT-4o (T3 personal account) and the open-source VLM, LLaVA-1.5.\n\nIn our implementation, we optimize the inference speed by running coarse and fine-grained 3D understanding in parallel. Additionally, we apply several implementation tricks to reduce the inference time of the 3D reconstruction process:\n1. We use One-2-3-45++, a two-stage pipeline where the first stage outputs 3D model shapes/normals and the second stage outputs colors/textures. For VQA and robotic planning tasks, we only need the output from the first stage, which reduces inference time by ~85% (completing two stages takes ~55s).\n2. We reconstruct each object in parallel using a combination of local inference and API calls, as One-2-3-45/One-2-3-45++ [3, 4] is commercialized.\n\nFrom **Table. T4**, we can see that the 3D reconstruction is the bottleneck. Yet, we highlight that:\n1. This process runs only once per image. If the user asks a second question for the same image, the inference time of our entire framework equals the VLM inference time.\n2. The speed is expected to improve with future versions of 3D foundation models. For example, LRM [5], a recent single-view 3D reconstruction model, reconstructs an object within 5s.\n\n---\n\n**Q5**: Sensitivity of the performance to the quality of 3D reconstruction and handling imperfect/failed reconstruction.\n\n**A5**: Single-view 3D reconstruction is a challenging ill-posed problem, especially given the complexity of our datasets with diverse scenes, object categories, quantities, and camera angles. To tackle this, we refine the 2D representation for each object by inpainting to handle occlusions (Section 3.1).\n\nIn most cases, our pipeline results in high-quality reconstruction. For visual comparisons of reconstructed scenes, please refer to PDF **Fig. F1** and  Appendix Fig. 9. However, imperfect reconstruction cases can still occur. Our statistics indicate that in a set of 100 images containing 1-7 objects, ~6 images have imperfectly reconstructed objects. However, our experiments demonstrate that even in these cases, the spatial reasoning-enhanced VLMs generally perform well in VQAs due to the robustness of our 6D pose and size estimation. Examples of such imperfect reconstruction cases are also provided in **Fig. F2**.\n\n---\n\n**Q6**: Meaning of \"VLM discovers potential tasks\".\n\n**A6**: Given a single RGB image of a scene, we query the VLM to propose meaningful, diverse tasks, each with a one-sentence description. Instead of directly querying the VLM for task proposals, we use a hybrid approach that integrates role-play (envision itself as a robotic/human hand) and object-based initialization (guide it to sequentially focus on each identified object). Please refer to Appendix C.2 for method details and Appendix D for prompt.\n\n---\n\nReference:\n\n[1] Foundationpose: Unified 6d pose estimation and tracking of novel objects. CVPR 2024\n\n[2] VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. CoRL 2023\n\n[3] One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. CVPR 2024\n\n[4] One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. NeurIPS 2023\n\n[5] LRM: Large Reconstruction Model for Single Image to 3D. ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PA4u5ir86B", "reviewer": "Reviewer_C3yx", "summary": "This paper introduces SpatialPIN to enhance the spatial reasoning capabilities of Visual Language Models (VLMs) by explicitly incorporating 3D priors from 3D foundation models. Extensive experiments demonstrate that SpatialPIN is effective for spatial Visual Question Answering (VQA) and robotic pick-and-place tasks.", "questions": "1.\tCan the proposed method be compared with affordance-based methods like Voxposer? What are the advantages and disadvantages?", "limitations": "Refer to weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe method is plug-and-play and can enhance the spatial reasoning capabilities of VLMs without requiring additional training.\n2.\tExperiments are comprehensive, validating the approach on both spatial VQA and robotic tasks.", "weaknesses": "1.\tThe writing in the methodology section needs improvement. Sections 3.1 and 3.2 mix method descriptions with prompts and corner cases, making it challenging for readers to understand.\n2.\tI'm uncertain about the extent to which fine-grained spatial relationships benefit robots. More experiments are needed, particularly beyond simple pick-and-stack tasks. Additionally, validation in real-world scenarios, combining camera matrices and depth estimation, would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Sections 3.1 and 3.2 mix method descriptions with prompts and corner cases, making it challenging for readers to understand.\n\n**A1**: \nThank you for pointing this out! Since our methodology involves VLMs interacting with 3D priors, the output of VLMs sometimes serves as input for 3D foundational models (2D Understanding), and sometimes VLMs use the output of 3D foundational models as input (Coarse and Fine-Grained 3D Understanding). Our original intention was to guide readers through our pipeline step by step, which is why the method descriptions interleave with the prompts.\n\nTo make our Methodology section easier to understand, we will use colored panels to detach the prompts and the method descriptions. We will also put some detailed but lengthy examples of prompting into the Appendix for better coherence. Besides, we consider labeling the section number and workflow index on our pipeline in Fig. 2 and changing the naming of \"Prompting: ...\" to the module name of our pipeline such as \"Understanding Objects by Constraining\" in the subsections in our Methodology. Finally, we will make sure we clearly explain the input/output workflow in our pipeline.\n\n---\n\n**Q2**: What is the extent to which fine-grained spatial relationships benefit robots? More experiments are needed, particularly beyond simple pick-and-stack tasks.\n\n**A2**: \nThank you for the suggestion. We definitely agree that more robotics experiments would make our framework more comprehensive. To address your concern about the extent to which fine-grained spatial relationships benefit robots, we add a new experiment of peg insertion. Peg insertion is a task that requires a finer level of spatial reasoning and higher precision than pick-and-stack. The challenges are threefold: \n1. VLM needs to reason about the size of the peg (how much to grasp).\n2. VLM needs to determine which principal axis and by how much the robotic arm should rotate the peg to align its orientation with the cube.\n3. This also evaluates the accuracy of the 3D centers of the peg, the cube, and the small hole, as well as the principal axes output by our framework.\n\nWe set up the peg insertion problem in the ManiSkill simulator, applying real-world physics properties. We randomize the size and location of the peg and the cube, and the size and position of the small hole on the cube. We creat 30 scenes.\n\nWe compare our method with current baselines (details in Section 4.2 of our paper) plus VoxPoser [1]. For the method using direct 3D information from our framework, we assume the robot grasper should grasp the smallest side of the peg. There is no principal axes alignment, and RRT* directly plans the trajectory from the peg to the small hole. For VoxPoser, we set up the peg insertion problem in RLBench, as their code supports this scenario. Using the same approach as above, we create 30 scenes. Results are shown in **Table. T5** and the demo in **Fig. F3**. From the results, our method has higher success rates than VoxPoser. For both the direct 3D information method and the SpatialVLM [2] method, the success rates are lower since the orientation of the peg is not aligned or adjusted properly. However, there are still success cases where the randomized orientation of the peg happens to be aligned with the cube at the beginning.\n\nAt the same time, we would like to highlight that we included 8 experiments, ranging from different forms of VQAs and robotics tasks to coming up with potential trajectories given an input image, without counting ablation studies. The number of experiments is on par with, if not more than, many existing works such as SpatialVLM [2], published in top venues. SpatialVLM has 2 experiments and 2 ablations for VQAs, and 1 experiment for robotics.\n\n---\n\n**Q3**: Can the proposed method be compared with affordance-based methods like VoxPoser? What are the advantages and disadvantages?\n\n**A3**: \nVoxPoser [2], similar to ours, can generate trajectories for manipulation tasks given unknown scenes in a zero-shot manner. In VoxPoser, the VLM is not used for spatial reasoning but is used to detect object affordances to generate 3D value maps, which serve as objective functions/dense rewards for motion planners. The objects of interest are granted affordance maps while other objects are granted avoidance maps. The use of affordance maps enables VoxPoser to generate more feasible grasps on some of the objects than ours (e.g., grasp the handle of a bag), but we consider this part can be enhanced by a simple extension to our pipeline (e.g., using VLM to detect affordances) in our 2D Scene Understanding stage. \n\nHowever, VoxPoser's generated 3D value maps are coarse as a smoothing operation is applied to them, making it challenging for the motion planner to handle scenes/tasks involving a large number of objects or spatial tasks that require high precision regarding objects' shapes and poses (e.g., peg insertion). Please see **Table. T5** for results and **Fig. F3** for demo. Instead, our method injects explicit 3D scene understanding information into VLMs through progressive prompting, enabling VLMs to plan for tasks that require a finer level of spatial understanding such as precise peg insertion. Therefore, as shown in **Table. T5**, our method outperforms VoxPoser by a fair margin in the peg insertion task.\n\n---\n\n**References:**\n\n[1] Huang, Wenlong, et al. \"VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models.\" CoRL, 2023.\n\n[2] Chen, Boyuan, et al. \"Spatialvlm: Endowing vision-language models with spatial reasoning capabilities.\" CVPR, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive comments. We appreciate the reviewers for recognizing the novelty and empirical evaluation of our work i.e., "the approach is novel, addresses an important gap in current VLMs' spatial reasoning capabilities" (K1EW), "experiments are comprehensive" (C3yx), and "validating on both spatial VQA and robotic tasks" (C3yx, gF9G).

Following reviewers' suggestions, we provide some additional experiments (results are given in the PDF file) to make the analysis more comprehensive, including:

- **Reviewer gF9G**: Qualitative spatial VQA on ScanQA dataset (similar to qualitative IaOR-VQA) compared to Chat-3D and Chat-3D-v2 (Table. T1).
- **Reviewer K1EW**: Qualitative and quantitative IaOR-VQA on new open source VLM Llama 3.1-70B (Table. T2 and T3).
- **Reviewer K1EW**: Detailed inference speed of our framework in seconds (Table. T4).
- **Reviewers gF9G and C3yx**: New robotics experiment on the task of peg insertion in comparison to SpatialVLM and VoxPoser (Table. T5).

We also provide answers to each concern in the following individual responses. We greatly appreciate the reviewers for their insightful and valuable comments. We believe all the individual questions have been addressed, but we are happy to address any further comments and questions from the reviewers.

Meta-review: The paper presents SpatialPIN, a framework that enhances the spatial reasoning of Visual Language Models (VLMs) by integrating 3D priors from foundation models without additional training. SpatialPIN improves performance in spatial Visual Question Answering (VQA) and robotics tasks like pick-and-stack and trajectory planning. The proposed method and results are reasonable, while there are remaining concerns on the novelty and scalability of the work. After carefully reading the paper, reviews, rebuttals, the AC recommends agreeing with the majority of the reviewers on accepting the paper.
Decision: Accept (poster)