SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors

Authors: chenyang  ma
kai  lu
ta-ying  cheng
niki  trigoni
andrew  markham
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 16:53:39

Abstract: Current state-of-the-art spatial reasoning-enhanced VLMs are trained to excel at spatial visual question answering (VQA). However, we believe that higher-level 3D-aware tasks, such as articulating dynamic scene changes and motion planning, require a fundamental and explicit 3D understanding beyond current spatial VQA datasets. In this work, we present SpatialPIN, a framework designed to enhance the spatial reasoning capabilities of VLMs through prompting and interacting with priors from multiple 3D foundation models in a zero-shot, training-free manner. Extensive experiments demonstrate that our spatial reasoning-imbued VLM performs well on various forms of spatial VQA and can extend to help in various downstream robotics tasks such as pick and stack and trajectory planning.

Main paper: 
 introduction: 
 Equipping vision-language models (VLMs) the capacities of spatial reasoning unlocks exciting applications, such as general-purpose reward annotation [52] , robotic data generation [61] , and grounding 3D object affordances [26, 38] . However, the spatial reasoning capabilities of VLMs on fine-grained spatial understanding tasks are somewhat limited. Current state-of-the-art (SOTA) spatial reasoning-enhanced VLM [12] is mostly tested on spatial visual question answering (VQA), such as determining objects' relative positions and orientations; experiments on higher-level tasks, such as scene comparisons and trajectory planning, which require more nuanced comprehension, are underexplored.Many works enhance the spatial reasoning capabilities of VLMs by training/fine-tuning them on standard spatial VQA datasets [12] . As a result, VLMs primarily learn surface-level associations between image-text-data triplets. Given the scarcity and difficulty of obtaining spatially rich embodied data or high-quality human annotations for 3D-aware queries, we hypothesize that these VLMs may not generalize to questions outside their dataset distribution or adapt to more challenging tasks that require an advanced level of spatial understanding.Recent studies [73, 7, 65, 69] in image space understanding show that VLMs, equipped with internetscale language knowledge, and multimodal foundation models capture complementary knowledge that can be combined to conduct new tasks spanning both modalities without additional training. Given the recent advancements in 3D foundation models [4, 41, 29] , this work explores whether there exists an alternative approach to enhance VLMs with higher-level spatial-awareness by incorporating 3D priors from these models.To this end, we propose SpatialPIN, a framework that utilizes progressive prompting and interactions between VLMs and 2D/3D foundation models as "free lunch" to enhance spatial reasoning capabilities in a zero-shot, training-free manner. By using these foundation models to decompose, comprehend, and reconstruct an explicit 3D representation, SpatialPIN grasps the core understanding of the 3D space presented by the 2D image. This allows generalizations to various 3D-aware tasks, from VQAs to 3D trajectory planning. We provide an extensive empirical study combining multiple off-the-shelf and handcrafted datasets, ranging from fundamental spatial questions regarding relative positions and orientations to providing fine-grained 3D information on objects' locations, sizes, inclinations, and dynamic changes, and plan for robotics tasks with full 3D trajectories. Results show that this straightforward approach significantly outperforms SOTA VLMs trained from extensive spatial VQAs (see SpatialPIN examples in Figure 1 ), consolidating our belief that a truly 3D-aware VLM can actually be imbued by simply injecting explicit, fundamental knowledge of the 3D scene. With the entire framework being fully modularized, each component can be easily replaced with the latest improvements within its specific domain.In summary, our main contributions are threefold: To prevent the models from overfitting to the standard problems from spatial VQA datasets [12] , we hope to derive a method that utilizes fundamental 3D foundation models to provide explicit scene understandings, then leverage the generalization capabilities of VLMs to tackle unforeseen tasks-all within a zero-shot, training-free manner.Scene perspectiveOur modular pipeline, SpatialPIN, enhances VLMs' spatial understanding of an image through progressive interactions with the scene decomposition, comprehension, and reconstruction processes with prompting. For image scene understanding (Sec. 3.1), we use VLM to describe objects by appearance and 2D location, complemented by language-guided segmentation and repainting models to obtain occlusion-free object masks. Elevating 2D understanding to coarse 3D (Sec. 3.2), we use metric depth estimation and perspective fields to estimate the 3D scene size and conduct perspective canonicalization with VLM. For fine-grained 3D understanding (Sec. 3.3), we partially reconstruct the 3D scene, with the full 3D representation of foreground objects and the background as a plane. With the reconstructed 3D scene, we summarize spatial information and prompt it to the VLM for various downstream tasks. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We discussed limitations in the Discussion and Conclusion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
2d image scene understanding: 
 Prompting: Objects Understanding by Constraining We start with querying VLM to identify and understand objects given I. We explicitly ask VLM to describe the objects by precise color, texture, and 2D spatial locations. This step is vital for two reasons: 1) enhance VLM's understanding of the objects, 2) differentiate between items of similar or identical categories and appearances.As a concrete example, given the left image of Fig. 2 , VLM outputs: " object 0: laptop of color rose gold, texture metallic at location left-center. object 1: camera of color black, texture smooth at location center-right. ..."The concise descriptions of identified objects are used as input text prompts for a language-guided segmentation model, enabling the acquisition of K segmentation masks {M occ k } K k=1 , with each mask corresponding to a unique object. However, an object i ∈ [1, K] may be occluded by other object(s), leading to an incomplete mask M occ i , which may be burdensome when we elevate the image to a 3D representation in the later stage. To resolve this, we create an inpainting mask, M inp i , for each object, in which all objects except the one itself are removed and replaced with white pixels. The inpainted masks are again fed to the language-guided segmentation model along with input text prompts such that occlusion-free object masks, {M of k } K k=1 , are obtained. This two-step segmentation process for object i is formulated as:EQUATIONwhere seg denotes the language-guided segmentation and τ i denotes the description of object i. In practice, to cleanly remove objects without residual fragments for inpainting, we apply dilation to and expand the white areas. Inpainted background I bg is acquired by removing and replacing all objects with dilated white pixels.Scene Size Estimation Using the estimated metric depth [29] and estimated camera intrinsic matrix by finding field of view (FOV) through perspective fields [29] , we backproject to determine the dimensions of the 3D spatial scene.Prompting: Perspective Canonicalization 3D information without any knowledge regarding the camera perspectives lead to ambiguities [12] . Consider a question "What is the orientation of the bowl relative to the laptop?" with the input scene in Fig. 2 , but taken from a top-down perspective.VLMs may output "downward to the left", but the correct answer should be "front-left" because humans perceive orientation from a horizontal angle. To address this, we provide the VLM with I, estimated scene size, and maximum and minimum dimensions, allowing it to reason about the camera shot angle (horizontal/top-down/bottom-up). Scene size information helps differentiate shot angles by providing clues about spatial layout and object proportions. For instance, if the depth variation is small, the VLM can infer a top-down or bottom-up angle along with visual cues.As a concrete example, given the left image of Fig. 2 , VLM outputs: We partially reconstruct the 3D scene with full representation of foreground objects while simplifying the inpainted background as a plane, as shown in Fig. 3 (a). We summarize spatial information from the reconstructed scene and prompt it to the VLM. Please see our Appendix for implementation details about reconstruction."Scene Initialization Given the occlusion-free object masks, {M of k } K k=1 , we use single-view 3D reconstruction model [41] to acquire object 3D models, {O k } K k=1 , with canonical poses determined during reconstruction. Pinhole camera is set at the origin, looking at positive depth-axis. With the estimated background plane size (Sec. 3.2), we move the background plane, O bg (visually identical to I bg ), along the depth-axis to fit precisely within the camera.Scene Reconstruction To resolve the imprecision of backprojection, our goal is to position object 3D models into the reconstructed 3D scene without visual discrepancies and ensure accurate depth. Instead of using naive backprojection, for an object i ∈ [1, K], we perform raycasting from object 3D center t c i on the camera plane to object 3D center t bg i on the background plane with metric depth d i . The 3D coordinate t i of object i is:EQUATIONThe rotation R i of the 6D pose of object i, P i = [R i | t i ] is explained previously. After integrating all 3D object models into the 3D scene, we refine each object's scale to accurately reflect depth variations by rendering binary masks and evaluate the length of their contour lines relative to their occlusion-free masks, through the lens of the pinhole camera, t c .We determine the principal axes (x-axis, y-axis, and z-axis) of each object using the minimal oriented bounding box (OBB), which is essential for unlock novel applications.Prompting: Objects and Spatial Context Understanding The reconstructed 3D scene from I with accurate object poses and scales is denoted as V 0 . As the final step of progressive prompting, we feed VLM the fine-grained 3D information derived from V 0 , grounding on the canonicalized perspective (Sec. 3.2). For example, with the input image in Fig. 2 and a horizontal camera shot angle, depth corresponds to the positive y-axis (similarly, in a top-down/bottom-up view, depth is Yes, the laptop is further away from the camera compared to the mug.The box is more tilted along the horizontal axis, as it is leaning on the table and appears to be at a slight angle.The box is positioned at a diagonal angle to the bottle on the table. the negative/positive z-axis) in a right-handed coordinate system. The width and height axes can be determined accordingly, aligning each axis's orientation with human perception.We feed VLM a paragraph describing the objects' poses, sizes, and principal axes in physical units, alongside their spatial relationships. To augment VLM's understanding, we also feed V 0 with visualized object axes (see Fig. 2B ). Visualizing 3D spatial information is pivotal in improving VLMs' understanding of 3D spatial contexts derived from 2D images, validated by 3DAxiesPrompts [37] . Yet, we want to emphasize that we do not feed hardcoded information, such as objects' relative distances and inclinations, to VLMs. Instead, we aim for the summarized 3D information to enhance VLMs' general spatial understanding.As a concrete example for the left image on Fig. 2: "Obj 1 spatial context: 3D center: By partially reconstructing the 3D scene with visual alignments, our framework enables VLMs to use tools like rapidly-exploring random tree star (RRT*) [31] to generate accurate, collision-free paths based on task specifications (more details in Appendix). This capability unlocks novel and interesting applications when combined with task-specific prompting techniques, shown in Experiments (Sec.4).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments: 
 We conduct experiments to answer the following questions: 1) Does our framework enhance the general spatial reasoning capabilities of VLMs, and how well does it perform? 2) What novel applications does our framework unlock for VLMs, and how well do we perform in these applications? 3) How effective is each module in our framework?Since we evaluate our approach on a wide range of tasks to test VLMs' higher-level spatial awareness, some tasks are novel and lack existing/open source datasets. Therefore, for all our experiments, we use a combination of 4 existing datasets and 2 hand-crafted datasets.Implementations The language-guided segmentation model is Language Segment-Anything [44] and the repainting model is LaMa [55] . We use One-2-3-45++ [41] for single-view 3D reconstruction, perspective fields [29] for camera intrinsic estimation, and ZoeDepth [4] for depth estimation. For partial 3D scene reconstruction, we use Blender [17] as the 3D software. All inference is run on 1 NVIDIA A10 GPU with 24GB RAM.We experiment on the basic form of spatial VQA introduced by SpatialVLM (IaOR-VQA), and two new forms introducted by us (IaAD-& IrSD-VQA). For IaOR-VQA, please check SpatialVLM [12] for details. For IaAD-& IrSD-VQA, please see our Appendix.Intra-Image Object Relations VQA (IaOR-VQA) As the basic form of spatial VQA, it involves spatial reasoning about object relative orientations and sizes. This is divided into qualitative (e.g., "is [A] in front of [B]", "is [A] smaller than [B]") and quantitative (e.g., "how far apart are [A] and [B]", "measure the width of [A]") questions.We follow the evaluation method of SpatialVLM [12] . Since SpatialVLM did not release their evaluation dataset, we reproduce one using RGBD images from NOCS [57] (object dataset), RT-1 [6] , and BridgeData V2 [56] (robotics manipulation datasets). We sample 13, 20, and 20 distinct scenes from each. We generate QA pairs using the SpatialVLM data generation pipeline [51] , followed by manual refinement. We check correctness for qualitative questions and calculate distances for quantitative questions. We annotate 300 qualitative and 200 quantitative spatial VQA pairs (SpatialVLM has 331 and 215 for each).Intra-Image Angular Discrepancies VQA (IaAD-VQA) We propose a new form of Spatial VQA that needs spatial reasoning about objects' inclinations. It includes qualitative (e.g., "is [A] tilted", "is [A] more tilted than [B]") and quantitative questions (e.g., "how many degrees is [A] tilted vertically", "measure the angle between [A] and [B]").Since this form of Spatial VQA involves out-of-plane rotations, YCBInEOAT [64] (object tracking dataset) is a suitable choice. We sample 30 scenes from it and annotate 50 questions each for qualitative and quantitative spatial VQA pairs.Inter-Image Spatial Dynamics VQA (IrSD-VQA) We further propose a more challenging form of Spatial VQA. Given two images with multiple objects, the objects in the second image may move, rotate, incline, or the image may have a change in camera angle. The VLM needs to reason about these changes. Example qualitative questions include "does [A] move, rotate, or incline", "does [A] incline along the y-axis" while quantitative questions include "how far does [A] move", "how many degrees does [A] rotate horizontally".As it is difficult to find a dataset that meets these requirements, we craft our own. We capture 20 image pairs using an iPhone 12 Pro Max, with each image containing 1 -5 objects, and annotate 50 questions each for qualitative and quantitative spatial VQA pairs.The results in Tables 1 and 2 on qualitative and quantitative IaOR-VQA demonstrate that providing various VLMs fine-grained 3D information enhances their spatial reasoning capacities by a large margin. Surprisingly, VLMs with math and geometry reasoning capacities (e.g., GPT-4V, GPT-4o) show substantial improvements with this information.Table 1 : Qualitative IaOR-VQA. We exclude comparisons to PaLI [14] , PaLM-E [20] , and PaLM 2-E [3] as they are not open source, and include experiments with GPT-4o [1] in addition to GPT-4V [47] , LLaVA-1.5 [40] , and InstructBLIP [18] . We use the HF version of SpatialVLM [51] .GPT-4o LLaVA-1. The results in Tables 3 and 4 demonstrate the effectiveness of our approach on both qualitative and quantitative IaOR-VQA and IrSD-VQA tasks. Notably, the performance on quantitative IaOR-VQA is suboptimal compared to quantitative IrSD-VQA, despite the latter being more challenging. We believe this is because, for quantitative IrSD-VQA, the VLM sometimes confuses the camera and world coordinate frames, comparing the object's principal axes with the world axes to reason about changes in angles.Fig. 4 presents qualitative examples on all forms of spatial VLM.Pick and stack is a classic robotics task. Given a robot's egocentric observation of a scene with multiple objects and a task description, our pipeline uses traditional planning to solve the problem. This task demands advanced spatial reasoning, as the model must comprehend 3D locations, sizes, and physical properties of the objects (i.e., how much to grasp and how high to drop? Is the object deformable or articulated so the robotic grasper needs to grasp more firmly?). For instance, grasping and stacking a soft toy bear on a cube is significantly different from stacking a solid apple on a mug.The model reasons about grasping and stacking policies, directly outputting 3D trajectories for the robot's end effector using traditional path planning algorithm as external tool.We set up the pick-and-stack problem in the ManiSkill [22] simulator, applying real-world physics properties. Rigid and articulated objects are chosen from the YCB dataset [10] and are randomly allocated on the table within the robotic arm's reach, with observations from different perspectives. We create 50 scenes. Since robot observations from simulated scenes suffer from sim2real gap and consider that most real-world robots have depth sensors, we use ground truth camera matrix and depth.We compare our method to the following baselines: 1) direct 3D information output from our framework without GPT-4o [1] reasoning about physics and object properties and 2) SpatialVLM with our RRT* trajectory generation module.Results Table 5 shows the results, with a qualitative example demonstrated in Fig. 5 . The results indicate that using precise 3D information from our framework significantly improves the success rate, and incorporating VLM reasoning further enhances performance.We present a novel task that requires advanced spatial reasoning capacities of VLMs. Given a single RGB image of any scene comprising unknown environments and objects, the VLM discovers potential tasks and plans their execution with full 3D trajectories, with the motivation that it can be used for robot learning in future research. To solve this complex task and visualize the execution using our framework, we introduce: 1) a task proposal approach using VLM, 2) a novel axes-constrained 3D planning approach that enables spatial reasoning-imbued VLM to plan the object motion based on the proposed tasks by specifying waypoints. Please see Appendix for the pipeline and details.Dataset We create a diverse evaluation dataset by combining self-captured photos (38) using an iPhone 12 Pro Max and scenes (13) from NOCS [57] . Our dataset covers diverse scenes (e.g., office, kitchen, bathroom), and features a rich diversity of object categories (116) and quantities (185), with each image containing 1 -7 objects and 1 -3 tasks proposed for each object (278 tasks/planned trajectories in total). The dataset's diversity is further enhanced by the variety of perspectives (e.g., frontal, top-down, side views). This deliberate choice of diverse angles, both in our own image capturing process and through the random extraction of frames from NOCS, aims to simulate a realistic and challenging array of scenes for evaluation. See Appendix for statistics and visuals.Qualitative Demonstration We present a qualitative example in Fig. 5 . Additional examples in Appendix shows our framework's capability to produce diverse and accurate task trajectories spanning various scenes and tasks. Human Evaluation: User Study We rely on human preference evaluation as one of our quantitative metrics. We ask 25 users to rate 5 translation and 5 rotation task executions in terms of task description alignment. For these complex context-dependent manipulation tasks, we instead ask users to judge 10 executions relative to human action, and to encapsulate their perception of the action in our with a single sentence. These sentence description will be used to test human understanding of our planned trajectories (please see Appendix). Note that our user study size is similar to those representative works such as ControlNet [74] and Prompt-to-Prompt [24] . Results in Table . 6.We assess the interpretability of our generated task executions from a machine's perspective using SOTA video understanding model, Video-LLaVA-7B [35] . We use two approaches: binary classification and descriptive generation. For classification, we feed the model with the task descriptions generated by VLM and ask question (is the video doing. . .?). In generation, we prompt Video-LLaVA-7B to articulate its interpretation of our task executions. To quantify the correspondence between the model's perception and the tasks, we use OpenCLIP cosine similarity score [15] . However, we find that even SOTA video understanding model shows limited performance. To assess false positive rate in classification, we deliberately misalign the sequence of generated task executions with their corresponding task descriptions, expecting a theoretical accuracy of 0%. Contrary to expectations, Video-LLaVA-7B reports a false positive rate of 36.3%. To adjust for this anomaly, we subtract this rate from the model's raw accuracy for correctly aligned video-task pairs. This method, while unconventional, provides a more fair and reasonable evaluation of machine video understanding, underscoring the current challenges faced by video understanding models in accurately interpreting complex video content. Results in Table . 7.We evaluate the effectiveness of each module in our framework on IaOR-VQA by 1) seeking alternative designs of the overall pipeline and 2) removing each component in our ablations.Overall Design To demonstrate our framework's generalization across a wide range of objects, We replace our 2D + 3D pipeline with: 1) SOTA mesh-free single image object pose and size estimation model, ShAPO [27] , 2) SOTA mesh-based single image object pose and size estimation model, SAM-6D [36] , and feeds it with the object 3D model reconstructed by One-2-3-45++ [41] , and 3) the data generation backbone of SpatialVLM [12] . Since models 1) and 2) do not provide language annotations for their outputs, we first summarize the numerical outputs using our approach in Sec. 3.3. Then, GPT-4V identifies QA pairs.Removing 2D Understanding Module In this case, the VLM no longer examines the objects through prompting, and only the object name is input into the language-guided segmentation model.This means there is no scene size estimation, and the VLM does not conduct perspective canonicalization. During 3D scene reconstruction, we assume the image plane width to be 1 meter.To validate the fine-grained 3D scene understanding module, we replace object mask raycasting with backprojection using the object's 2D center and remove the object scale calibration.To demonstrate the overall effectiveness of our 3D understanding modules, we simply backproject the input image with the estimated metric depth.Results Table 8 demonstrates the effectiveness of each module in our framework. The results also highlight the limitations of using off-the-shelf SOTA mesh-free and mesh-based single-image object pose and size estimation methods as our backbone. These methods are not language-driven and may struggle to generalize to novel objects in diverse input scenes. 
discussion and conclusion: 
 We present SpatialPIN, a framework designed to enhance the spatial reasoning capabilities of VLMs through prompting and interacting with 3D priors in a zero-shot, training-free manner. We see our work as a step towards equipping VLMs with more generalized spatial reasoning capacities, demonstrated through applications in various forms of spatial VQA and both traditional and novel robotics tasks.Limitations Readers may be curious about the inference speed of our framework. The bottleneck is the 3D object reconstruction process and the API call to closed-source VLMs (∼ 20 seconds per image). However, we want to highlight that this process runs only once per image, and the speed is expected to improve with future versions of 3D foundation models.We use One-2-3-45++ [41] , which provides the camera pose. For models without this information, we develop an efficient method to determine the pose by comparing object masks with rendered 3D model templates, inspired from matching-based 6D pose estimation works [36, 46, 9, 42] .We generate a set of object templates, denoted as {T i j } J j=1 , each rendered from the object's 3D model O i . These templates are created by positioning the camera at various locations on an icosphere surrounding the object in SE(3) space, which simulates a spherical coverage around the object to capture its geometry from all angles uniformly. For each template T i j , we compute a matching score against the occlusion-free object mask M of i . We propose a simple yet effective score matching method. We draw a bounding rectangle around the segmented object inside M of i and across all {T i j } J j=1 , and crop the bounding rectangle. We then calculate the shape similarity between the contour line of cropped M of i and that of each cropped T i j using Hu moments [25] . Additionally, we crop and resize the bounding rectangle to the same dimension, and evaluate the similarity based on the pixel area of the cropped and resized masks. Our score matching method can be formulated as:EQUATION)This dual approach allows for a comprehensive comparison that incorporates both the geometric configuration and the scale of the object representations. The best-matched template can be found by arg min J j=1 L(M of i , T i j ).  
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We discussed it in the Experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: We followed the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our work does not have broader social impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: No such concern.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Given a single RGB image of a scene with unknown environments and objects, the VLM identifies potential tasks and plans their execution using full 3D trajectories, complete with visualization. Figure 7 shows the pipeline, which incorporates task proposal and axes-constrained motion planning modules into SpatialPIN's pipeline from our main paper (Fig. 2 ).We query VLM to propose meaningful, diverse tasks, each with a one-sentence task description. Instead of directly querying VLM for task proposal, we employ a hybrid approach that integrates role-play and object-based initialization. In the role-play scenario, we prompt VLM to envision itself as a robotic/human hand working in the scene to perform household tasks. For the object-based initialization, we guide VLM to sequentially focus on each identified object within the scene. When the scene contains more than one identified objects, VLM is instructed to suggest two tasks emphasizing interactions between the manipulating object and any of the detected objects, and an additional task focused solely on the manipulating object. If only one object is detected, VLM is directed to propose a task involving just that object. This strategy guarantees a broad spectrum of task suggestions, ensuring comprehensive object engagement.To further tailor the task proposals, we impose specific constraints, directing VLM to consider the practical affordances of objects while encouraging creative assumptions (e.g., a bowl's capacity to hold water) and potential interactions (e.g., transferring water from a cup into a bowl). Additionally, we delineate clear boundaries by excluding tasks that entail the disassembly of objects, functionality tests, or the involvement of imaginary objects, thereby focusing on feasible and meaningful tasks.As a concrete example, given the image on the left of Fig. 2 , with the manipulating object to be the red can, VLM will propose the following tasks: Axes-Constrained Motion Planning through Waypoint We introduce a novel method to guide VLM to conduct motion planning within a 3D scene based on a proposed task by planning motion waypoints along the manipulating object's principal axes. More specifically, we define four types of manipulations that VLM can use: Rotation type 1: Axial rotation. The object rotates around its principle axes.Rotation type 2: Rotation relative to the target object.-pitch: Tilt similar to pouring water, around a horizontal axis formed by the cross product of the connecting directional vector and the target's vertical axis.-yaw: Horizontal rotation, like a camera panning, around a vertical axis formed by the cross product of the connecting directional vector and the pitch axis.-roll: Rotation like drilling a surface, around the connecting directional vector.Translation type 1: Defines the goal relative to the target object's principle axes, with translation values for its [x, y, z] axes in centimeters. [0, 0, 0] cm indicates the goal is the center of the target object.Translation type 2: Sets the goal relative to a directional vector between two reference objects, specifying how far (in cm) object 1 should move towards or away from object 2 along this vector.Since VLM inherently lacks the capability to provide 3D coordinates and low-level actions directly [60, 19] , our method offers a practical workaround by translating natural language instructions into precise motion waypoints. This approach significantly enhances VLM's utility in spatial reasoning and manipulation tasks without requiring direct 3D coordinate generation capabilities. Also, the four types of manipulations we defined are both simple and comprehensive, covering a broad spectrum of manipulation tasks.As a concrete example, given the image on the left of Fig. 7 , with "Task name: Can to Bowl Transfer", VLM will plan as follows:"Task Name: Can to Bowl Transfer Manipulating obj idx: 3 Interacting obj idx: 4 1. Move Manipulating Obj [3] to [6, 0, 7] cm relative to Target Obj [4] 's local [x, y, z] axes.2. rotate_wref: Rotate Manipulating Obj [3] relative to Target Obj [4] around [pitch] axis by [75] degrees."In practice, the quality of motion planning by VLMs can be enhanced using various prompting techniques. One such technique is chain-of-thought (CoT) [63] , where another LLM guides the VLM to plan each axes-constrained sparse waypoint step by step.Dataset Statistics See Fig. 8 for statistics and visuals. We standardize all image dimensions by resizing all to 640 × 480.More Qualitative Demonstration We present more qualitative examples spanning various scenes and tasks, as shown in Fig. 9 . Ours RoboGen [61] Behavior-100 [54] RLbench [28] MetaWorld [72] We aim to understand human perception of our generated task executions by asking participants to provide a one-sentence description. We then evaluate the alignment between these descriptions and the ground truth task descriptions proposed by VLM, using OpenCLIP [15, 43] . Table. 9 reveals a high degree of alignment.Intriguingly, it appears humans understand our task executions more accurately than machines do (Table . 7 ). We hypothesize this discrepancy stems from the limitations of current video understanding models, whereas humans draw on their prior experiences for a deeper comprehension.Experiment on Task Diversity We evaluate the diversity of the proposed tasks in terms of semantic meaning using Self-BLEU and the embedding similarity [75, 48, 49] following RoboGen [61] , where lower scores mean higher diversity. We also compare with previous reinforcement learning (RL) benchmarks. From Table . 10, ours method generates most diverse tasks as it is open to all scenes with no constraint.Your goal is to plan fine-grained motions for the manipulating object to complete the tasks using four manipulations, explained as follows:Rotation:rotate_self: Axial rotation. The object rotates around its local [x/y/z] axis by [degrees] .rotate_wref: Rotation relative to the target object:-pitch: Tilt similar to pouring water, around a horizontal axis formed by the cross product of the connecting directional vector and the target's z-axis.-yaw: Horizontal rotation, like a camera panning, around a vertical axis formed by the cross product of the connecting directional vector and the pitch axis.-roll: Rotation like a drill entering a surface, around the connecting directional vector.The degrees can be specified in two ways:-Exact [degrees]. Positive values rotate the manipulating object towards the target object.-Fixed_towards/fixed_back. 'fixed_towards' orients the object towards the target, mimicking actions like pouring (pitch), facing (yaw), or drilling into (yaw+roll) the target. 'fixed_back' reverses this alignment.Translation: translate_tar_obj: Defines the goal relative to the target object's local axes, with translation values for its [local_x, local_y, local_z] axes in centimeters. [0, 0, 0] cm indicates the goal is the center of the target object.translate_direc_axis: Sets the goal relative to a directional vector between two reference objects, specifying how far (in cm) object 1 should move towards or away from object 2 along this vector (positive closer, negative away). Object indices must differ, and if one reference object is the manipulating object, its current location is used.Strictly follow caveats:1. Apply rotate_wref thoughtfully and sequentially around different axes as needed.2. Use the provided spatial information and image effectively for understanding and planning within the 3D scene. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: The paper presents SpatialPIN, a framework that enhances the spatial reasoning of Visual Language Models (VLMs) by integrating 3D priors from foundation models without additional training. SpatialPIN improves performance in spatial Visual Question Answering (VQA) and robotics tasks like pick-and-stack and trajectory planning. The proposed method and results are reasonable, while there are remaining concerns on the novelty and scalability of the work. After carefully reading the paper, reviews, rebuttals, the AC recommends agreeing with the majority of the reviewers on accepting the paper.

Reviews: review: paper_summary: paper_summary: This paper presents a pipeline designed to equip 2D Vision Language Models  with the capability to understand 3D spatial relationships. The key benefits of this framework are its zero-shot, training-free nature. The effectiveness of this approach has been validated through experiments on spatial Visual Question Answering (VQA) and various robotics tasks.
strength_weakness: strength_weakness: 1. The motivation behind this paper appears ambiguous. The authors assert that "high-level 3D-aware tasks are underexplored," yet there is notable prior work such as "Spatial VLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities" and a series of studies involving GPT-4-V for robotics that address high-level 3D-aware tasks. It may be beneficial for the authors to refine the principal motivation behind SpatialPIN to distinguish it more clearly from existing research.
2. The distinction between SpatialPIN and other forms of spatial prompting is not clearly articulated in the introduction. This section could benefit from a more detailed comparison to enhance clarity and strengthen the justification for SpatialPIN's unique contributions.
3. The claim that operating without fine-tuning represents a contribution seems questionable, as existing works involving LLM agents and LLM prompting also employ techniques that do not require tuning. Clarification on why this aspect is particularly innovative or advantageous in the context of SpatialPIN would be helpful.
4. There is room for improvement in the writing, including the presentation in Figure 1, the text , and the sections detailing the main contributions.
questions: questions: See Weaknesses.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper presents SpatialPIN, a framework designed to enhance the spatial reasoning capabilities of Vision-Language Models (VLMs) through prompting and interacting with 3D priors from multiple foundation models in a zero-shot, training-free manner. The authors argue that current state-of-the-art spatial reasoning-enhanced VLMs, which are trained on spatial visual question answering (VQA) datasets, may not generalize well to more complex 3D-aware tasks. SpatialPIN aims to address this by incorporating explicit 3D scene understanding through progressive prompting and interactions between VLMs and 2D/3D foundation models. The framework is evaluated on various spatial reasoning tasks, including different forms of spatial VQA and robotics applications like pick-and-stack and trajectory planning.
strength_weakness: strength_weakness: 1. While the integration of 3D priors is novel, the combination of techniques used in the framework may not be entirely new, as it builds on existing 3D and VLM methodologies.
2. The reliance on multiple 3D foundation models might introduce complexities and dependencies that are not fully addressed in terms of scalability and robustness.
3. The use of proprietary VLMs as the core renders the proposed pipeline less insightful as the proprietary VLMs' capability of utilizing 3D information is a mystery and the performances are hard to analyze.
questions: questions: 1. Can the authors provide more details on the inference time for the full pipeline? How might this be optimized for real-time applications?

2. How sensitive is the performance to the quality of the 3D reconstruction proposed in Section 3.3.? What happens when the reconstruction is imperfect or fails and how do the authors handle this issue?

3. What do the authors mean by "the VLM discovers potential tasks"? Is it to allow the VLM to come up with feasible tasks given the objects in the scene and then to solve the self-proposed tasks? If so
limitations: limitations: The paper acknowledges the limitations related to inference speed and dependency on the quality of 3D foundation models. However, it would be constructive to discuss potential solutions or future directions to address these limitations.
paper_summary: paper_summary: This paper introduces SpatialPIN to enhance the spatial reasoning capabilities of Visual Language Models (VLMs) by explicitly incorporating 3D priors from 3D foundation models. Extensive experiments demonstrate that SpatialPIN is effective for spatial Visual Question Answering (VQA) and robotic pick-and-place tasks.
strength_weakness: strength_weakness: 1.	The writing in the methodology section needs improvement. Sections 3.1 and 3.2 mix method descriptions with prompts and corner cases, making it challenging for readers to understand.
2.	I'm uncertain about the extent to which fine-grained spatial relationships benefit robots. More experiments are needed, particularly beyond simple pick-and-stack tasks. Additionally, validation in real-world scenarios, combining camera matrices and depth estimation, would be beneficial.
questions: questions: 1.	Can the proposed method be compared with affordance-based methods like Voxposer? What are the advantages and disadvantages?
limitations: limitations: Refer to weaknesses.

Comments: title: Thank the authors for the detailed response.
Response to the authors' rebuttal
References
comment: Thank you so much for your responses and for raising our score. Your comments have been helpful in pushing the paper towards better quality.

Since it is near the end of the discussion period, we cannot contribute more meaningfully to good discussions. Still, we want to highlight our contributions:

1. We investigate how to address an important gap in current VLMs' spatial reasoning capabilities. We propose SpatialPIN, a modular plug-and-play framework that progressively enhances VLM's 3D reasoning capabilities by prompting and interacting with 3D foundational models, which is the first of its kind.

2. Our approach better considers object semantics by providing VLMs with explicit 3D information. We address the limitations of current methods that rely solely on training on spatial VQA datasets, which may not generalize well to more complex 3D-aware tasks.

3. We validate our method with extensive experiments, ranging from spatial VQAs to various robotic tasks. The number of experiments to measure SpatialPIN's robustness is on par with, and even exceeds, many previous works.
I have read the rebuttal and other reviews. Some of my concerns have been solved by the rebuttal, but I still feel that the technical contribution of this work is somewhat incremental and the novelty is limited. I have increased my score to "4: Borderline Reject" accordingly.
We are happy that our replies address your questions! We sincerely appreciate your kind response and positive feedback.
Thank you for maintaining a positive rating and we are happy to have addressed your concerns!
Dear Reviewer,
 

We greatly appreciate your constructive feedback and concerns! We have carefully answered your questions with experiments, and sincerely hope that these have addressed your concerns. Once again, we look forward to your response after going through our rebuttal and other reviews. If there are any further issues or feedback, we are happy and eager to address them promptly.
 

Thank you once again!
Thank you for the insightful response. My concerns have been addressed overall, and I intend to maintain the original rating.
Thank you for providing the detailed answers. My concerns have been mostly resolved. The rating of 6 is maintained.
**References:**

[1] Chen, Boyuan, et al. "Spatialvlm: Endowing vision-language models with spatial reasoning capabilities." CVPR, 2024.

[2] Du, Yuqing, et al. "Vision-Language Models as Success Detectors." CoLLAs, 2023.

[3] Rocamonde, Juan, et al. "Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning." ICLR, 2024.

[4] Wang, Yufei, et al. "RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation." ICML, 2024.

[5] Huang, Wenlong, et al. "VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models." CoRL, 2023.

[6] Wang, Zehan, et al. "Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes." arXiv, 2023.

[7] Qi, Zekun, et al. "Shapellm: Universal 3d object understanding for embodied interaction." arXiv, 2024.

[8] Zhang, Sha, et al. "Agent3D-Zero: An Agent for Zero-shot 3D Understanding." arXiv, 2024.