SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding
Authors: Zhao Jin
Rong-Cheng Tu
Jingyi Liao
Wenhao Sun
Xiao Luo
Shunyu Liu
Dacheng Tao
Venue: NeurIPS 2025 poster ()
Keywords: 3D Visual Grounding
Multi-modal Agent
Zero-shot
Primary area: applications

Abstract: 3D Visual Grounding (3DVG) aims to localize target objects within a 3D scene based on natural language queries. To alleviate the reliance on costly 3D training data, recent studies have explored zero-shot 3DVG by leveraging the extensive knowledge and powerful reasoning capabilities of pre-trained LLMs and VLMs. However, existing paradigms tend to emphasize either spatial (3D-based) or semantic (2D-based) understanding, limiting their effectiveness in complex real-world applications. In this work, we introduce SPAZER — a VLM-driven agent that combines both modalities in a progressive reasoning framework. It first holistically analyzes the scene and produces a 3D rendering from the optimal  viewpoint. Based on this, anchor-guided candidate screening is conducted to perform a coarse-level localization of potential objects. Furthermore, leveraging retrieved relevant 2D camera images, 3D-2D joint decision-making is efficiently performed to determine the best-matching object. By bridging spatial and semantic reasoning neural streams, SPAZER achieves robust zero-shot grounding without training on 3D-labeled data. Extensive experiments on ScanRefer and Nr3D benchmarks demonstrate that SPAZER significantly outperforms previous state-of-the-art zero-shot methods, achieving notable gains of $\mathbf{9.0\}$% and $\mathbf{10.9\}$% in accuracy.

SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding
Zhao Jin'
Rong-Cheng Tu
Jingyi Liao!
Wenhao Sun'
Xiao Luo?
Shunyu Liu' Dacheng Tao
College of Computing and Data Science, Nanyang Technological University, Singapore University of California, Los Angeles
(zhao. jin, rongcheng. tu, jingyii1i 012, wenhao006) edu. sg xiaoluoθcs uula. {shunyu. 1iu, dacheng tao @ntu, edu. sg
Abstract
3D Visual Grounding (3DVG) aims to localize target objects within a 3D scene based on natural language queries. To alleviate the reliance on costly 3D training data, recent studies have explored zero-shot 3DVG by leveraging the extensive knowledge and powerful reasoning capabilities of pre-trained LLMs and VLMs. However, existing paradigms tend to emphasize either spatial (3D-based) or seman- tic (2D-based) understanding. limiting their effectiveness in complex real-world applications. In this work, we introduce SPAZER combines both modalities a progressive reasoning framework. first holistically analyzes the scene and produces a 3D rendering from the optimal viewpoint. Based on this, anchor- guuded candidate screening is conducted to perform a coarse-level localization of potential objects. Furthermore, leveraging retrieved relevant 2D cam- era images, 3D-2D joint decision-making is efficiently performed to determine the best- best-matching object. By bridging spatial and semantic reasoning neural streams, SPAZER achieves robust zero-shot grounding without training on 3D-labeled data. Extensive experiments on ScanRefer and Nr3D benchmarks demonstrate that SPAZER significantly outperforms previous mate-offttheart zero-shot methods, achieving notable gains of 9.0% and 10.9% in accuracy. Our codes available at htttts://g com J2 9962 37-9992//PP27ER SPAZER
VLM-driven agent that
Introduction
3D Visual Grounding (3DVG) focuses on accurately localizing the referred object the 3D scene based on a user-provided query text. This task demands a comprehensive understanding of both natural language and spatial structure of the 3D scene, serving pivotal role in various real-world applications, such as embodied robotics 23 2d and augmented reality (2] In the early phase, I typically follow a fully supervised learning paradigm, training models using manually annotated textual descriptions and 3D bounding boxes. Despite their effectiveness, the reliance on aaree-caae labeled 3D data, which is expensive and labor-intensive to obtain, significantly limits their scalability and practical adoption.
3DVG approaches [6] (57 53
25
16
To reduce the dependence on costly 3D training data, zero-shot 3D visual grounding has been actively explored, largely facilitated by recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs). By leveraging the general world knowledge and reasoning
Co-corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Query Text: "this brown leather Phairlocator under the first dark brown immeeiimdiattll inside the door"
Spatial Localization
LocaliatiionSSemaniiic
Semantic Verification
Object is chair at (x, y:. 2)... Object 2 is a table at (Ka 22)..
3D-based Model
2D-based Model
Location: V Attribute:
Location: X Attribute:
Location: V Attribute: /
(a) 3D-based Method
(b) 2D-based Method
(c) Our Ourproposed SPAZER
Figure 1:
Schematic comparison between our proposed SPAZER and existing zero-shot 3DVG
approaches. Compared to (a) 3D-based methods 449 54 22 that focus on spatial localization but tend to ignore fine-graine tra-ira object attributes, and (b) 2D-based methods [8] that match visual appearance yet lack precise spatial awareness, (c) our approach progressively performs 3D spatial localization and 21) semantic verification, enabling more reliable grounding results.
abilities of pre-trained L.MSs or VL.Ms, these methods achieve zero-shot 3DVG without 3D-specific training. Since native 3D data (e.g., point clouds) cannot directly interpreted by LLMs VL.Ms, key challenge in this setting is bridging the modality gap-i.e.. transforming 3D data into proper representations that language or image models can understand. To address this, existing methods mainly follow two design paradigms: 3D-based and 2D-based. 3D-based methods [49 [4| 22 generally operate on point clouds by extracting object-level textual descriptions with categories and 3D positions, which are then processed by LLMs for reasoning. In contrast, 2D-based methods [48]
leverage 2D video sequences (which are inherently part of the scanned 3D dataset) to understand 3D scenes, using pretrained VLMs match visual frames with the language query.
Although these two categories of zero-shot methods have shown promising results, they generally specialize in either spatial reasoning semantic understanding, limiting their effectiveness in complex 3D scenes that require both. As illustrated in Fig. [ the query text not only indicates the spatial location of the target chair "under the first dark brown table" - but also highlights its visual attribute as "brown leather". Since 3D-based methods (a) typically utilize 3D coordinate- based textual description, they focus more spatial reasoning yet struggle to distinguish objects with similar location but differing appearance. Conversely, 2D-based methods (b) tend to match visually relevant objects frame-by- hame-by-framme, but lack explicit awareness of object positions within the 3D scene. However, accurate 3D visual grounding requires the effective coordination both spatial and semantic reasoning, especially in dynamic real- world environments.
To address the above issue, we propose SPAZER, a Spatial-Semannee Progressive reasoning Agent for ZEro-shot 3D visual gRounding. Inspired by cognitive neuroscience findings 4 50 that spatial reasoning and object recognition follow distinct neural streams in the human brain, our agent SPAZER decouples spatial and semantic reasoning based distinct modalities. In the spatial reasoning stage, we propose a holistic multi-view gendering-based 3D representation. This enables our agent to directly observe and understand the 3D scene itself instead of processed textual descriptions. Leveraging this representation, SPAZER first identifies the optimal viewpoint for observing the target object, then determines potential candidate objects that match the query description from the seleeted perspective. Meanwhile, our agent incorporates a retrieval -agevall--gmeeeeed anchor filtering and annotation strategy provide reliable guidance for spatial reasoning. After coarsely identifying possible candidates in the scene, relevant 2D close-up images will be retrieved from raw scanning data for fine- -rained semantic reasoning. Following 3D spatial reasoning to identify candidate object locations and 2D semantic reasoning for detail verification, SPAZER realizes 3D-2D joint decision-making to produce the final grounding result. In summary, our contributions
follows:
We introduce SPAZER, training-free 3D visual grounding agent that integrates 3D spatial localization and 2D semantic verification through a coarse-to-fine progressive reasoning process, enabling efficient 3D-2D joint decision-making

---

We propose to represent 3D point clouds in holistic rendered views instead of traditional object-level descriptions to enable VLMs direct observe and understand 3D scenes.
We propose retrieval-augmented candidate object screening strategy to improve the spatial reasoning ability of our agent, which exhibits enhanced robustness in object localization.
Extensive experiments demonstrate the superior performance of our method, largely surpassing previous state-of-the (SOTA) by 9% on ScanRefer and 10.9% % on Nr3D. SPAZER is even comparable to SOTA supervised methods on Nr3D, with marginal gap (~ 1%) in overall accuracy.
2
Related work
Supervised 3D visual grounding. As fundamental 3D vision-language task, 3D Visual Grounding
(3DVG) has gained continuous research attention in recent years 11 53 3
16
47
g 56
15
30
11
44
51 14
55 39.. The core problem in 3DVG lies in accurately localizing target object
in the 3D scene based on the given referring description. Traditional methods typically tackle this task under a supervised learning paradigm, mainly following two-stage or single-stage network architectural designs. Two-stage methods [6 57
53 7 8) adopt detect-and-mater strategy: they first extract object-level features from the scene using pre-trained 3D detection or segmentation and then fuse these with language features to compute the best-matching object. directly fuse point cloud and textual features to
models 32 20 34,
In contrast, single-stage methods 25 predict the target object's 3D bounding box in end-to-end manner. In recent years, with the development of vision-language pre- pre-training (VLP) technologies, an increasing number of methods
16 47 43
33 58 41 have adopted cross-modal joint pre-training for the 3DVG task. As the pre-training data continues scale up, these models demonstrate increasingly stronger generalization capabilities across datasets.
21
(5
13
LLM-based agent for zero-shot 3DVG. Although supervised 3DVG methods exhibit encouraging performance, their success remains highly dependent on the availability of large-scale high-quality annotations, which remain relatively limited in current 3D datasets. Inspired by the success LLM
agents in various multimodal reasoning and generation tasks (26 46 35
37 38 36), zero-shot 3DVG has emerged as a promising alternative to alleviate the reliance on 3D training data. Early approaches
build agent workflows based LLMs (e.g., ChatGPT (9)). decomposing the 3DVG task into series of sub-tasks that can be handled by the LLM. For instance, LLM-Grounder [49] formulates zero-shot 3DVG as tool-augmented reasoning task. It leverages
LLM decompose complex queries, interact with 3D perception tools (e.g.. OpenScene (3D), and perform spatial reasoning to identify the referred object. Building on the concept of visual programming [2]. ZSVG3D (54] prompts LLMs to generate interpretable visual programs composed of modular spatial functions, enabling enhanced structured reasoning. EaSe [27] employs LLMs to generate symbolic spatial relation encoders in the form of executable Python code. While LLM-based 3DVG agents are effective at modeling spatial relations, their performance is constrained by limited visual understanding capabilities, which are also crucial for accurately grounding objects in complex 3D scenes.
VLM-based agent for zero-shot 3DVG. To complement the limitations of LLM-based agents in visual perception, recent studies have turned to VLM--ased agents to enhance semantic understanding in 3D scenes. SeeGround 22 proposes hybrid 3D representation composed of spatially localized object descriptions and query-aligned local renderings, which provides additional visual contexts for the VLM, thereby enhancing grounding performance in complex 3D environments. Different from aforementioned methods that are build upon 3D point clouds, VLM-Grounder (48) proposes to realize 3DVG using only 2D video sequences, which are readily available in 3D scanned datasets. It stitches multiple frames to reduce input token length and combines various off-the-shell tool models to obtain 3D bounding boxes of VLM-selected objects. While 2D camera images offer finer-graine semantic cues compared to rendered views, the limited 3D spatial layout and relation awareness makes VIMMGrrunnder prone to spatial reasoning errors. In summary, although promising zero-shot 3DVG results have been achieved, existing methods typically rely solely on either 3D geometry representation or 2D camera images, and the potential of their joint exploitation for more comprchensive peatial-semantic reasoning remains underexplored.
3

---

3D Holistic View Selection
3D-2D Joint Decision Decision-Making
Input Scene
Multi view Observation
Camera Puramdtens K T
Query: "a brown leather chair located under the first dark brown table immediately inside the door
:" view 1, the layout of the room is visible, showing the tables and chairs most clearly...
Candidate Object Screening
Target class: chair
:" In global vlew, the chairs labeled 1. 4 are directly under this table Examining camera images, the chair labeled is confirmed to be brown and leather ... Other chairs like 5 and 1, do not match the color and material. So the best matching one chair
Detected Objects-
Matching
:" The first dark brown ta table inside the door appears to be near the left of the image. The chairs under this lable are labeled with IDs 1, 3, 4, and 5...
Anchor Filtering
Visual Object Table
Figure 2: Overview of the SPAZER framework. Given input 3D scene and query text of the target object, SPAZER performs zero-shot 3D visual grounding through spatial-semantic progressive reasoning process. first conducts 3D Holistic View Selection (Sec. (.2)) and Candidate Object Screening (Sec. (S3) to identify spatially plausible candidates by globally analyzing the 3D scene. Then, 2D-3D Joint Bocision-Makking (Sec. (3.4) is performed by leveraging 2D camera views for fine-grained semantic verification of the candidates, ultimately grounding the target object without any 3DVG-specific training.
3 Method
3.1 Overview
In the 3D visual grounding task, given query textual description, the objective is localize the referring target object within the 3D scene. Our proposed agent SPAZER solve this task in a zero-shot paradigm, i.e. directly performing inference and requiring no training on 3D visual grounding datasets. It integrates VLM as the core for reasoning and decision-makin with suite of 3D and 2D perception tools that facilitate scene understanding multiple levels of granularity. The complete architecture of SPAZER is illustrated in Fig. 2 It begins with holistic analysis of the scene's global 3D geometry to determine potential candidates that align with the query text. This process is achieved through two components: 3D Holistic View Selection (Sec. 3.2) and Candidate Object Screening (Sec. (Scc.B.3). Subsequently, SPAZER further verifies the semantic details of candidate objects using 2D camera views, and derives the refined grounding result through 2D-3D Joint Decision- Decision-Making in Sec, 3.4
equipped
3.2 3D holistic view selection
Since VLMs are not capable of directly interpreting 3D data, existing methods either convert 3D into textual narratives 44 [54,, or combine textual descriptions with locally rendered perspec- tive views (22]. Although such text-centric 3D representations provide explicit object coordinates, their reliability is highly sensitive the accuracy of predicted bounding boxes. Inaccuracies in object category or location predictions can severely compromise the spatial reasoning process. To overcome these limitations, we propose directly empower the agent with multi-view observation capabilities, enabling it to understand the 3D scene itself without relying on intermediate textual representations.
Multi-view observation. In this step. SPAZER leverages 3D rendering tools generate holistic views of the input point cloud from multiple perspectives. Firstly, the viewpoint is set directly above the center of the scene to obtain the Bird's Bird's-Eye View (BEV). To ensure that the BEV image fully captures the entire scene, the distance between the viewpoint and the scene center is calculated as
1 max mmxx(Zz,1 1n) = x 2
(I)
tan (0/2)

---

where lg and by denote the scene extents along the x-axis and y-axis, respectively, and & represents the rendering camera's 's Field of View (FoV). Furthermore, we introduce n additional viewpoints from different perspectives to observe the scene. The n viewpoints are distributed uniformly along circular path above the scene, each separated by 360% /n.0 all sharing a fixed oblique viewing angle = 45 and the distance to the scene center equals to d. For example, when n. = 4, the viewpoints are placed at the front, back, left, and right of the scene, providing four angled top-down renderings.
View selection. After obtaining the all (n + 1) rendered global views Iad = { La...... 1aa) we feed them along with the query text 9 into the VLM, which is prompted to select the optimal viewpoint that best captures the descrdde object:
Psd VLM q),
(2)
where I3d denotes the selected 3D rendering view. After this step, our agent has holistically observed the scene and rendered optimal view, which serves as the global 3D representation for the subsequent progressive localization of the target object.
3.3 Candidate object screening
Leveraging the selected optimal viewpoint, SPAZER proceeds to infer the position of the target object that satisfies the query language. To facilitate this procedure, we follow the previous setting [54] [2] to utilize an off-the-shelf 3D network to extract object layout information, which can be represented as: O = ((b, 4)) = 1,2,...,NN where b, represents the 3D bounding box of the i-th object, and denotes its associated class label.
Retrieval-augmenntr anchor filtering. Since VLMs struggle directly output precise spatial coordinates, we provide visual anchors as cues by leveraging object layout information. Firstly, the agent filters out irrelevant objects by analyzing the query text to identify the target object class c*. Subsequently, a text-based matching algorithm is employed on c* and each detected object category e, to compute the similarity score s(c", .4). And the object class with the highest similarity score will be used to filter out irrelevant objects whose predicted class not equaling to it. However, due to the free-form nature of the query text, object categories may be semantically ambiguous (e.g., "cabinet" might be referred to as "shelf" or "board" in the query). As a result, relying solely on text- text-based matching can lead to unreliable results. To address this issue, we propose visual retrieval- -uggeettd matching mechanism. We first set threshold r to filter out low-confidence matching results:
غ 2= Cj-, if s(cc,,,,)) Ø,
where 1 = arg max s(c", i=!...,N
(3)
otherwise
If valid è is obtained, only the detected objects with predicted class label equaling to è will be retained, forming the subset Ocst (b),,)) O c c). When s(c", C.. < T, our agent will switch from
matching to vision-based matching to improve the reliability. It first constructs a visual object table, which stitches the cropped object images based on the position of each 3D bounding box. This step shares a similar process as camera view mapping in Sec. 3.4 which will be elaborated later. Then the VLM is prompted to automatically filter out objects that do not match the target category based on the visual object table. In this way, when unreliable object category predictions occur, our agent can still re-evaluate based on visual cues, thereby enhancing robustness.
Candidate screening. After filtering out irrelevant objects, the remaining object IDs will be annotated on the previously selected view images as visual anchors. For each ID, the 3D-to-2D coordinate transformation is computed based on the rendering parameters of the current view, and the annotation placed at the center of the box. The VLM is then prompted identify potential anchors from the annotated IDs. At this stage, although it is feasible to directly select the target object based on the query text, 3D renderings often lack fine- aee-grainedd details and struggle to tell view-dependent spatial relationships. So we relax this process to coarse selection of the Top-k most likely objects:
OTop. = Top-k (VLM(I55 O.q))
In subsequent experiments, we adjust the value of k to ensure a balance between increasing the likelihood of covering the target object and avoiding excessive image inputs and token overhead.
3.4 3D-2D joint decision-making
In the preceding phase, SPAZER has conducted a initial reasoning and identified the coarse candidate pools. Next, we incorporate informative 2D camera views to enrich semantic context, thereby
5

---

facilitating 3D-2D joint decision- making, The 2D camera images are commonly available in original RGB-D scans, capturing detailed visual appearance of the scene. However, due to the large number of images (e.g., up to thousands of images per scene in ScanNet [9), how to efficiently sample
keyframes remains a challenge in existing studies [48 58 33. Benefiting from the global 3D spatial reasoning process, our agent has obtained k candidate anchors. Accordingly, we efficiently sample k corresponding keyframes based on the 3D-to-2D 2D mapping of the anchor positions.
Camera view mapping, Our agent utilizes the camera intrinsics along with the extrinsics to compute the mapping from 3D points to 2D camera views. To determine the most informative view for each candidate anchor, we project its associated 3D bounding box P3D E Rxx9 onto camera view images and evaluate per-view visibility. Specifically, we extract nine key points from each anchor's bounding box, including its eight vertices and the center. These 3D points, defined in world coordinates, are first
converted to homogeneous coordinates P3D E R4x9 (i.e., add 1-padding for affine transformation). Then we perform a transformation analogous to point cloud registration [45] them into the camera coordinate system using the extrinsic matrix T E
18 [9] to convert R 44x4 resulting in
P Paam = Tew P3D. Next, we project the 3D points in camera coordinates onto the 2D image plane using the intrinsic matrix K E R3x3 For each point, let Peam = eee Ve: =o]" be its 3D position in the camera frame. The corresponding 2D pixel coordinate P2D = [u, u] computed by:
uu' v 1
[e/ZZ]
K
Ve 2c
where u and v represent the horizontal and vertical coordinates of the projected point in the image plane. To handle occlusions, we compare the projected depth with the depth map. A point is considered visible if its projected coordinate (u, v) lies within the image bounds and the depth difference is negligible. By computing the camera view mapping across multiple images, anchor- relevant views can be determined. Since an anchor object may appear in multiple views, we compare the number of visible projected points across all camera images to select the one that observes the most complete object bounding box as its corresponding 2D view. Finally, the selected 2D camera images for all candidate anchors are obtained as the set Tza = (12 L2,.... }. where each I2ga corresponds to the most informative view associated with the j-th candidate anchor.
Joint decision-making. After obtaining corresponding camera view images of the selected Top-k candidate anchors, our agent performs joint decision decision-making by leveraging both 3D global rendering and 2D camera images. To enable the VLM seamlessly integrate different perceptual modalities during reasoning, the same candidate object is annotated with consistent ID across rendered image and different video frames. In this process, SPAZER leverages the global 3D rendering for spatial reasoning, focusing mainly on re-evaluating the view-independent relations (e.g., in the middle/corner of the room). Mcanwhile, utilizing 2D camera views, it further verifies semantic details of candidate objects (e.g., color, shape, material) and infers view-dependent relations (e.g., left, right). After jointly considering both modalities for decision decision-making, the agent identifies the matching target and outputs its 3D bounding box: b" = VLLM Sd: Izd; O,4).
4 Experiment
4.1 Experimental setup
Datasets. We evaluate our method on two widely-used 3D visual grounding benchmarks: ScanRe- fer (6] and Nr3D (I. which are built upon the ScanNet (9] dataset. ScanRefer comprises 51,583 human-written natural language descriptions across 800 indoor scenes from ScanNet, each referring to a specific object in 3D space. Queries are classified into "Unique", where only a single object of the target class exists, or "Multiple", where other same-class distractors are present, requiring more precise discrimination. Nr3D contains 41,503 descriptions collected via two wwo-player reference game aimed at improving linguistic precision. Each query refers to target object that is always accompanied by at least one distractor, and queries are labeled as either "Easy" (one distractor) or "Hard" (multiple distractors). Moreover, queries are also categorized as "Viee--Deeendett" (Dep.) (Indep.) depending on whether understanding spatial relations like "left" or "right" requires specific camera viewpoints. To enable fair comparison and reduce expenditure, our main experiments are conducted on the same ScanRefer and Nr3D subsets (48).
or "View-Independen"

---

Table 1: Quantitative comparison with supervised and zero-shot 3DVG methods on ScanRefer 6
Unique
Multiple
Overall
Method
Zero-shot
LLM/VM
Acc@0.25
Acc@0.5
Acc@0.255
Acc@0.5
Acc
Acc@0.5
ScanRefer 6
67.6
46.2
32.1
21.3
39.0
26.1
TDVG-Transsormmrr
12
81.9
60.6
39.3
28.4
47.6
34.7
BUTD-DETR
84.2
66.3
35.1
52.2
ChatScene
13
Vicuna 7B
89.6
82.5
47.8
42.9
$5.5
50.2
Video-3D
LLaVA-Viae VVdeo 7B
78.3
45.3
GPT4Scene
3
Qwen2 Divn2-VLL-7B VL.7B
90.3
83.7
56.4
50.9
62.6
57.0
OpenSeene
31
CLIP GFT-4 turbo GPT-4 turbo
20.1
13.1
11.1
4.4
13.2
6.5
LLM-Grounder
17.1
ZSVG3D [4
63.8
58.4
27.7
24.6
36.4
32.7
VLM-Grounder
48
GPT-4o
66.0
29.8
48.3
33.5
51.6
32.8
SeeGround
22
Qwen2-VL-72B
75.7
68.9
39.4
CSVG 52)
Mistral-Large-2400
68.8
61.2
38.4
27.3
49.6
39.8
SPAZER (Ours)
GPT-4o
80.9
72.3
51.7
43.4
57.2
48.8
Evaluation metrics. The ScanRefer dataset directly evaluates the accuracy of localizing the 3D bounding box of the object described in the query. The measuring metrics annotated Acc@0.25 and @0.5, which represent the proportion of output bounding boxes whose IoU with the ground truth exceeds 0.25 or 0.5, respectively. In comparison, the Nr3D benchmark offers ground truth 3D bounding boxes for all objects, and the evaluation focuses on the selection accuracy.
Implementation details. Our agent adopts GPT--4o the default VLM. The number of views is set to 4, and the Top-k parameter is set to = 4. For ScanRefer dataset, we follow prior works [54 22] and use pre-trained model [4 to obtain the 3D bounding boxes. All ablation studies conducted Additional implementation details are included in the Appendix.
on the same subset of Nr3D as 48
4.2 Quantitative comparison
ScanRefer. Tab. I shows the performance comparison on the ScanRefer dataset. Our SPAZER
significantly outperforms all existing zero-shot methods, achieving the highest overall accuracy. Compared to previous state-of-the-art 2D-based method VLM-Grrunder (48) and 3D-based method CSVG (52), SPAZER achieves gains of +24.4% / +16.0% and +7.6% / +8.9% in Acc@0.25 A.c@0.5, respectively. In addition, our Acc Acc@0.25 25 already matches or exceeds many fully-supervised 3DVG methods, such ScanRefer (6) 3DVG-Transfomme [57]. and BUTD-DETE (16). without training on 3D datasets. This demonstrates strong zero-shot generalization. Although methods like Video-3D LLM [58] and GPT4Scene [33 show stronger performance, they require large-scale 3D-language data for extensive pre-training In contrast, our method exhibits ease of deployment, and already achieves comparable Acc@0.25 showcasing its potential for practical application.
Nr3D. As shown in Tab. 2 our SPAZER surpasses all existing zero-shot methods in overall accuracy and even approaches the performance of fully-supervised methods, demonstrating the effectiveness important note that the default
of our spatial-semantic progressive reasoning framework. setting of the Nr3D dataset provides ground-truth 3D bounding boxes (without class label), meaning that the evaluation focuses primarily on the model's ability to reason over language to identify the correct object, rather than assessing the 3D localization precision. Under this setting, SPAZER achieves an overall accuracy of 63.8%, significantly outperforming prior zero-shot approaches such as VLM-Grounde (48] and EaSe 27). Moreover, recent methods like CSVG [52] and Transcrib3DD [TO] use additional ground-trun -tuthh object classes 3D bounding boxes during inference. In this case, our method also achieves superior overall accuracy, especially with significant gains in the Hard category.
4.3
Qualitative comparison
Fig illustrates qualitative grounding results of SPAZER in comparison with the representative 2D- based method YLM-GGounder (88 and 3D-based method SeeGround [22]. (a), multiple identical objects (tables) appear in the scene, and spatial cues in the query (e.g., "in the middle") are required correctly localize the target. VIMM-Grundde fails to resolve the spatial ambiguity due to its limited 3D understanding. In (b), the query includes fine-grained semantics (e.g., "a white lamp top of it"), which are hard to capture from 3D point cloud data alone, leading SeeGround mispredict the target. In contrast, our SPAZER combines 3D spatial reasoning with 2D semantic verification,
7

---

Table 2: Quantitative comparison with supervised and pero-shot 3DVG methods on Nr3D ..
Method
Zero-shot
LLM/VLM
GT object class
Easy
Hard
Dep.
Indep.
Overall
Referlt3DNet
43.6
27.9
32.5
37.1
35.6
3DVG-Transfrmet
48.5
34.8
34.8
43.7
40.8
ViL3DRel g
X
70.2
57.4
62.0
64.5
64.4
3D-VisTA
59
72.1
56.7
61.5
65.1
64.2
MiKASA
69.7
65.4
Scene Verse
72.5
57.8
56.9
67.9
64.9
7SVG3D 54
GPT-4 turho
X
46.5
31.7
36.8
40.0
39.0
SeeGround 22
Owen2-VL-722
X
54,5
38.3
42.3
48.2
46.1
VLMMCrrunder
GPT-4o
55.2
39.5
45.8
49.4
48.0
27
GPT-40
52.9
SPAZER (Ours)
GPT-4o
68.0
58.8
59.9
66.2
63.8
CSVG 52
Mistral-I 2rrge-2407
67.1
51.3
53.0
62.5
59.2
Transerib3D 10
GPT-4
79.7
60.3
60.1
75.4
70.2
2
GPT-4o
67.8
SPAZER (Ours)
GPT-4o
76.5
69.3
65.6
73.2
VLM- Grounder
SeeGround
Query SPAZER (Ours)
(6) table midldle it is squore.
(b) shelf there a white lamp top it
(c) the trash the floor to left cabinet, there is small white bin beside to the right.
table grey chair is behind
choins. grey choir is leaning back and has black chair to the of
Figure 3: Qualitative comparison of 3DVG results. Incorrectly predicted 3D bounding boxes highlighted in red, while correct ones are in green. Key spatial and semantic cues are underlined.
allowing accurate object identification in both cases. Further, in more complex cases like (c) and (d), where both spatial and semantic must be jointly reasoned over (e.g., "small white bin beside it". "leaning back", "black chair to the right of it"), SPAZER clearly outperforms both baselines by effectively leveraging the complementary strengths of both 3D geometry and 2D semanties.
4.4 Ablation study
Effect of 3D holistic view selection (HVS). As mentioned in Sec. 3.2 our SPAZER first holistically renders multiple perspective views and identifies optimal one for the query text. To analyze
effectiveness of such design, we compare the following two variants: (1) BEV: Skip view selection
and use the BEV by default, which also consistent with prior work [33. (2) Random: Use
random view instead of agent-selected one. In Tab. 3 BEV performs poorly
may fail to reliably
capture the target object. Random performs slightly better than BEV. In contrast, our agent leverages spatial understanding to select more informative views for subsequent steps, leading to improved overall accuracy (+3.6% vs. Random and +6.3% vs. BEV in average).
Effect of candidate object screening. At the candidate screening stage (Sec, 3.3). irrelevant object anchors are filtered out. As shown in Tab. 3 incorporating our proposed Retrieval-augmentee anchor filtering (RAF) on top of the text-based baseline consistently improves accuracy by 2% to 7%. This because text-based matching is vulnerable inaccurate object class prediction, whereas our RAF introduces the visual object table as complementary cue, leading to improved robustness.

---

Table 3: Ablation study the component design of our agent. The default setting is highlighted
View Selection
Anchor Filtering
w/o JDM (3D only)
w/ JDM (3D+2D)
BEV
Random
HVS
Text-based
RAF
# Hard
Dep.
Overall
#
Hard
Dep.
Overall
X
x
(a)
36.8
37.5
41.2
(g)
48.3
49.0
53.2
d
(b)
36.0
39.6
44.4
(h)
51.8
49.0
54.8
X
/
/
(c)
38.6
42.7
46.8
(i)
52.6
53.1
56.4
X
x
/
(d)
34.2
36.5
43.2
(
47.4
51.0
56.8
x
/
(c)
31.6
41.7
46.8
(k)
50.9
55.2
59.2
X
/
/
(1) 43,9
49.0
52.4
())
58.8
59,9
63.8
Effect of 3D-2D joint decision-making (JDM). The JDM module in Sec. 3.4 serves the core of our approach, as it enables the integration of 3D spatial reasoning and 2D semantic verification within our agent. To enable comprehensive analysis, in Tab. 3 further include comparisons between w/o JDM and w/ JDM under each of the previously defined ablation settings. Here, w/o JDM refers to our agent making decisions using 3D-only input, which is same previous 3D-based methods [54 [22]. In this setting, our method achieves an overall accuracy of 52.4%, outperforming ZSVG3D 54 and SeeGround (22 by 13.4% and 6.3%, respectively. With the introduction of informative camera views as complementary 2D semantic context through camera view mapping, our agent achieves substantial improvement- an average accuracy gain of 11.6 %. This result strongly supports the effectiveness of our spatial spatial-semantic joint reasoning.
Effect of k in Top-k. In the didate screening stage, our agent gives the Top-k most possible tar-
78.0
79.2
8000
76.8
77.6
79.2
78.8
78.4
semmona (%)
70,0
get object IDs. We evaluate the Top-k accuracy (i.e., whether the ground-truth object is in the k can- didates) and the overall grounding accuracy under different values of k. As shown in Fig. 4 (a) As k in- creases, the Top-k accuracy steadily improves, indicating higher likeli- hood of including the ground-trui object into candidates. This also contributes to the improvement of overall accuracy. However, increasing from 4 to 5 leads to a performance drop, suggesting that an excessively large l may introduce distracting candidates for subsequent decision-makern
70
61.6
63.8
62.0
63.8
62.0
60
56.4
56.8
58.0
59.2
52.4
Ovemll
Ovemil Top-k
Top -k
50
2
Parameter in Top-k
(b) The number of views
Figure 4: Ablation studies on parameter k and n.
Number of rendered views n. In Fig. (b), show how varying the number of rendered views affects the accuracy. When n = 0, we only use the BEV by default. As n increases from 0 to 4, the accuracy gradually improves. This is because more global views are provided for the agent to select from, allowing better observation of the target object. However, as n further increases from 4 to 8, the accuracy starts decline due to increasing overlap between views, which introduces redundant information and adds unnecessary noise, ultimately impairing the agent's 's decision decision-making, The Top-k accuracy follows a similar trend to the overall accuracy, albeit with smoother variations. This suggests that even when the target object is included in the candidates, suboptimal 3D global view can still hinder the subsequent 3D-2D joint reasoning, highlighting the importance of view selection.
Influence of VLMs. The VLM serves as the "brain" of our agent, enabling effective query text interpretation and spatial-semantic rea- soning. To investigate the impact of the adopted VL.M on overall agent performance. we conduct comparative analysis of several mod- els, including proprietary ones (GPT-40-mini, GPT-4o, GPT-4.1) and open-source alternatives (Qwen2-VL-72B [2). Qwen2.5-VL-72B (2).
Table 4: Ablation study of the VLM used in our agent.
VLM
Overall
Qwen2- VL-72B
53.6
Qwen2.5-VL. VL--2B
56.0
Tab. 4 leads to the following conclusions: (1) The GPT series mod-
GPT-4o-mini
46.4
els exhibit stronger reasoning capabilities on the 3DVG task than open-source models; and (2) stronger VLMs contribute to improved performance of our agent, suggesting that continued advancements in VLMM may further unlock its potential. Note that although GPT-4.1 can further boost the performance
GPT-4o
63.8
GPT-4.1
64.3
of our agent, we still adopt GPT-4o as the default VL.M to ensure fair comparison with previous methods.

---

Table 5: Performance comparison on the complete set and subset of the Nr3D dataset.
Dataset
Method
VLM
Easy
Hard
Dep.
Indep.
Overall
SeeGround
Awen2-V1L-722)
54.5
38.3
42.3
48.2
46.1
Complete
Ours
Dwen2-V1.-722
59.3
44.8
46.3
54.9
51.8
Ours
Qwen2.5-VL-72B
62.4
46.9
49.9
56,8
54.3
SeeGround 22
Qwen2-VL-72B
51.5
37.7
44.8
45.5
45.2
Subset
Ours
Qwen2-VL-728
62.5
43.0
51.0
55.2
53.6
Ours
Qwen2.5-VL-72B
60.3
50.9
54.2
57.1
56.0
Table 6: Inference time of each step in our SPAZER. Ours significantly outperforms VLM-Grounder in efficiency. Both methods adopt GPT-4o as the VLM.
Method
Step
Time (s)
Total (s)
3D Holistic View Selection
5.2
Ours (SPAZER)
Candidate Object Screening
8.5
23.5
3D-2D Joint Decision- Maainn
9.8
VLo-Grrunder
50.3
Performance on subset vs. complete dataset. In consideration of budget and evaluation efficiency, we follow previous work VLM-Grounder [48] evaluate our agent on the subset (250 selected samples) of each dataset. To further verify whether the results on the subset are comparable to those on the full dataset, we conduct additional experiments using open-source VLMs, as shown in Tab, [ We observe that both our method and prior work SeeGround [22] exhibit consistent performance across the two dataset partitions, with overall accuracy variations under 2.0, which is negligible compared to the improvement achieved by our method.
Inference time. In Tab. 6 break down the inference time of each step in SPAZER. The time consumption across different steps is relatively balanced, with the view selection step being faster since it requires
additional computation. Moreover, compared to VI.M-Grounder (48), which also leverages 2D) camera images for reasoning, our method achieves significantly higher inference efficiency. This is because we rely on VLMM-sleecee anchors and require only a small number of images, avoiding the need to sample and filter all video frames.
Conclusion and limitations
In this work, we propose SPAZER, a novel spatial-semante progressive reasoning agent for zero-shot 3D visual grounding. Instead of traditional object-level descriptions. SPAZER leverages holistic 3D rendered views to provide global spatial context, allowing VLMs to directly observe and interpret the 3D scene. It further employs retrieval-augmmente candidate screening strategy to enhance spatial reasoning and improve robustness against object category ambiguity. Extensive experiments verified the effectiveness of SPAZER, showcasing the potential of the 3D-2D joint reasoning paradigm in zero-shot 3DVG and its promise as a scalable alternative to supervised approaches.
Limitations. As we follow prior works and adopt pre-trained models to obtain 3D bounding boxes. the grounding accuracy will be related to their localization accuracy. addition, the calculation of 3D-to-camera mapping can be affected by inaccurate camera parameters and depth information. For more detailed discussions of limitations, failure cases, and error types, please refer the Appendix.
Acknowledgments and Disclosure of Funding
This project is supported by the National Research Foundation, Singapore, under its NRF Professor- ship Award No. NRF-P2024-001.
10

---

References
[1] Panos Achlioptas, Ahmed Abdelrcheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. ReferIt3D: Neural listeners for finc- fine-graing grained 3D object identification in real-world scenes. In European Conference on Computer Vision (ECCV), pages 422-440, 2020.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenhin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13393, 2025.
[3] Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 3DJCG: A unified framework for joint dense captioning and visual grounding on 3D point clouds. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16464 16473, 2022.
[4] Christopher F Chabris, Thomas E Jerde, Anita W Woolley, Margaret E Gerbasi, Jonathon P Schuldt, Scan L Bennett, J Richard Hackman, and Stephen M Kosslyn. Spatial and object visualization cognitive styles: Validation studies in 3800 individuals. Group brain technical report, 2:2, 2006.
[5] Chun-Peng Chang, Shaoxiang Wang, Alain Pagani, and Didier Stricker. MiKASA: Multi-key-ancher & scene-aware transformer for 3D visual grounding. In Proceedings of the IERE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 14131-14140, 2024.
[6] Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. ScanRefer: 3D object localization in RGB-D -D scans using natural language. In European Conference on Computer Vision (ECCV), pages 202-221, 2020.
[7] Dave Zhenyu Chen, Qirui Wu, Matthias Nicßner, and Angel X Chang. D3Net: A unified speaker- listener architecture for 3D dense captioning and visual grounding. In European Conference on Computer Vision (ECCV), pages 487-505. Springer, 2022.
[8] Shizhe Chen, Pierre-Lout Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3D object grounding. Advances in Neural Information Processing Systems (NeurlPS), 2022.
[9] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly- annotated 3D reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 5828-558 5839,, 2017.
[10] Jiading Fang. Xiangshan Tan, Shengjie Lin, Igor Vasiljevic, Vitor Guizilini, Hongyuan Mei, Rares Ambrus, Gregory Shakhnarovich, and Matthew Walter. Transerib3D: 3D referring expression resolution through large language models. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9737-9744. IEFE, 2024.
[11] Wenxuan Guo, Xiuwei Xu. Ziwei Wang, Jianjiang Feng, Jie Zhou, and Jiwen Lu. Text-guided sparse voxel pruning for efficient 3D visual grounding. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[12] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14953-14962. 2023.
[13] Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, al. Chat-Scene: Bridging 3D scene and large language models with object identifiers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[14] Pin-Hao Huang, Han-Hung Lee, Hwann-Tzong Chen, and Tyng-Luh Liu. Text-guided graph neural networks for referring 3d instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages
2021.
[15] Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3D visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15524-15533, 2022.
[16] Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In European Conference on Computer Vision (ECCY), pages 417-433. Springer, 2022.
[17] Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang. Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Scene Verse: Scaling 3D vision-language language learning for grounded scene understanding. In European Conference on Computer Vision (ECCV), pages 289 310, 2024.
11

---

[18] Haobo Jiang, Yaqi Shen. Jin Xie, Jun Li. Jianjun Qian, and Jian Yang. Sampling network guided cross- entropy method for unsupervised point cloud registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6128-6137, 2021.
[19] Haobo Jiang, Zheng Dang, Zhen Wei, Jin Xie, Jian Yang, and Mathieu Salzmann. Robust outlier rejection for 3d registration with variational bayes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1148-1155 1157, 2023.
[20] Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. PointGroup: Dual-set point grouping for 3D instance segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 4867-4876 2020.
[21] Zhao Jin, Munawar Hayat, Yuwei Yang. Yulan Guo, and Yinjie Lei. Context-aware alignment and mutual masking for 3D-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10984 10994, 2023.
[22] Rong Li, Shijie Li, Lingdong Kong, Xulei Yang, and Junwei Liang, SeeGround: See and ground for acro-shot open- vocabulary 3D visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[23] Daizong Liu, Yang Liu, Wencan Huang, and Wei Hu. A survey text-guided 3D visual grounding: elements, recent advances, and future directions. arXiv preprint arXiv:2406. 05785, 2024.
[24] Ziyang Lu, Yunqiang Pei, Guoqing Wang. Peiwei Li, Yang Yang, Yinjie Lei, and Heng Tao Shen. ScanERU: Interactive 3D visual grounding based on embodied reference understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3936-3944, 2024.
[25] Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3D-SPS: Single-stage 3D visual grounding via referred point progressive selection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16454-16463, 2022.
[26] Junyu Luo, Weizhi Zhang. Ye Yuan, Yusheng Zhao, Junwei Yang. Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. Large language model agent: A survey on methodology, applications and challenges. arXiv preprint arXiv:2503.2) 2/60, 2025.
[27] Boyu Mi, Hanqing Wang. Tai Wang. Yilun Chen, and Jiangmiao Pang. Evolving symbolic 3D visual grounder with weakly supervised reflection. arXiv preprint arXiv:2502.01400 2025.
[28] Taiki Miyanishi, Daichi Azuma, Shuhei Kurita, and Motoaki Kawanabe. Cross3dvg: Cross-dataset 3D visual grounding on different RGB-D scans. In International Conference on 3D Vision (3DV), pages 717-727. 2024.
[29] OpenAL. ChatGPT. htttps://chat openai.
[30] Qihang Peng, Henry Zheng, and Gao Huang. Proxytransformation: Preshaping point cloud manifold with proxy attention for 3D visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[31] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser. et al. OpenScene: 3D scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 815 824, 2023.
[32] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3D object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9277 9286, 2019.
[33] Zhangyang Qi. Zhixiong Zhang. Ye Fang, Jiaqi Wang, and Hengshuang Zhao, GPT4Scene: Understand 3D seenes from videos with vision-language models. arXiv preprint arXiv:2501.01428 2025.
[34] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3D: Mask transformer for 3D semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8216-8223. IEEE, 2023.
[35] Yongliang Shen, Kaitao Song. Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. HuggingGPP: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems (NeurIPS), 36:38154 144-55-55 38180, 80, 2023.
[36] Rong-Cheng Tu, Wenhao Sun. Zhao Jin, Jingyi Liao, Jiaxing Huang, and Dacheng Tao. SPAgent: Adaptive task decomposition and model selection for general video generation and editing, arXiv preprint arXiv:2411.18983, 2024.
12

---

[37] Tu, Zhao Jin, Jingyi Liao. Xiao Luo, Yingje Wang, Li Shen, and Dacheng Tao. Mllm- guided vlm fine-tuning with joint inference for zero-shot composed image retrieval. arXiv preprint WwXiv:250.19907. 2025.
[38] Rong- Cheng Tu, Wenhao Sun, Hanzhe You, Yingjie Wang. Jiaxing Huang. Li Shen, and Dacheng Tao. Multimodal reasoning agent for zero-shot composed image retrieval. arXiv preprint arXiv:2505.19955 2025.
[39] Ozan Unal, Christos Sakaridis, Suman Saha, and Luc Van Gool. Four ways improve visual fusion for dense 3d visual grounding. In European Conference on Computer Vision (ECCV), pages 196-213, 2024.
[40] Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias NieBner. Rio: 3D object instance re-localization in changing indoor environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7658-7667, 2019.
[41] Haochen Wang, Yucheng Zhao, Tiancai Wang, Haoqiang Fan, Xiangyu Zhang, and Zhaoxiang Zhang. ROSS3D: Reconstructive visual instruction tuning with awaaeeeessss arXiv preprint arXiv:2504.01901, 2025.
[42] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen. Xuejing Liu. Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- aision-language model's perception the world at any resolution. arXiv preprint arXiv:2409. /2/91, 2024.
[43] Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. EmbodiedScan: A holistic multi-modal 3D perception suite towards embodied AI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19757-19767, 2024.
[44] Yuan Wang. Ya-Li Li, WU Fastman 7Y, and Shengjin Wang. LIBA: Language instructed multi- gaaiiligaanllrrity bridge assistant for 3D visual grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 81 $114-8123 2025.
[45] Yue Wang and Justin M Solomon. Deep closest point: Learning representations for point cloud registration. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3523-3532, 2019.
[46] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023.
[47] Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. EDA: Explicit text-decoupling and dense alignment for 3D visual grounding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 19231-19242, 2023.
[48] Runsen Xu, Zhiwei Huang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. VIMM-Gronnde; VLM agent for zero-shot 3D visual grounding. In Conference on Robot Learning (CoRL). 2024.
[49] Jianing Yang. Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David Fouhey, and Joyce Chai. LLM-Grounder: Open-vocabulary 3D visual grounding with large language model as an agent. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7694-7701. IEEE, 2024.
[50] Jihan Yang. Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14171 2024.
[51] Hanxun Yu, Wentong Li, Song Wang, Junbo Chen, and Jianke Zhu. InsL3D-LMM: Instance-aware 3D scene understanding with modal instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[52] Qihao Yuan, Jiaming Zhang, Kailai Li, and Rainer Stiefelhagen. Solving zero-shot 3D visual grounding constraint satisfaction problems. arXiv preprint arX1v:2411.14594, 2024.
[53] Zhihao Yuan, Xu Yan. Yinghong Liao, Ruimao Zhang, Sheng Wang. Zhen Li, and Shuguang Cui. In- stanceRefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages
2021.
13

---

[54] Zhihao Yuan, Jinke Ren, Chun-Mei Feng, Hengshuang Zhao, Shuguang Cui, and Zhen Li. Visual programminng for zero-shot opea-locabulary 3D visual grounding. In Proceedings of the IELE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20623-20633, 2024.
[55] Yiming Zhang. ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple objects. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15236, 2023.
[56] Yuqi Zhang, Han Luo, and Yinjie Lei. Towards CLIP-driven language-free 3D visual grounding via 2D-3D relational enhancement and consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13063 13072, 2024.
[57] Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. IDVG-Transformer:: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2928 2937, 2021.
[58] Duo Zheng, Shijia Huang, and Liwei Wang. Video-3D LLM: Learning position-awar video representation for 3D scene understanding. In Proceddings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[59] Ziyu Zhu, Xiaojian Ma. Yixin Chen, Zhidong Deng. Siyuan Huang, and Qing Li. 3D- DISTI:::: Pre- Preitrained transformer for 3D vision and text alignment. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1-11-29,1, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the core contributions of our work, including the design a spatial-semante reasoning agent for zero-shot 3D visual grounding, the design of 3D holistic view selection and candidate screening. These claims are directly supported by the experimental results in the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: This paper includes dedicated "Limitations" Sec. Specifically, it notes that grounding accuracy depends on the quality of 3D bounding boxes from pre-trained models and may affected by inaccuracies camera parameters and depth estimates. Additional failure cases and error types are further analyzed in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not present theoretical results formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides detailed descriptions of the model architecture, imple- mentation details, and evaluation protocols. We also include necessary hyperparameters and dataset settings for reproduction. Additional implementation details (such as the input prompts for VLM) are provided the Appendix due space constraints. Code will be made publicly available upon publication to facilitate reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will release the full codebase with data preprocessing, model implementa- tion, and evaluation scripts upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
pub 1 ic/ gui des / CoodeSuomei ss ConPolicy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https?
//nips.c. ctppbliccuuucuuuuuuuccco des /es/CooeSuubrnn ss onPo licy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our method is training-free free and does not require model optimization. Never- theless, we clearly describe all necessary experimental settings, including adopted evaluation datasets, hyyee--arametes and justification of the choices them, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
17

---

The paper does not report error bars or statistical significance tests, as the core results are obtained using a deterministic setup. In particular, configure the VLM with low temperature setting to minimize sampling variance and ensure consistent outputs across runs.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Most experiments rely on closed-source VLMs and only small portion of our pipeline (mainly the 3D rendering step) requires GPU computation, which imposes no specific requirement on GPU model. For experiments involving open-source models, we provide the relevant compute settings and resource information in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://ttwss.e...........pa/aaaa
Answer: [Yes]
Justification: We have carefully reviewed the NeurlPS Code of Ethics and affirm that our submission adheres to its principles.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
18

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss potential broader impacts of our work, including both its positive applications and possible risks, in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve the release of generative models, scraped datasets, other components with high misuse potential.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
19

---

Answer: [Yes]
Justification: All external assets used our work-including datasets, codebases, and models-an properly cited with their corresponding licenses and sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw thcode com / data sets has curated licenses for some datasets. Their licensing guide help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets introduced this work are released under the CC-BY 4.0 license and are accompanied by README file provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/cod//model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: This work does not involve any crowdsourcing tasks research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
20

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve human subjects and therefore does not require IRB or equivalent ethical approval.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: Large language models (LLMs) are a core component of our method. Specifi- cally, them for progressive spatial-semantic reasoning in the 3D visual grounding process. The paper details each step in which the LLM is involved, and we provide the exact prompts used at each stage the Appendix to ensure transparency and reproducibility.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non standard components.
Please refer to our LLM policy (https: http:///ppp.cccooccceeeeee/////a 2025/
LLM) for what should should not be described.
21

---

SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding
Appendix
A. Implementation details
B. Additional experimental results and analysis:
B.
Error type analysis
B.2
Additional evaluation cross-dataset generalization
B.3
Stage-wise ablation the effect of VLMM capacity
B.4
Case study implied object grounding
C. Limitations and border impact:
Analysis of failure cases
Broader impact
A Implementation details
Computational resources. Note that most of our experiments adopt GPT-4o as the VLM and requires no GPU-based computation. The experiments involving Qwen2-VL-728 and Qwen2.5-VL. 72B conducted on multiple NVIDIA H100 GPUs.
Model details. The default VL.M of our agent is GPT-4o (gnt-40-2004400--06) And the temperature is set to 0.2 improve the reproducibility of the results. On ScanRefer dataset [6, we Mask3l) [34
to obtain 3D bounding box predictions, which consistent with prior works 54
Prompt design. Our agent adopts several different prompts for the VLM. We first conduct target class prediction using the prompt Tab. 7 which similar with VLM-Grounder [48]. For view selection (Sec.3.2), 3.2). we simply tell the VLM to select the view that can observe the query-described object most clearly using the prompt in Tab. [8] In candidate object screening (Sec. we prompt the VLM to select Top-k candidate objects based the annotated object IDs, as shown in Tab. 9 Eventually, 3D-2D joint recision-making (Sec, 3.4) is achieved using the detailed prompt in Tab. I0
Table 7: Prompt for reasoning the object category from the query description. "(text} represents the input query.
Prompt Template for Target Class Prediction
You are working on 3D visual grounding task, which involves receiving a query that specifies a particular object by describing its attributes and grounding conditions to uniquely identify the object.
Now, I need you to first parse this query and return the category of the object to be found. Sometimes the object's 's category is not explicitly specified, and you need deduce it through reasoning. If you cannot deduce after reasoning, you can use "un "unknown" for the category. Your response should be formatted in JSON.
Here are some examples:
Input: Query: this is brown cabinet. it to the right of picture.
Output:
"target_class": "cabinet"
22

---

Input: Query: it is wooden computer desk. the desk is in the sleeping across from the living room. the desk is in the corner of the room, between the nightstand and where the shelf and window are.
Output:
"target_class": "desk"
Now start your task: Input: (text)"
Table 8: View selection prompt for identifying the best 3D view to locate the target object. (target_classe is the predicted object class and " ((dd""teeaaa (text) denotes the query text.
Prompt Template for View Selection
You are good finding the object 3D scene based on given query description. These images show different views of room. You need to find the (target_clade class] in this query description: (text]) j."
Please review all view images to find the target object and select the view that you can see the target object most clearly.
Output your answer in JSON format with these keys:
"reasoning": "Explain how you identi fied the target object, and why you choose this view." "view": "2" // The number of the view is in the top left corner of the corresponding image.
Table 9: Candidate screening prompt for identifying the Top-k object IDs based on given query. { starget_cllle class) is the predicted object class and "[text ) denotes the query text. (n_topk) set to 4, (object_id_list contains the valid object IDs after anchor filtering.
Prompt Template for Candidate Screening
Here is the annotated image of the selected view. All objects belonging to the (target_ class) class labeled by unique number (ID) in red color them.
Please select the object ID that best matches the given query description: (text]" )
shape, color, relative position with Carefully analyze the specified conditions (such (n_ttpk) best-matched object IDs. surrounding objects) in the given query, then select top- cop-(n_tte
The selected top- (n (n_topp topk] } object IDs should be sorted in descending order of confidence. The object ID should be chosen from this list: (object id list)
Output your answer in JSON format with these keys:
"reasoning": "Explain how you identified and ranked the top- (n_took target object IDs.". " "object_id": [1, 2, 3, 4, 5] // A list of (n_topk) selected target object IDs.
23

---

Table 10: Input prompt for 3D-2D joint crriiiion-maiing.. (target_class] classs is the predicted object class and (text] denotes the query text. {ob jocte id list] contains the valid object IDs after anchor filtering.
Prompt Template for 3D-2D Joint Decision-Making
You are provided with set of images depicting an indoor scene:
global view image showing the room's 's 3D layout from a fixed perspective.
Several camera images captured from different viewpoints around the room.
All objects of interest in the scene labeled with unique object IDs (in red), which consistent across both the global and camera images.
Your task is to identify the object ID that best matches the given query description. Follow the steps below:
Start with the global view image:
Analyze the overall spatial layout and object distribution in the room.
Use the global view evaluate view-independent spatial relationships, which do not rely on specific viewpoint:
Examples include: near, close to, next to, far, above, below, under, on, top of, middle, opposite
2. Then examine the camera images:
Validate candidate objects identified from the global view.
Evaluate visual features: color, shape, size, texture, and material.
Use camera views to judge view-dependent spatial relationships, which depend on camera perspective:
Examples include:
left, right, in front of, behind, back, facing, looking, between, across from, leftmost, rightmost
Tip:
Annotations may not always be at the center of the object. Focus the full spatial extent and choose the that best represents the main body of the described object across both views.
3. Iterate if needed:
If no candidate fully matches the query, return to the global view and reassess alterna- tives.
Repeat verification with camera images until you confidently identify the best match.
Task:
Select the object ID of the target class: (target_class
Query description: (text)
Object IDs to choose from: (object_ioe id list}
Output format (JSON):
"reasoning"; "Explain how you analyzed spatial relation- ships (view-dependert vs view-independents cross-veri the object across views, and selected the best-matched ID.", "object_id"" // e.g.,
24

---

B Additional experimental results and analysis
Ours (3D only)
Ours (3D+2D)
4%
4%
14%
45%
57%
19%
24%
5%
12%
Spatial
Detection
Figure 5: Error type distribution on ScanRefer dataset. Ours (3D only) indicates that our agent selects the target object directly at the candidate screening stage, without performing the subsequent 3D-2D joint decision- maniing. Ours (3D+2D) represents the full pipeline of our SPAZER method.
B.1 Error type analysis
In Fig. 5 we present the distribution of different error types in SPAZER's predictions to provide
insights into potential directions for further improvement. The main error types include: 1) Semantic: Errors caused by the model overlooking critical semantic cues, such as color, shape, etc.; 2) Spatial: Errors where the model fails correctly interpret spatial relationships, including relative positions between objects or absolute directions (e.g., northwestern-mon moot);; 3) Detection: Cases where the 3D detector fails to detect the target object or predicts the wrong category; Other: Mainly due to referring ambiguities in the query text, where multiple objects in the scene could reasonably match the description based on human judgment.
Uni-modal and multi-modal. Compared to the 3D-only paradigm, incorporating 3D+2D sig- nificantly reduces errors in Semantic category, indicating that 2D images provide important supplementary semantic cues for the agent. Additionally, the reliable view-dependent relational information from 2D images also helps reduce the occurrence of Spatial errors.
Analysis and future work. Based the error type distribution of SPAZER (right side of Fig. Spatial errors account for the largest proportion, indicating that the primary challenge in 3DVG lies in understanding complex spatial relationships. To address this, we plan to further explore more effective 3D representations in future work. In addition, a considerable portion of errors is caused by the detector, suggesting that reducing the agent's reliance the detector is of the key issues to be addressed in future work.
Table 11: Cross-dataset evaluation on RIORefer. Both methods use the same detector trained on ScanRefer. SPAZER achieves superior zero-shot performance without any 3DVG-specific training.
Method
Training Data
Setting
Acc@:5 (%)
Acc@50 50 (%)
Cros 28
ScanRefer
Supervised
29.2
14.4
SPAZER (Ours)
Zero-shot
34.0
16.4
B.2 Additional evaluation on cross-dataset generalization
To further verify the generalization ability of our SPAZER framework, we conducted additional ex-
periments on the RIORefer benchmark (28) which is built upon 3RScan 44) and differs considerably from ScanNet
9 in terms of scene layout, object category distribution, and scanning conditions.
This benchmark is therefore suitable for evaluating cross-datasct robustness and zero-shot transfer
25

---

Table 12: Stage-wis, wise ablation on VLM capacity. We replace GPT-4o with GPT--4omiini one reasoning stage at a time to measure sensitivity.
Experiment
Stage 1: View Selection
Stage 2: Anchor Filtering
Stage 3: Final Reasoning
Overall Ace (%)
(SPAZER)
GPT-4o
GPT-4a GPT-4o
GPT-4o GPT-4o
63.8
B
GPT-4o-mini
60.4 (1.34)
GPT-4o
GPT-4o-miii
GPT-4o
57.2 ( (1.66)
GPT-4o
GPT-4o
GPT-40-mini
50.4 (4 13.4)
capability. this experiment, both SPAZER and the supervised baseline (Cross3DVG) adopt the same 3D detector trained on ScanRefer to ensure a fair comparison. Notably, Cross3DVG is trained in a supervised manner with language-object annotations from ScanRefer, whereas SPAZER operates in zero-shot setting without any 3DVC-specifie training.
As shown in Tab. TT SPAZER surpasses the supervised baseline even when evaluated across domains, demonstrating strong cross-dataset generalization and robustness to distribution shifts. This result highlights the adaptability of our framework, which effectively transfers knowledge from opon-world vision-languee models (VLMs) to unseen datasets without fine-tuning. We attribute SPAZER's s superior cross-dataset performance to its reliance on the VLM's broad vision-language prior, instead of dataset-speeifie supervision. This design enables SPAZER to understand diverse object appearances and handle natural- -aauual- queries robustly, even under unfamiliar scene conditions.
B.3 Stage-wis wise ablation on the effect of VLM capacity
To further analyze how SPAZER's performance depends on the reasoning capability of the Vision- Language Model (VLM) at different reasoning stages, we conducted stage-wise ablation experi- ments. Specifically, we replaced GPT-4o with a smaller model (GPT-4o-min -iiiiiini at one stage at a time, while keeping the others unchanged. The results are summarized in Table [2] As shown in Table [2] the final reasoning stage (Stage 3: 3D-2D joint decision-making) is the most sensitive to the VLM's capacity, showing notable 13.4% drop when GPT-4o is replaced with GPT-4o-minh This is expected, as this stage requires fine- -en-ggaieed spatial-semantic reasoning and cross-view understanding to accurately identify the target. The anchor filtering stage (Stage 2) experiences moderate degradation (4 6.6%). since its objective is to shortlist a set of likely candidates. Even with a weaker VLM, the correct anchor may still appear in the top-k shortlist. In contrast, the view selection stage (Stage 1) is relatively robust (1 3.4%). Because the same object can often be captured from multiple valid viewpoints, this stage places lower demands on fine-grained reasoning accuracy, making it less sensitive to VLM degradation. Overall, these results confirm that SPAZER follows a coarse-to- progressive reasoning paradigm. Early stages (view selection and anchor filtering) involve simpler reasoning steps and show higher tolerance to variations in VLM capability. In contrast, the final grounding stage requires precise spatial-semantee alignment and thus exhibits stronger dependency on the VLM's 's reasoning power.
B.4
Case study on implied object grounding
Our proposed SPAZER is capable of handling implied object grounding, where the referred tar- get is not explicitly mentioned but can be inferred from contextual cues. Unlike traditional text- matching- approaches that rely solely on literal word overlap between query text and object class labels, our method integrates visual-semantic reasoning within the VLM and an anchor filtering mechanism guided by the VLM--infereed target class. This design allows the model to interpret implied intents (e.g.. "something to drink") and associate them with relevant visual entities (e.g.. bottles, cups, or glasses) even when such categories are not directly stated in the text.
As illustrated in Fig. 6 given the query "I am thirsty. Please bring me something from my desk," the model first identifies the most informative view of the desk and then filters objects related to drinking, such as bottles or cups. Among them, object 37 (a bottle) consistently observed across multiple views and best matches the query semantics, while others (e.g., 66. 77, 1) are filtered out. This case study demonstrates that our reasoning-guided filtering effectively bridges explicit textual grounding and implicit semantic inference. By combining lightweight text-matching for efficient anchor pruning with VLM-based reasoning for category understanding, the model can infer the intended target even when the query describes an implied object. Consequently, it achieves robust performance on complex queries where conventional matching strategies fail to capture the latent semantics of user intent.
26

---

Query thirsty. Pleose bring something my desk."
Global Views
View Selection
query for comathiing from
providel clearly fhems making best identify target object."
Selected Viewr
Yiual Object Table
Condidate Screening
The query suggests looking for objects related drinking bettles, cupa, glessess ane quenching cups
desk, images, abject is Object appears mug glosses Object
which nost patentially
Object although clear"
20 Camera Images
Decision- Making
"Tee query suggests looking for bottle cup. In the global view image. object 37 is the doak and appears to be a bell matching the query. Conera view confirm that object 37 is considtuntly shown on the desk near object (a biptop). Other objects like 65 (prisner) and
(epttpp) do not match the descrip lpoiio. Therefore, object 37 the best match""
Figure 6: Case study on implied object grounding by SPAZER.
C Limitations and border impact
the chair is south of northwestern-men bookshelf. the chair is gray and has four legs.
this is a black chair. the back of the chair against the wall.
(a) Spatial relation
table, the white table is sitting behind the two brown desks the far left corner of the room.
the gray padded chair is en rollers it is the only rolling chair at the table
(b) Semantic detail
the chair is in one of the sides of the room. the color the chair is brown.
is a brown door, it is across from armchair.
(c) Other (referring ambiguity)
Figure 7: Typical types of failure cases. The prediction and ground-truth are highlighted in red and green, respectively. (a) Relation error includes relative relation (e.g., against the wall) and absolute relation (e.g., northwestern-mont as shape (on rollers), color (white), material, etc. (c) Other errors are primarily caused by the referring ambiguity, i.e., multiple objects in the scene satisfy the query.
(b) Semantic error mainly involves detailed object attributes, such
C.1 Analysis of failure cases
Based on Fig. [5] there are mainly four types of failure cases. Since we adopt a detect-and-me match paradigm similar to previous works [54] 22), detection-related errors are currently unavoidable. In future work, we plan to explore how to enable the agent to directly produce localization results. The remaining three types of errors are illustrated through case studies in Fig- Regarding spatial relations, failures mainly occur in: 1) complex positional relationships, which often involve both the orientation of the target object and its relation to surrounding objects; 2) directional
27

---

terms (e.g., south, northwest). In future research, we plan to incorporate visual prompts indicating orientation into the 3D representation.
For semantic details, when the scene contains multiple visually similar objects, the candidate screening stage may fail include the correct target into the Top-k list, preventing effective semantic verification in subsequent steps. This reflects limitation of our current method.
Lastly, we observed that some samples in the dataset exhibit referring ambiguity. As shown in Fig. (c) both the predicted result and the ground truth can satisfy the query description based on human interpretation. Therefore, the construction bigher-quality 3DVG datasets stands out as eritical challenge that needs to be addressed in future research.
C.2 Broader impact
Our VLM-driven agent SPAZER for 3D visual grounding offers potential benefits in areas such as human-rober interaction, augmented reality, and assistive technologies by enabling more intuitive object localization from language. However, it may also carry risks, such biases inherited from pre-trained models, which could affect performance in diverse environments. Future work should address these concerns through fairness-awan training, improved interpretability, and responsible deployment.
28


Reviews:
[{"review_id": "MVCHz9k98e", "reviewer": "Reviewer_8wRM", "summary": "The paper introduces SPAZER, a zero-shot 3D visual grounding agent designed to integrate 3D and 2D reasoning through a progressive reasoning framework. SPAZER first performs holistic 3D spatial analysis using multi-view BEV rendering, then employs a pre-trained 3D detector for coarse object localization, and finally uses 2D images for fine-grained semantic verification and joint decision-making. Experimental results on ScanRefer and Nr3D benchmarks show significant improvement over existing zero-shot methods.", "questions": "1. Can the authors explain how their framework addresses open-vocabulary tasks?\n\n2. Can the authors explain their decision to rely on text-matching anchor filtering instead of LLM-based approaches? Could an LLM-driven approach mitigate the current limitations?\n\n3. Can the authors provide examples and analyses of how their method outperforms existing methods in out-of-scannet scenarios?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. The authors introduces a novel framework which intergrates 3D and 2D reasoning, addressing a the gap in existing visual grounding agents. \n\n2. The performance on ScanRefer and Nr3D significantly outperforms prior zero-shot methods.\n\n3. The authors conduct detailed ablation studies, clearly demonstrating the contributions of each component.\n\nWeaknesses\n\n1. Although multiple views are rendered, they are essentially variants of BEV. This approach inherently fails in certain scenarios, like scenes with ceilings and multi-floor buildings, significantly limiting its applicability.\n\n2. This method heavily relies on a pre-trained 3D detector, fundamentally restricting the upper-bound capabilities. The 3D detector before VLM systematically harms its performance in open-vocabulary tasks. Furthermore, existing detectors like Mask3D have not claimed zero-shot abilities, conflicting with the zero-shot setting in this work.\n\n3. The anchor filtering mechanism is based on text matching algorithms rather than LLMs. Consequently, it cannot effectively identify implied object targets (e.g., \"I'm thirsty, please bring me something from my desk.\"). Although a switching mechanism is proposed, it merely acts as a remedial measure post-failure, not resolving the fundamental issue. \n\n4. The experiments are conducted on ScanNet-based benchmarks, which mitigates the above issues. I strongly recommend the authors to conduct broader evaluations in diverse scenarios.\n\n5. The authors appear not to have uploaded their supplementary material correctly.", "was_revised": true, "final_justification": "The experiments on out-of-scannet experiments strengthen the paper's claim and other supplement experiments addressed most of my concerns.", "rebuttal": "## W1: Applicability in scenes with ceilings and multi-floor buildings.\nWe thank the reviewer for highlighting this important point. \n\nFor scenarios with **ceilings**, our current method already addresses this by removing points above a height threshold relative to the scene’s maximum height. Specifically, we filter out the top 0.4 meters to eliminate ceiling points before rendering.\nWe find this strategy to be simple yet effective in our experiments (Acc@25: **57.2** w/ ceiling removal vs. **52.4** w/o).\n\nFor **multi-floor** scenarios, we acknowledge that our current implementation does not explicitly distinguish floor levels. However, the commonly used benchmarks (e.g., ScanRefer, Nr3D, RIORefer) predominantly feature single-floor indoor scenes, making this less of a concern in our current setting.\nAnd we believe our framework **can be extended** to better support multi-floor scenarios. For instance: a coarse floor segmentation based on height clustering or semantic segmentation could isolate floors; and the VLM could further leverage semantic cues from the query language (e.g., “second-floor desk”) to localize the relevant region. We appreciate the reviewer’s suggestion and consider this an important direction for future work.\n\n## W2: The adopted detector conflicts with the zero-shot setting.\nIt is correct that existing detectors like Mask3D are not designed for zero-shot detection. However, our setting focuses on zero-shot 3D visual grounding, which is a **fundamentally different task**: the goal is not to recognize object categories, but to identify the correct object in the 3D scene based on a free-form natural language query.\n\nImportantly, training a 3DVG model requires **language-object pair annotations** for learning 3D-language correlation. \nIn SPAZER, the detector is used only to generate candidate object locations; it is not responsible for understanding the language-object correspondence. Instead, all reasoning—from interpreting spatial and semantic cues in the query to selecting the best-matching object—is performed by the VLM in a zero-shot manner, without any 3DVG-specific training.\n\nThus, SPAZER remains fully consistent with the zero-shot 3DVG setting, in line with previous works such as ZSVG3D (24'CVPR) and SeeGround (25'CVPR), which also adopt pre-trained Mask3D. For fair comparison, we likewise use Mask3D, allowing a direct and equitable comparison of different models’ grounding reasoning capabilities.\n\n## W2 & Q1: How the proposed framework addresses open-vocabulary tasks.\nWe address open-vocabulary tasks mainly by leveraging VLM's ability, while utilizing a 3D detector to provide essential layout and spatial localization cues.\n\nSpecifically, we adopt the VLM to **interpret the query, reasons about object identity, and infers spatial relations** through rendered 3D views and 2D camera images. While the detector provides only closed-set object categories, our framework alleviates this limitation in two ways: (1) a matching algorithm is used to associate VLM-inferred target class with semantically similar detected categories; (2) even in the presence of noisy detections or implied objects, our retrieval-augmented matching strategy enables the VLM to directly assess each box based on visual cues without relying on the detector’s label.\n\nTherefore, SPAZER inherits the open-vocabulary capability directly from the VLM: it understands language, observes the scene and selects the best-matching object. \n\n## W3 & Q2: Text-matching anchor filtering cannot effectively identify implied object targets.\nFirst, we want to clarify that the process of identifying target object class and anchor filtering is primarily conducted by the VLM. The text matching algorithm only plays an **auxiliary role** in this process. It serves as a lightweight filtering strategy, attempting to match the VLM-inferred target class with the detector-provided class labels to help remove irrelevant objects during visual anchor annotation. In this way, the VLM can focus its reasoning on the most relevant regions of the scene.\n\nOf course, the matching step can be replaced with VLM-based reasoning, and we consider the following two cases:\n1. **VLM-matching**: Feed all detected object classes to the VLM, and it determines the matched class. It achieves similar results as text-matching. We believe it is because the matching step is relative easy and a simple text-matching algorithm can handle it effectively while being more efficient than using a VLM.\n2. **w/o matching**: Annotate all anchors (usually > 100) and VLM decides how to filter out irrelevant ones by itself. In this setting, the accuracy drops significantly due to the large amount of redundant information in the raw detection results. Providing the VLM with too many annotated anchors makes it difficult to focus on the key regions. \n\n| Filtering Strategy | Acc@25 (%) | Acc@50 (%) |\n|--------------------|------------|------------|\n| Ours               | **57.2**   | **48.8**   |\n| VLM-matching       | 56.4       | 48.4       |\n| w/o matching        | 38.4       | 32.8       |\n\nSo our design is an effective **combination of text-matching and VLM reasoning**: when the detector's predicted class aligns with the VLM-inferred class, we leverage the detector's class prior to filter out irrelevant objects, thereby greatly improving the reasoning efficiency of the VLM. On the other hand, when the detector's predictions are unreliable or the VLM struggles to infer the query object’s category, our RAF module enables the VLM to directly determine the class of each box based on its own visual observations, allowing it to recognize implied object targets.\n\n***Case study of identifying implied objects***:\n\n**Query Text**:  \n\"I am thirsty. Please bring me something from my desk.\"  \n\n**Scene**: ScanNet scene_0050_00  \n\n> **View Selection**  \n> \"The query asks for something to drink from the desk. In view 4, the desk is clearly visible, and there is **a bottle on it which could be a drink**. This view provides the clearest view of the desk and the items on it, making it the best choice to identify the target object.\"\n\n> **Candidate Screening**  \n> \"The query suggests looking for objects related to drinking, **such as bottles, cups, or glasses, that are located on the desk**. In the images, object 37 is a water bottle, which is the most direct match for quenching thirst. Object 66 appears to be a mug, which is also suitable for drinking. Object 60 and 77 seem to be cups or glasses. Object 1 is on the desk and could potentially be a container, although less clear.\"\n\n> **Joint Decision-Making**  \n> \"The query suggests looking for a bottle or cup.  In the global view image, **object 37 is on the desk and appears to be a bottle**, matching the query. Camera views confirm that **object 37 is consistently shown on the desk near object 1 (a laptop)**. Other objects like 66 (printer) and 1 (laptop) do not match the description. Therefore, **object 37 is the best match**.\"\n\n**GT**: object 37 (a bottle of water on the desk)\n\nWe sincerely apologize for not being able to include more intuitive visual illustrations in the current submission. We will include clearer visualizations and examples in the revised version to facilitate better understanding of our framework in handling implied objects.\n\n## W4 & Q3: Broader evaluations in out-of-scannet scenarios.\nTo address this concern, we conducted additional experiments on the RIORefer [1] benchmark, which is based on 3RScan [2] and differs significantly from ScanNet in terms of scene layouts, object types, and scanning conditions. \n\nIn this setting, both methods use the same detector trained on ScanNet. Besides, Cross3DVG is trained on ScanRefer using language-object annotations, while our SPAZER requires no 3DVG training. \n\n| Method         | Training Data | Setting     | Acc@25 (%) | Acc@50 (%) |\n|----------------|---------------|-------------|--------------|-------------|\n| Cross3DVG      | ScanRefer     | Supervised  | 29.2         | 14.4        |\n| **SPAZER (Ours)** | -             | Zero-shot   | **34.0**     | **16.4**    |\n\nHere, SPAZER outperforms the supervised baseline even across domains, indicating strong cross-dataset generalization and robustness to distribution shifts. We will include these results and corresponding discussion in the revised manuscript.\n\nWe believe SPAZER performs well in cross-dataset scenarios because it relies on VLM's broad visual-language knowledge, rather than dataset-specific labels or training. This allows it to understand diverse object appearances and natural language queries, even in unfamiliar environments.\n\n## W5: Regarding the supplementary material.\nWe sincerely thank the reviewer for pointing this out. The supplementary material was indeed prepared but appears to have been unintentionally omitted during submission. It contains the following sections:\n\n- **A. Implementation Details**  \n  - Computational resources  \n  - Model configuration and prompt design\n\n- **B. Additional Experimental Results and Analysis**  \n  - **B.1**: Error Type Analysis  \n    *Spatial relation reasoning remains the major challenge.*  \n  - **B.2**: Performance on Subset vs. Full Dataset  \n    *Our method and prior works exhibit consistent trends across two dataset partitions.*  \n  - **B.3**: Inference Time  \n    *SPAZER achieves ~2× faster inference compared to VLM-Grounder.*\n\n- **C. Limitations and Broader Impact**  \n  - **C.1**: Failure Case Analysis  \n  - **C.2**: Broader Impact Discussion\n\nWe appreciate the opportunity to clarify these points and will revise both the main paper and supplementary material to address all reviewer feedback. The complete supplementary will be properly included in the next version.\n\n[1] Miyanishi, Taiki, et al. \"Cross3dvg: Cross-dataset 3d visual grounding on different rgb-d scans.\" 3DV. 2024.\n\n[2] Wald, Johanna, et al. \"Rio: 3d object instance re-localization in changing indoor environments.\" ICCV. 2019."}, {"review_id": "sAiU82NmwE", "reviewer": "Reviewer_2LyA", "summary": "This paper presents SPAZER, a novel agent for zero-shot 3D visual grounding that integrates both spatial and semantic reasoning in a progressive framework.  SPAZER leverages a Vision-Language Model (VLM) to jointly reason over both modalities without 3D-specific training. The agent first performs a holistic analysis of the 3D scene by rendering multiple global views. Subsequently, it refines object identification through joint decision-making. SPAZER significantly outperforms existing zero-shot methods.", "questions": "1.  How sensitive is SPAZER’s performance to the quality and domain of these detectors? Can the authors provide quantitative analysis or qualitative discussion on the impact of noisy or misaligned detections?\n2. The SPAZER framework involves multi-view 3D rendering, candidate screening, and multi-modal (3D-2D) reasoning, which may introduce computational and memory overhead. Can the authors clarify the computational cost?\n3. Does the evaluation conduct on the whole val set or a subset?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The paper proposes a well-motivated, coherent framework that effectively integrates both 3D spatial reasoning and 2D semantic verification through a progressive multi-stage pipeline.\n2. The authors provide extensive quantitative and qualitative experiments on two widely-used benchmarks (ScanRefer and Nr3D), consistently demonstrating superior performance over both zero-shot and supervised baselines.\n# Weaknesses\n1. The approach still relies on the accuracy of off-the-shelf 3D object detection models to extract bounding boxes and class labels. Errors in these components may propagate through the pipeline.\n2. The performance largely depend on the ablility of VLMs (Qwen 56 Acc vs 4o 63.8 Acc)", "was_revised": true, "final_justification": "The author has addressed my concern. Please include this content in the main manuscript, especially the full set evaluation.", "rebuttal": "## W1 & Q1: Analysis on the impact of detection errors. \nWe thank the reviewer for highlighting the importance of robustness to detection results. We designed a series of experiments to systematically analyze the impact of two common types of detection errors: (1) Bounding box category noise, and (2) Bounding box misalignment (position and size perturbation).\n\n**(1) Robustness to category noise**\n\nThanks to the design of our Retrieval-augmented anchor filtering (RAF) module, SPAZER is able to reason over the visual appearance of each detected object instead of relying solely on the predicted class name. As shown in our experiments (see Table below), our method demonstrates **stronger robustness over category noise**—particularly under the extreme case where all predicted class labels are removed, SPAZER still maintains a reasonable level of accuracy. In contrast, previous methods that depend text-based 3D representations struggle or completely fail in such noisy settings due to their heavy dependence on accurate class labels.\n\n| Corruption Ratio     | SPAZER         | SeeGround         |\n|----------------------|----------------|-------------------|\n|                      | Acc@50         | Acc@50            |\n| 0% (clean)           | 48.8           | 37.6              |\n| 10%                  | 45.6           | 32.8              |\n| 20%                  | 39.6           | 28.4              |\n| 50%                  | 34.4           | 17.2              |\n| 100% (no cls label)  | 32.8           | 6.4               |\n\n**(2) Sensitivity to geometric perturbation**\n\nTo assess the robustness of SPAZER to geometric misalignment, we simulate misaligned detection boxes by injecting Gaussian noise into both the position and size of each bounding box. The table below shows a clear performance drop as the perturbation strength increases.\nWe acknowledge that SPAZER currently adopts detector-predicted 3D bounding boxes directly as anchors—this design choice stems from the fact that **current VLMs lack the ability to directly predict 3D bounding boxes**. As a result, SPAZER is naturally more sensitive to geometric perturbations. This limitation is shared with prior works (e.g., CSVG3D, SeeGround) that follow a similar paradigm, and we view improving robustness to box localization noise as a promising direction for future development.\n\n| (position, size) | Acc@25  | Acc@50  |\n|------------------|---------|---------|\n| (0.00, 0.00)     | 57.2    | 48.8    |\n| (0.05, 0.02)     | 48.0    | 35.6    |\n| (0.10, 0.05)     | 40.8    | 19.2    |\n\nWe believe this is an important limitation and a valuable direction for future research, as stated in our paper (Sec. 5).  However, our method takes an **important step forward by explicitly addressing the robustness to noisy object class predictions**—an issue that previous works have largely overlooked.\nIn the future work, we aim to explore more autonomous grounding approaches, possibly incorporating self-generated or self-corrected 3D anchors to reduce dependence on external detectors.\n\n## Q2: The computational cost of SPAZER. \nIn Table below, we break down the inference time of each step in SPAZER. \n\n| Method            | Step                             | Time (s) | Total (s) |\n|-------------------|----------------------------------|----------|-----------|\n| **Ours (SPAZER)** | 3D Holistic View Selection       | 5.2      |           |\n|                   | Candidate Object Screening       | 8.5      | **23.5**  |\n|                   | 3D–2D Joint Decision-Making      | 9.8      |  |\n| VLM-Grounder | -                                | -        | **50.3**  |\n\nThe time consumption across different steps is relatively balanced, with the view selection step being faster since it requires no additional computation. Moreover, compared to VLM-Grounder, which also leverages 2D camera images for reasoning, our method achieves significantly higher inference efficiency. This is because we rely on VLM-selected anchors and require only a small number of images, avoiding the need to sample and filter all video frames.\n\n## Q3: Evaluation on the whole val set or a subset?\nIn consideration of budget and evaluation efficiency, we follow previous work VLM-Grounder to\nevaluate our agent on the **subset** (250 selected samples) of each dataset. To further verify whether the results on the subset are comparable to those on the whole val dataset, we conduct additional experiments using open-source VLMs, as shown in the table below. We observe that both our method and prior work SeeGround exhibit consistent performance across the two dataset partitions, with overall accuracy variations **< 2.0**, which is negligible compared to the improvement achieved by our method. This confirms the validity of using the subset for fair and meaningful comparison.\n\n| Dataset | Method             | VLM              | Easy | Hard | Dep. | Indep. | Overall |\n|---------|--------------------|------------------|------|------|------|--------|---------|\n| Whole val set    | SeeGround     | Qwen2-VL-72B     | 54.5 | 38.3 | 42.3 | 48.2   | 46.1    |\n|         | Ours               | Qwen2-VL-72B     | 59.3 | 44.8 | 46.3 | 54.9   | 51.8    |\n|         | Ours               | Qwen2.5-VL-72B   | 62.4 | 46.9 | 49.9 | 56.8   | 54.3    |\n| Subset  | SeeGround     | Qwen2-VL-72B     | 51.5 | 37.7 | 44.8 | 45.5   | 45.2    |\n|         | Ours               | Qwen2-VL-72B     | 62.5 | 43.0 | 51.0 | 55.2   | 53.6    |\n|         | Ours               | Qwen2.5-VL-72B   | 60.3 | 50.9 | 54.2 | 57.1   | 56.0    |\n\n## W2: The performance largely depends on VLMs. \nSince the VLM serves as the \"brain\" of SPAZER—responsible for understanding the query text, reasoning about spatial relations, and inferring object attributes—we believe it is reasonable that the overall performance depends on the capability of the underlying VLM.\n\nHowever, even when using a relatively less powerful VLM like Qwen2-VL-72B, our method still outperforms prior approaches such as SeeGround that use the same VLM. This suggests that our spatial-semantic progressive reasoning pipeline contributes significantly beyond the choice of VLM itself.\n\nFurthermore, our framework is designed to be modular and model-agnostic, allowing easy substitution of the VLM component. As newer and stronger VLMs continue to emerge, our method can naturally benefit from these advances without requiring any task-specific training or architecture changes, making it **convenient to adapt in future deployments**."}, {"review_id": "LTeJshu7LW", "reviewer": "Reviewer_e7pa", "summary": "This paper proposes a new zero-shot 3D visual grounding technique, titled SPAZER, which leverages VLMs to output in text the 3D bounding box coordinates of the object which is referred in the language prompt. To achieve this, the method first identifies the set of candidate objects from an optimally selected 3D view of the scene. From this 3D view and from a set of selected 2D images that clearly depict the candidates, the method then selects and localizes the referred object. This allows joint reasoning about the 3D configuration of objects in the scene and fine-grained semantic cues, which are both crucial for the task at hand. The authors validate SPAZER on the two major 3D visual grounding benchmarks, i.e. ScanRefer and Nr3D, showing top performance among zero-shot methods that do not train explicitly on 3D data.", "questions": "1. Can the authors provide some efficiency comparison of the method to competing approaches? Relevant quantities are latency, parameter efficiency, and FLOPS.\n\n2. Can the authors provide a more in-depth ablation and examination of the effect of the particular VLM which is employed by their method on the visual grounding performance? Is the method more vulnerable / more robust to VLM failures in the 3D viewpoint selection stage, the candidate selection stage, or the final grounding stage?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The examined problem and setting is important, as natural language needs to be integrated with the 3D environment for better understanding in embodied applications. Also, the zero-shot setting is relevant and interesting.\n\n2. The proposed pipeline of SPAZER is novel and constitutes a creative utilization of state-of-the-art VLMs to marry them with traditional visual recognition models.\n\n3. The experimental comparisons are comprehensive and thorough and they demonstrate a state-of-the-art performance in 3D visual grounding. The method is also ablated in detail.\n\nWeaknesses:\n\n1. No comparison or analysis is conducted on the computational efficiency of the method. It is not clear whether the multi-step pipeline of SPAZER, which i.a. includes at least 3 successive evaluations of a bulky VLM such as GPT 4o, still has comparable latency to competing VLM-based approaches or to traditional supervised methods, some of which are more accurate than SPAZER.\n\n2. While the examined task resides in 3D, the proposed method only operates on 2D views. This has been the choice of previous works too, but I still wonder whether the explicit reasoning about 3D from these 3D views, e.g. via estimating their depth maps, could easily lead to better localization of 3D bounding boxes, as this technique has proven effective in the related context of 3D object detection from cameras.\n\n3. The obtained results largely depend on the performance of the vision-language model which is used. For example, in Fig. 4, performance degrades for more than 4 views, even though for more views the information presented to the model is more and can in principle lead to better results. Is this performance deterioration for more generated views a limitation of the employed VLM or a fundamental limitation of the proposed method?\n\n4. While the related work is surveyed quite well, there are still important works which are missing, especially from the traditional, supervised 3D visual grounding line, e.g. [A], [B], [C] - the latter work in particular also employs multi-view 2D images from 3D for visual grounding. The authors should include a few additional works in the literature.\n\n[A] Text-guided graph neural networks for referring 3D instance segmentation. In AAAI, 2021.\n\n[B] Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding. In ECCV, 2024.\n\n[C] Multi3Drefer: Grounding text description to multiple 3d objects. In ICCV, 2023.", "was_revised": true, "final_justification": "The paper examines an important problem in a highly relevant, zero-shot setting. The proposed pipeline is novel and constitutes a creative utilization of state-of-the-art VLMs to marry them with traditional visual recognition models for 3D grounding. Finally, the experimental comparisons are comprehensive and thorough and they demonstrate a state-of-the-art performance in the task at hand. The rebuttal by the authors has addressed my concerns with respect to efficiency comparisons, effect of selected VLM on the performance of the pipeline, and missing related works, which is why I have increased my rating to 5: Accept.", "rebuttal": "## W1 & Q1: The computational efficiency of SPAZER.\nWe thank the reviewer for the suggestion. In the Table below, we report the inference time (latency) of each component in SPAZER and compare it against VLM-Grounder, which also uses GPT-4o as the VLM. Our method demonstrates significantly better inference efficiency (23.5s vs. 50.3s). Since we identify candidate objects using 3D global views, we only require **a small number (k) of informative camera images** for further reasoning, thus avoiding the need to exhaustively sample and filter all video frames.\n\n| Method                   | Step                                             | Time (s) | Total (s) |\n|---------------------------|--------------------------------------------|-------------|-------------|\n| **Ours (SPAZER)** | 3D Holistic View Selection          | 5.2         |               |\n|                                 | Candidate Object Screening       | 8.5         |               |\n|                                 | 3D–2D Joint Decision-Making     | 9.8         | **23.5**  |\n| VLM-Grounder        | -                                                   | -             | **50.3**  |\n\nNote that the cost of 3D rendering and 3D-to-camera projection is almost negligible compared to the VLM inference, which dominates the total runtime. Since SPAZER is entirely training-free, it **does not introduce any additional trainable parameters** beyond the underlying VLM. Therefore, parameter efficiency is primarily determined by the VLM used (e.g., GPT-4o), which is shared across compared methods. \n\n## W2: Whether explicit 3D reasoning like depth estimation from 2D views could lead to better localization.\nWe agree that incorporating explicit 3D reasoning—such as depth estimation or geometry-aware modules—has the potential to further improve localization accuracy, especially in complex or cluttered environments.\n\nBut our current method is designed with a **training-free, zero-shot setting** in mind, following prior works like SeeGround and ZSVG3D. To ensure fair comparison, we avoid introducing additional modules that require supervised fine-tuning.\n\nIn fact, **our framework inherently possesses 3D awareness**. It draws inspiration from how humans typically perform 3D visual grounding during annotation—by interactively navigating 3D scenes through rotation and multi-view observation (e.g., in tools like MeshLab or Open3D) until they can confidently localize the object described by language. This motivated us to design a multi-view rendering and reasoning pipeline that enables the VLM to aggregate spatial and semantic cues from multiple perspectives. Although we do not explicitly estimate depth, this design equips the model with a degree of implicit 3D awareness, allowing it to reason about visibility, relative positioning, and occlusion.\n\nIn future work, we are very interested in integrating explicit geometry cues, such as estimated depth maps or surface normals, to further enhance the 3D reasoning capability.\n\n## W3: Why performance deterioration for more generated views.\nAs shown in Fig. 4(b) of the manuscript, SPAZER achieves the best performance when using n = 4 views, while a slight performance drop is observed as the number of views increases beyond that. We believe this phenomenon is arises from the **viewpoint quality** instead of the number of views.\n\nWhen n = 4, the selected views are rendered from **top-oblique** angles around the room, offering broad and minimally occluded observations of the scene. However, as n increases, additional views are often captured from lower or corner angles, which tend to suffer from occlusions caused by walls, furniture, or self-occluding objects.\n\nTo further investigate this, we conducted a **controlled experiment**: even with n=4, when all views were rendered specifically from the four room corners—i.e., the additional views introduced when increasing from n=4 to n=8—performance dropped to **~58.4% Acc@50**, compared to 63.8% using the top-oblique setting. These results highlight that spatial understanding is more influenced by view quality than by the number of views. Setting a large value of n may introduce too many low-quality or occluded views, which can overwhelm the model and lead to performance degradation.\n\n| View Setting           | Acc@50 (%) |\n|------------------------|-------------|\n| 2 views                | 58.0        |\n| 4 views (Top-oblique)  | **63.8**    |\n| 4 views (Room-corner)  | 58.4        |\n| 8 views                | 59.2        |\n\nSince this stage only requires the VLM to perform **coarse reasoning based on global views**, and high-quality close-up camera images will be incorporated in the subsequent stages, we believe introducing an excessive number of views is also unnecessary at this point.\n\nWe will include this additional analysis and ablation in the revised version.\n\n## Q2: More in-depth ablation of the effect of VLMs and sensitivity at different reasoning stages.\nTo analyze how SPAZER’s performance depends on the reasoning capacity of the VLM at different stages, we conducted **stage-wise ablation experiments**, replacing GPT-4o with a smaller model (GPT-4o-mini) at one stage at a time. The results are summarized below:\n\n| Experiment | Stage 1: View Selection | Stage 2: Anchor Filtering | Stage 3: Final Reasoning | Overall Acc (%) |\n|------------|-------------|--------------|--------------|--------------|\n| A (SPAZER) | GPT-4o      | GPT-4o       | GPT-4o       | 63.8         |\n| B          | GPT-4o-mini | GPT-4o       | GPT-4o       | 60.4 (↓ 3.4) |\n| C          | GPT-4o      | GPT-4o-mini  | GPT-4o       | 57.2 (↓ 6.6) |\n| D          | GPT-4o      | GPT-4o       | GPT-4o-mini  | 50.4 (↓ 13.4)|\n\nAs the results show:\n- The final stage (Stage 3: 3D–2D joint decision-making) is the most sensitive to VLM capability, with a **13.4%** drop when downgraded. This is expected, as this stage requires the VLM to reason across multiple views and resolve fine-grained spatial-semantic ambiguities to identify the target.\n- The anchor filtering stage (Stage 2) shows moderate degradation (**↓ 6.6%**), since its goal is to shortlist a set of likely candidates. Even under partial VLM failure, the target may still appear in the top-k shortlist.\n- The view selection stage (Stage 1) is relatively robust (**↓ 3.4%**). Since the same object can typically be observed from multiple valid perspectives, this stage imposes relatively lower demands on the VLM's fine-grained reasoning ability, making it less sensitive to moderate variations in VLM performance.\n\nThese results confirm that our framework follows a **coarse-to-fine progressive reasoning** paradigm. **Early stages** like view selection and anchor filtering involve relatively simpler tasks and thus exhibit higher tolerance to VLM degradation. In contrast, the **final grounding stage** requires fine-grained spatial-semantic reasoning, making it more sensitive to the VLM's capacity.\n\nWe thank the reviewer for the valuable suggestion, and we will include this analysis and ablation in the next version.\n\n## W4: Inadequate related work.\nWe thank the reviewer for highlighting this point and for recommending these valuable references.\nWhile we aimed to provide a comprehensive overview of the related literature, we acknowledge the omission of several important works from the supervised 3D visual grounding line. In particular, the suggested works:\n\n[A] Text-guided Graph Neural Networks for Referring 3D Instance Segmentation (AAAI 2021),\n\n[B] Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding (ECCV 2024), and\n\n[C] Multi3DRefer: Grounding Text Description to Multiple 3D Objects (ICCV 2023)\n\nare highly relevant to our task and provide complementary perspectives, including graph-based reasoning and multi-view fusion.\nWe will incorporate these references and revise the related work section to better position our method in the context of prior supervised approaches."}]

Author rebuttal:

Meta-review: This paper proposes a new VLM-driven agent SPAZER, by integrating both spatial and semantic reasoning for robust zero-shot grounding without 3D-specific training. Through holistic scene analysis and joint 3D-2D decision-making, SPAZER surpasses prior state-of-the-art methods on benchmark datasets, achieving accuracy improvements.

The main strengths of the work: The work introduces a novel framework that effectively integrates 3D and 2D reasoning for visual grounding, yielding state-of-the-art results on key benchmarks. It is well-motivated with comprehensive experiments.

The detailed rebuttal effectively addresses the reviewers' major concerns, leading to the final decision to accept the work.
Decision: Accept (poster)