Title: SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Visual Grounding

Multi-modal Agent

Zero-shot
Abstract: 3D Visual Grounding (3DVG) aims to localize target objects within a 3D scene based on natural language queries. To alleviate the reliance on costly 3D training data, recent studies have explored zero-shot 3DVG by leveraging the extensive knowledge and powerful reasoning capabilities of pre-trained LLMs and VLMs. However, existing paradigms tend to emphasize either spatial (3D-based) or semantic (2D-based) understanding, limiting their effectiveness in complex real-world applications. In this work, we introduce SPAZER — a VLM-driven agent that combines both modalities in a progressive reasoning framework. It first holistically analyzes the scene and produces a 3D rendering from the optimal  viewpoint. Based on this, anchor-guided candidate screening is conducted to perform a coarse-level localization of potential objects. Furthermore, leveraging retrieved relevant 2D camera images, 3D-2D joint decision-making is efficiently performed to determine the best-matching object. By bridging spatial and semantic reasoning neural streams, SPAZER achieves robust zero-shot grounding without training on 3D-labeled data. Extensive experiments on ScanRefer and Nr3D benchmarks demonstrate that SPAZER significantly outperforms previous state-of-the-art zero-shot methods, achieving notable gains of $\mathbf{9.0\}$% and $\mathbf{10.9\}$% in accuracy.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new zero-shot 3D visual grounding technique, titled SPAZER, which leverages VLMs to output in text the 3D bounding box coordinates of the object which is referred in the language prompt. To achieve this, the method first identifies the set of candidate objects from an optimally selected 3D view of the scene. From this 3D view and from a set of selected 2D images that clearly depict the candidates, the method then selects and localizes the referred object. This allows joint reasoning about the 3D configuration of objects in the scene and fine-grained semantic cues, which are both crucial for the task at hand. The authors validate SPAZER on the two major 3D visual grounding benchmarks, i.e. ScanRefer and Nr3D, showing top performance among zero-shot methods that do not train explicitly on 3D data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The examined problem and setting is important, as natural language needs to be integrated with the 3D environment for better understanding in embodied applications. Also, the zero-shot setting is relevant and interesting.\n\n2. The proposed pipeline of SPAZER is novel and constitutes a creative utilization of state-of-the-art VLMs to marry them with traditional visual recognition models.\n\n3. The experimental comparisons are comprehensive and thorough and they demonstrate a state-of-the-art performance in 3D visual grounding. The method is also ablated in detail.\n\nWeaknesses:\n\n1. No comparison or analysis is conducted on the computational efficiency of the method. It is not clear whether the multi-step pipeline of SPAZER, which i.a. includes at least 3 successive evaluations of a bulky VLM such as GPT 4o, still has comparable latency to competing VLM-based approaches or to traditional supervised methods, some of which are more accurate than SPAZER.\n\n2. While the examined task resides in 3D, the proposed method only operates on 2D views. This has been the choice of previous works too, but I still wonder whether the explicit reasoning about 3D from these 3D views, e.g. via estimating their depth maps, could easily lead to better localization of 3D bounding boxes, as this technique has proven effective in the related context of 3D object detection from cameras.\n\n3. The obtained results largely depend on the performance of the vision-language model which is used. For example, in Fig. 4, performance degrades for more than 4 views, even though for more views the information presented to the model is more and can in principle lead to better results. Is this performance deterioration for more generated views a limitation of the employed VLM or a fundamental limitation of the proposed method?\n\n4. While the related work is surveyed quite well, there are still important works which are missing, especially from the traditional, supervised 3D visual grounding line, e.g. [A], [B], [C] - the latter work in particular also employs multi-view 2D images from 3D for visual grounding. The authors should include a few additional works in the literature.\n\n[A] Text-guided graph neural networks for referring 3D instance segmentation. In AAAI, 2021.\n\n[B] Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding. In ECCV, 2024.\n\n[C] Multi3Drefer: Grounding text description to multiple 3d objects. In ICCV, 2023.\n\nQUESTIONS:\n1. Can the authors provide some efficiency comparison of the method to competing approaches? Relevant quantities are latency, parameter efficiency, and FLOPS.\n\n2. Can the authors provide a more in-depth ablation and examination of the effect of the particular VLM which is employed by their method on the visual grounding performance? Is the method more vulnerable / more robust to VLM failures in the 3D viewpoint selection stage, the candidate selection stage, or the final grounding stage?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper presents SPAZER, a novel agent for zero-shot 3D visual grounding that integrates both spatial and semantic reasoning in a progressive framework.  SPAZER leverages a Vision-Language Model (VLM) to jointly reason over both modalities without 3D-specific training. The agent first performs a holistic analysis of the 3D scene by rendering multiple global views. Subsequently, it refines object identification through joint decision-making. SPAZER significantly outperforms existing zero-shot methods.

STRENGTHS AND WEAKNESSES:
# Strengths
1. The paper proposes a well-motivated, coherent framework that effectively integrates both 3D spatial reasoning and 2D semantic verification through a progressive multi-stage pipeline.
2. The authors provide extensive quantitative and qualitative experiments on two widely-used benchmarks (ScanRefer and Nr3D), consistently demonstrating superior performance over both zero-shot and supervised baselines.
# Weaknesses
1. The approach still relies on the accuracy of off-the-shelf 3D object detection models to extract bounding boxes and class labels. Errors in these components may propagate through the pipeline.
2. The performance largely depend on the ablility of VLMs (Qwen 56 Acc vs 4o 63.8 Acc)

QUESTIONS:
1.  How sensitive is SPAZER’s performance to the quality and domain of these detectors? Can the authors provide quantitative analysis or qualitative discussion on the impact of noisy or misaligned detections?
2. The SPAZER framework involves multi-view 3D rendering, candidate screening, and multi-modal (3D-2D) reasoning, which may introduce computational and memory overhead. Can the authors clarify the computational cost?
3. Does the evaluation conduct on the whole val set or a subset?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SPAZER, a zero-shot 3D visual grounding agent designed to integrate 3D and 2D reasoning through a progressive reasoning framework. SPAZER first performs holistic 3D spatial analysis using multi-view BEV rendering, then employs a pre-trained 3D detector for coarse object localization, and finally uses 2D images for fine-grained semantic verification and joint decision-making. Experimental results on ScanRefer and Nr3D benchmarks show significant improvement over existing zero-shot methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The authors introduces a novel framework which intergrates 3D and 2D reasoning, addressing a the gap in existing visual grounding agents. \n\n2. The performance on ScanRefer and Nr3D significantly outperforms prior zero-shot methods.\n\n3. The authors conduct detailed ablation studies, clearly demonstrating the contributions of each component.\n\nWeaknesses\n\n1. Although multiple views are rendered, they are essentially variants of BEV. This approach inherently fails in certain scenarios, like scenes with ceilings and multi-floor buildings, significantly limiting its applicability.\n\n2. This method heavily relies on a pre-trained 3D detector, fundamentally restricting the upper-bound capabilities. The 3D detector before VLM systematically harms its performance in open-vocabulary tasks. Furthermore, existing detectors like Mask3D have not claimed zero-shot abilities, conflicting with the zero-shot setting in this work.\n\n3. The anchor filtering mechanism is based on text matching algorithms rather than LLMs. Consequently, it cannot effectively identify implied object targets (e.g., \"I'm thirsty, please bring me something from my desk.\"). Although a switching mechanism is proposed, it merely acts as a remedial measure post-failure, not resolving the fundamental issue. \n\n4. The experiments are conducted on ScanNet-based benchmarks, which mitigates the above issues. I strongly recommend the authors to conduct broader evaluations in diverse scenarios.\n\n5. The authors appear not to have uploaded their supplementary material correctly.\n\nQUESTIONS:\n1. Can the authors explain how their framework addresses open-vocabulary tasks?\n\n2. Can the authors explain their decision to rely on text-matching anchor filtering instead of LLM-based approaches? Could an LLM-driven approach mitigate the current limitations?\n\n3. Can the authors provide examples and analyses of how their method outperforms existing methods in out-of-scannet scenarios?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}