Promptable 3-D Object Localization with Latent Diffusion Models
Authors: Cheng-Yao Hong
Li-Heng Wang
Tyng-Luh Liu
Venue: NeurIPS 2025 poster ()
Keywords: 3-D Object Detection
Diffusion Model
Primary area: applications

Abstract: Accurate identification and localization of objects in 3-D scenes are essential for advancing comprehensive 3-D scene understanding. Although diffusion models have demonstrated impressive capabilities across a broad spectrum of computer vision tasks, their potential in both 2-D and 3-D object detection remains underexplored. Existing approaches typically formulate detection as a ''noise-to-box'' process, but they rely heavily on direct coordinate regression, which limits adaptability for more advanced tasks such as grounding-based object detection. To overcome these challenges, we propose a promptable 3-D object recognition framework, which introduces a diffusion-based paradigm for flexible and conditionally guided 3-D object detection. Our approach encodes bounding boxes into latent representations and employs latent diffusion models to realize a ''promptable noise-to-box'' transformation. This formulation enables the refinement of standard 3-D object detection using textual prompts, such as class labels. Moreover, it naturally extends to grounding object detection through conditioning on natural language descriptions, and generalizes effectively to few-shot learning by incorporating annotated exemplars as visual prompts. We conduct thorough evaluations on three key 3-D object recognition tasks: general 3-D object detection, few-shot detection, and grounding-based detection. Experimental results demonstrate that our framework achieves competitive performance relative to state-of-the-art methods, validating its effectiveness, versatility, and broad applicability in 3-D computer vision.

Promptable 3-D Object Localization with Latent Diffusion Models
Cheng-Yas Hong'
Li-Heng Wang
Tyng-Luh Liu'
"Institute of Information Science, Academia Sinica UUniversity Southern California
fsensible, liheng, liutyng)@iis sinica.edu...
Abstract
Accurate identification and localization of objects in 3-D scenes are essential for ad- vancing comprehensive 3-D scene understanding, Although diffusion models have demonstrated impressive capabilities across a broad spectrum of computer vision tasks, their potential in both 2-D and 3-D object detection remains underexplored. Existing approaches typically formulate detection they rely heavily on direct coordinate regression, which limits adaptability for more advanced tasks such as grounding-based object detection. To overcome these challenges, we propose promptable 3-D object recognition framework, which introduces diffusion-basee paradigm for flexible and conditionally guided 3-D object detection. Our approach encodes bounding boxes into latent representa- tions and employs latent diffusion models realize "promptable noise-to-box" transformation. This formulation enables the refinement standard 3-D object detection using textual prompts, such as class labels. Morcover, it naturally extends to grounding object detection through conditioning on natural language descrip- tions, and generalizes effectively few-shot learning by incorporating annotated exemplars as visual prompts. We conduct thorough evaluations on three key 3-D object recognition tasks: general 3-D object detection, few-shot detection, and grounding-basee detection. Experimental results demonstrate that our framework achieves competitive performance relative to state-of-the -f-te--r--- methods, validating its effectiveness, versatility, and broad applicability in 3-D computer vision.
"noise-to-box" process, but
Introduction
Precise identification and accurate localization of objects constitute foundational tasks critical for advancing the interpretation and analysis of visual data within computer vision. While numerous methodologies [54, 55] have achieved remarkable outcomes in traditional 2-D settings, exemplified by precise and efficient real-time detection in image domains, these approaches have increasingly encountered performance platcaus. contrast, object detection within 3-D environments introduces substantial complexity, posing additional challenges that render direct adaptation of successful 2-D methodologies inadequate for handling the intricate dynamics of 3-D data.
Recent advancements have leveraged diffusion- models, showcasing considerable promise in refining and enhancing solutions conventional computer vision tasks. In particular, object detection has benefited from these developments, as evidenced by the pioneering work of DiffusionDet [7]. which introduced the concept of diffusion processes "noise-to-box" transformations within de- tection frameworks. Subsequent extensions have successfully applied analogous diffusion concepts 3-D object detection, validating the versatility of diffusion methods across both 2-D and 3-D modalities. Nevertheless, prevailing diffusion seafussio--bsedd detection methodologies predominantly employ the classical diffusion process, directly predicting the target outputs rather than noise distributions, and operate entirely within the original feature space. Consequently, such methods exhibit limited
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(s) Prompl: Calegury Names
(b) Pronpl: Annotated Objects
Prompt: Reforring Expression
Countor: Gartonn
The desk s loatted
berk Ortinet, Sink
Sofo, befrigoretor
Prompli
Prompt
Condition
Category Names (bext)
Prompii
Ansotated Objects (text and annotated points)
Prompt for Tasks
Referring Expression (bext)
Thiffssion Process
Gasssian
Object
Box
Box Decoder
Shot Object
Encoder
Dillusion Model
Demoising Process
Grounding O Object Detection
Ao Viloo vit Aie
3-D Sceue
Reprosontative Fooituee
Box Corrupted on Box Latent
Multiple Object Detection Taaks
Figure I: Promptable 3-D object localization via latent diffusion models: Conventional diffusion- based object detectors primarily apply stochastic perturbations directly to the bounding box coordi- nates. In contrast, our proposed approach perturbs the entire bounding box representation within learned latent space, enabling more structured and information preserving diffusion process. This latent formulation, combined with explicit conditioning mechanisms, supports a controlled and adaptable diffusion framework. By incorporating these enhancements, the proposed method offers improved flexibility and robustness, facilitating seamless adaptation to a wide range of downstream tasks: e.g., (a) general 3-D object detection, (b) few-shot detection, and (c) grounding-based detection.
unformation-preserving
computational efficiency and operational flexibility, echoing the inherent limitations encountered by DDPM [19] models in controlled image generation tasks.
Addressing these constraints, this research proposes an innovative diffusion-based object detection framework motivated by conditional latent diffusion models commonly utilized in generative mod- eling tasks. Specifically, delineated in Figure 1, approach draws conceptual parallels with DETR-based architectures [61, 75, 10]. Initially, object anchor features indicative of potential object presence within a scene extracted and subsequently integrated with semantic textual embeddings corresponding to relevant object classes, thereby enriching feature representations. Subsequently, variational autoencoder (box VAE) facilitates the mapping of object queries and their associated initial bounding- coordinates into a latent embedding space. Leveraging the conditional latent diffusion paradigm, distinct conditioning criteria are systematically applied during the noise-to-box diffusion process, specifically tailored to downstream object detection scenarios, including general object detection, few-shot detection, and object grounding tasks. Distinguished from conventional ap- proaches, the proposed methodology exhibits substantial adaptability, enabling seamless application across diverse detection objectives. The primary contributions of our method are as follows:
Introduction of a conditional latent diffusion framework for improved adaptability and effectiveness 3-D object detection over existing diffusion-based methods.
Development of versatile conditional latent diffusion paradigm capable of addressing multiple object detection tasks through adjustments of conditioning parameters alone.
Empirical demonstration of competitive performance and superior flexibility in general 3-D object detection tasks and specialized downstream scenarios, notably 3-D few-shot detection and 3-D grounding detection.
2 Related work
2.1 Diffusion models in 3-D vision
Although initially designed for 2-D generation tasks, diffusion models have recently gained significant
traction in 3-D applications. Common applications of diffusion models in 3-D include text to 3-D generation [46, 82, 76, 48] image to 3-D generation [36,4] 41, 44), 3-D editing and manipulation [59, 8), and novel view synthesis [37, 62, 21]. Due to their strong capability in modeling complex data distributions, several studies have attempted to leverage diffusion models for visual perception tasks, such as semantic segmentation [94] and language grounded classification [60], and human pose

---

estimation [14, 13, 20]. In this work, we focus on applying diffusion models to diverse 3-D object identification and localization tasks, which remain relatively underexplored.
2.2 Diffusion models for object detection
Diffusion models have achieved remarkable success in numerous visual perception downstream tasks. Chen el al. [7] proposed DiffusionDet, the first work to apply diffusion models to object detection. DiffusionDet frames the process as "noise-to-box", which starts with fixed number of noisy box proposals and progressively refines them into the desired object boxes through a reverse denoising process. Building upon this framework, several works [93, 52, 30, 3] extend 2-D object detection to the 3-D domain and demonstrate promising results compared to previous anchor-base methods. Diffusion-SSS SS3D [17] and DiffDDETR [10] leverage diffusion models refine both bounding box proposals and class label distributions. This design, combined with a teacher-student framework, facilitates supirsupe 3-D object detection. Despite achieving promising results, prior works directly predict box parameters, which limits their flexibility to generalize to other downstream tasks. In contrast, our work introduces a unified framework that encodes bounding boxes into a latent representation, enabling object identification and localization across diverse scenarios.
2.3 Few-shot 3-D object detection
The goal of few-shot 3-D object detection is to identify and localize objects in 3-D data by learning from base classes with abundant labeled data and generalizing them new object categories with only a few labeled instances. Several existing approaches are based on the prototype learning paradigm [32, 90, 68]. Prototype learning extracts useful information from labeled data and utilizes the learned "prototype" to guide detection in unlabeled data. These methods [32, 90] are often built upon the VoteNet [50] architecture. Tang er al. [68] employ a VAE-based model learn representative prototypes. Other works, such as Liu et al. [35], focus on few-shot 3-D object detection outdoor scenes and autonomous driving scenarios. Meta-Det3D [84] addresses the few-shot 3-D object detection problem using meta-learning approach.
2.4 3-D visual grounding
3-D visual grounding aims to detect target objects that align with the given text description. A number of works formulate 3-D visual grounding as a segmentation task [91, 47, 87, 92, 11, 27, 80, 6, 31, 73, 67, 2, 28]. In contrast, our focus is on localizing objects that match the language query using bounding boxes. Methods for 3-D visual grounding can be roughly categorized into single-stage methods and two-stage methods. Single-stage methods [26, 42, 77, 51, 74, 16, 2] fuse text and vision features and directly output the predicted boxes based on the fused representations. In contrast, two-stage pipeline methods [5, 23, 81, 85, 12, 42, 26, 77, 95, 63, 51, 74, 57, 24, 88] first generate fixed set of box proposals. In the second stage, each box candidate matched with the text input to produce the final predictions. Our method extracts candidate object representations and encodes these features along with their corresponding coordinates into a structured latent space. A video stable diffusion model is then utilized to fuse this information with the provided text description. This fusion framework follows a structure similar to that of a two-stage pipeline. There are also works such [22, 72, 86, 79, 95, 96] that leverage the power of large language models to tackle 3-D visual grounding tasks. Although these works do not specifically focus on 3-D visual grounding problems, approach achieves stronger results under evaluation protocol in most scenarios without rel relying on heavily pretrained large language models or complex fusion pipelines.
3 Method
Inspired by the "noise-to-box" paradigm employed in diffusion-based object detectors, we introduce a promptable latent diffusion detector designed to enhance the adaptability and precision of 3-D object detection. The architecture comprises three primary components: (1) A 3-D scene feature extraction module that encodes visual features, which are subsequently fused with semantic features to generate object anchor representations for the diffusion process (Section 3.2). (2) These object anchor representations, along with associated bounding box coordinates initialized with random perturbations, are processed through box encoder module to obtain compact latent representation
3

---

of the bounding box (Section 3.3). (3) The latent representation is then refined via a conditional latent diffusion model, which iteratively adjusts the bounding box parameters through a learned diffusion process conditioned on the latent space (Section 3.4). This structured approach ensures more flexible and accurate 3-D object localization by leveraging conditional priors, making well-suited for applications that require precise and adaptable object detection in complex scenes.
3.1 Preliminaries
3-D object detection The task of 3-D object detection is fundamental in computer vision, involving the identification and precise localization of objects within a diree-dimensionl scene. Given a point cloud representation of scene, (p, E _ where n denotes the number of points, along with a task-dependent promptable conditioning input c, the goal is to predict a set of 3-D bounding boxes that accurately encapsulate target objects. Each bounding box is parameterized as b = (z, y, 2, h, w, O2, Oy: 0₂) E R², where (I, y.2) 2) represents the centroid of the box. (h, w, w, specifies spatial dimensions, and (0., Oy. 0z) encodes its orientation. However, in practice, orientation data poses significant challenges due to inconsistencies between ground truth annotations and model predictions, as noted in [17. 83]. For instance, ScanNet [9] and SUN RGB-D [65] datasets either assign a default orientation of zero or contain inconsistent orientation data across scenes. Following the approach of [10, 83], we therefore consider only the centroids and sizes of the bounding boxes. The objective of generative object detection is to produce bounding boxes that effectively delineate individual object instances. In our formulation, by incorporating task-specific constraints defined by the promptable conditioning input c, we explore generative approaches to produce bounding boxes adapted to various 3-D detection scenarios, including but not limited 3-D object detection based few shots and grounding.
Diffusion-based object detector Recent advances in object detection have increasingly embraced the "noise-to-box" paradigm, as explored in both 2-D [7] and 3-D [17, 3, 10] settings. During training, the diffusion-basse based detection decoder fa estimates the clean bounding box bo from a corrupted version b, the visual features x. and the corresponding timestep t. Unlike traditional diffusion models that predict the noise, this approach directly regresses by by minimizing the objective:
(I) where Rol(-) denotes the region-offinterest alignment operation, used to extract relevant visual features. During inference, an initial set of randomly sampled noise boxes by iteratively refined via the detection decoder and DDIM sampling steps [64], ultimately yielding the final predictions b
L. fl (RoI(x, b,). t) boll?:
Unlike prior diffusion- -aaee detection methods that directly regress bounding box coordinates, the proposed approach introduces a conditional latent diffusion model that predicts the noise in a latent space. This formulation is consistent with the prevalent practice in other cistusion-baeed generative modeling and offers improved flexibility for handling complex detection scenarios. Accordingly, the training objective is defined as:
Ls = (b,, e
(2)
where b, denotes the noisy latent representation obtained after applying /. forward diffusion steps to the encoded representation b Ebb o). Here, & denotes the Box VAE encoder, represents the object anchor features, and c is the conditional input. While conceptually analogous to the object queries in DETR -sylee architectures, our anchors are enhanced with semantic information through mooss-modal alignment between visual and textual embeddings. leading to more informative and context-aware representations. Further details are provided in Section 3.2 and Section 3.3.
3.2 Language-guided object anchor features
As illustrated in Figure and aligned with prior works [10, 7]. the proposed methodology adopts DETR-based framework employing learned object anchors. For 3-D visual feature extraction, rather than exclusively utilizing conventional pretrained visual backbones such as PointNet++ [49] or PVCNN [39], we leverage foundation model composed of two feature extractors, fo and fe, to generate semantically coherent 3-D visual representations, thereby enriching the encoded features with enhanced contextual relevance. Given point cloud representation of a scene, {p: 33) J= where n is the number of points, the visual features
{x." ] are extracted via:
fo : 3 + d and {2 fo(ma ((p.)=
(3)

---

Hungarian Riparttte Manching
Localization Regiession
The desk is in the back right of room to the left the bed Prompt: Referring Expression
Co
Prompt Encoder
Grounding Object Detection
Decoder
Prompt Encoder
Few-Shot Object Detection
Ca
Prompt: A Annotated objects
MLP
Latent Diffusion M
Vision Encoder
Gaussian Noise
fo
Global Visual Feature
[==
Box Encoder
[P:]2 J 3-D Sceme
Aignmentt
Query
Language
(2)
Cross
The cataggories)
Encoder
Key, Value
Attention
MLP
10 12-1
1.4
b: Initial Boxes
Calegories Names
Object Anchors
Figure 2: Model architecture. The proposed framework comprises three key components: (i) vision-language foundation model that extracts object anchor features, serving as candidate object representations; (ii) pretrained variational autoencoder (VAE) designed to encode bounding box coordinates into a structured latent space representation; and (iii) a latent-conditioned diffusion model that leverages conditioning features extracted by dedicated encoder, facilitating adaptability across diverse downstream tasks by incorporating task-specific constraints within the latent space.
where Jns (.) denotes performing downsampling via Farthest Point Sampling (FPS) to yield reduced set of na points. augment the object anchor features with semantic context beyond visual information, introduce cross-attention mechanism in which the extracted visual features {z]} 1 serve as queries, and the latent semantic representations {zī 1 i=-- obtained from the text encoder fo (with token length nz), act keys and values. (See Figure 2.) This tross-attention produces the conditioned object query features {or) J by:
Q = (z)),)), K() = PE ( == MHCA K(n, ER", o = pf E Rd
v()= xz) Rd,
(4)
(5)
where p ()) o (-) and pP correspond single-layer MLPs while (!) is a two-layer MLP to project the conditioned features effectively. The operator (HCA(F) refers to a multi-head cross-attention module [71] that facilitates semantic alignment between modalities.
3.3 Box representation
Unlike existing diffusion-basse detectors, which estimate the final bounding boxes solely based on their noisy versions, we propose leveraging conditional latent diffusion processor to enhance flexi- bility. Similar to conventional latent diffusion approaches, the first step involves using a VAE module to project the input modality into latent space. Specifically, we adopt a V-DETR [61]-like module as the VAE for the 3-D box representation, given its efficiency and high performance as demonstrated in [10]. As illustrated in Figure 3, object queries {o, [o 1i along with their corresponding bounding (I,, yt. 21, ha, wi, 1,) and the global visual features of the scene {z.) ) (M2 not equal to )2 are passed through a box encoder to extract the latent box representation. The first component of the encoder a variant of the multi-head self-attention (MHSA) module (including attention and residual operators) applied to object anchors:
boxes (b, |
ba = (b),,, Q' = ch (b² +0), K = -b +0),
V = =b= E RA
(6)
ba MHSA(Q", K V*) Rd
(7)
where (.(.), (.). the (-). and (-) represent single-layer MLPs. The second component
cross-attention module that takes the output ba from the self-attention module and the visual features of the scene {z, ] n2 i= as inputs:
Q Pb +b*),
=== + RPE), V==po(z") E Rd,
(8)
6 - 1--MCCAQE KS, (9, Rd.
(9)
5

---

Featue
Fordllowwrdd Network dleaiture head
Updkted Qusy Andhors
Mull--Hod
Muli- Honl
OljectAndhor
Self Attention
RIE
Crose
FeedFowward
C
K
MLP
Network (box hradd
Feedlooward
Bonddng Box
Box Encoder
Box Decoder
Figure 3: The Box VAE takes as imput object anchor features, 3-D spatial coordinates, and scene representations. Initial box centers (I,, 2.) are determined via FPS on anchor positions, while box sizes (h,, W4, 4)) are initialized using the dataset's ' normalized average dimensions. the box encoder, self-attention is applied with queries and keys formed by summing object anchor features and coordinate embeddings, and values taken as the anchor features. A cross-attention module further scene-level features, enhancing spatial- metiaalcontextual
refines this representation by conditioning awareness. The box decoder employs two feedforward networks: one predicts final bounding boxes, while the other updates query anchor features, enabling robust refinement and precise localization.
where (.) (-).)) and (.) are single-layer MIPs, while RPF represents the relative position encoding. In the original V-DETR, the output of the cross cots-atttntionn module, b, is passed through three independent FFN heads to predict the class, rotation angle, and bounding box. In our formulation, we treat the two attention modules the encoder of the box VAE and the FFN heads as the decoder. Since our approach requires utilizing b as the latent representation of the bounding boxes for the latent diffusion process, we modify the architecture accordingly.
3.4 Box refinement via conditional latent diffusion
Latent diffusion models (LDMs) [58] project the original data x into a latent space using an encoder tailored to the data modality-e., VQ- VAE [70, 53] for images audio VAE [34] for audio- -proor to applying the diffusion process. When combined with classifier-free guidance [18), LDMs enable
conditional generation. The model is trained by minimizing the following objective:
La = | Zc, C2, ell?
(10)
Here, eş Cz, t) incorporates a cross-attention mechanism [7 II, allowing the conditional embedding c; influence the latent variable z. The conditional score estimation then defined as:
é c2)) (1 We (Ze, C2, wep (Zr, L,
(11)
where w modulates the strength of classifier-free free guidance, and Ø denotes the unconditional embed- ding. To integrate the latent representations of bounding boxes b into the diffusion process, we adapt the conditioning mechanism defined in Equations (10) and (11) by substituting z; with b, the noisy latent representation of the bounding boxes. This conditional latent diffusion process enables iterative refinement of box features through dedicated decoder heads, which generate updated object anchors and bounding boxes conditioned on contextual information. In our implementation, the model is initialized using a pretrained video-based LDM from Stable Diffusion, providing strong prior for temporal and spatial coherence the latent space.
Model training and inference As depicted in Figure 2, the proposed framework jointly optimizes a cross-attention-ban based object anchor generator and conditional latent diffusion model. The overall training objective integrates a latent-space denoising loss with detection losses applied to the decoded bounding boxes. Specifically, the total loss is defined as:
Liotal = Aaii
Laif + >des Ldet:
(12)
where Law denotes the mean squared error between the predicted and true noise in the latent space, as defined in Equation (2), and Ldet includes classification and regression losses computed on the de- coded bounding boxes. We adopt a DETR-style pipeline, incorporating Hungarian bipartite matching and non-maximum suppression (NMS). To further enhance detection accuracy and robustness, focal loss [33] and asymmetric classification losses [66, 56) are additionally employed. The weighting coefficients Aaif and Adet dynamically adjusted throughout training, detailed in Section 4.

---

At inference time, each bounding box is initialized with noisy latent representation by. correspond- ing to the final timestep of the forward diffusion process. This representation is iteratively refined via DDIM sampling, guided by the detection decoder and conditioned on object anchor features and task-specific prompts, ultimately producing the final bounding box predictions bo.
Promptable 3-D vision tasks Owing its conditional and modular design, the proposed framework supports a wide range of 3-D object detection tasks, including general detection, few-shot detection, and grounding. Each task is described as follows:
The objective is to detect all objects present in 3-D scene. During training, the conditional prompt c, is expressed as a textual description: "The [class namel, class name2, ... class nameN] objects in the 3D scene." This aligns semantic cues with visual features to enhance representation learning. As class names are already used in anchor generation, their inclusion in the prompt is optional and omitting them incurs only minor performance drop.
General 3-D object detection:
Few-Shot 3-D object detection (FS3D): Following prior work [90, 68], the full class set C' is divided into base classes Chase with ample labels and novel classes Cnoell with limited samples, where Casee n Cuovel Ø. To address this setting, we adopt an episodic training strategy and introduce architectural modifications to support FS3D (detailed in the supplementary material). The query input consists of 3-D scene, while the support set is processed by 3-D encoder to extract visual exemplars. These exemplars, combined with textual class names, form the conditioning input cy-
Grounding 3-D object detection: This task aims to localize only the objects referred to by a natural language query, rather than detecting all instances the scene. Referring expressions more flexible than fixed category names and may include spatial, relational, or attribute- cues. These expressions serve as the conditional input cg to the latent diffusion process. To handle grounding scenarios in which single query may correspond to multiple target objects, as in MRlti3 we extend the inference pipeline with non- maximum suppression and top-K filtering. This allows the DETR-style framework to return multiple high-confidence predictions per query. The query-conditioned diffusion process then refines candidate boxes for accurate localization.
4 Experiments
In this section, we present experimental results aimed at demonstrating the effectiveness of the proposed method.
Training and loss functions The training process involves two primary stages: (1) training the diffusion model with frozen VAE, and (2) fine-tuning the decoder. In the first stage, contrary to previous works [10, 17] that typically use mean values of 0.25 for size noise and 1 L/Telass for label sampling, set the latent noise mean to 0.1, empirically demonstrating improved performance. Optimization employs the Adam optimizer with an initial learning rate of x 10-4, a cosine annealing schedule featuring a 500-step linear warm-up, and minimum learning rate of x 10-6 to ensure stable convergence.
During the diffusion stage, the loss function follows Equation (2), combining latent-space denoising loss Liitt and auxiliary detection loss Laet. as defined the Method section. To reuse pretrained video LDM weights, add lightweight MLP adapter between the V-DETR box-latent and the diffusion backbone. Concretely, given b R two-layer MLP projects a 4-channel canvas D' H. with H₁ x W,, N on which the video-IDM U-Net operates. Its output mapped back via another two-layer MIP to R xd and decoded to boxes. We compute the diffusion loss Lain on the noised canvas b': and the detection loss Laet after the inverse projection and the box decoder. This matches the 4-channel latent convention in Stable Video Diffusion. The model trains for 18K iterations with batch size of 8, accumulating gradients over 16 steps. The decoder remains frozen at this stage, with Ldet. indirectly guiding latent refinement. In the second stage, we unfreeze the VAE decoder, keeping the encoder fixed, and fine-tune the decoder with detection losses, using reduced learning rate and regularization to maintain pretrained decoder stability. To balance training objectives, the loss coefficients initially set Adift 1.0, Adet 0.2 gradually adjust to Xaint 0.5.
7

---

Table 1: General object detection on SUN RGB-D and ScanNetV2 datasets.
Table 2: Few-shot object detection on FS-SUNRGBD.
Bold texts denote the best results on each scenario
Method
WAReeS
SUN RGB-D [1]
nAP3-10
ScanNeV3 (65]
Methed
I-bst
FS-SUNRGED
3-4bot
S-lbot
WacNel 3DETR [-3
59,1
57.9
30.7
57-8
65.0
47.0
36.0
IAPP
IAPP55
(AAP5550
RAPPISS
IAPP
Groap 1008
63.0
45.
69.1
52.8
VoteNee
G (35)
17.52
12.84
V40E2 [61]
67.3
17.4
PrnnCCniizae VoteNaa [7]]
7.03
1.17
20.52
4.19
21.03
6.71
Pranaa Vong [75]
7.54
1.3M
31. 181
4.25
22.01
6.77
5830
43.2
MeDD [54]
6.77
0.73
15.37
2.00
24,23
5.88
DIERDETR (10)
65.7
44.
Polsoppea [X0]
21.51
8,16
Calirw11 (1)
52.0
Poneoppall (18]
Ours
taakggnasd Table 3: Few-shot object detection on Ne-SccnNet. Bold texts denote the best results on each scenario.
PS Sconnne
Method
Spir-1
Spli-2
Fistio
3-shot
5-chet
T-shot
3-stor
5-St0c
MoeNet
13:72
8,00
21.15
3.79
1.71
18.19
5.52
ArveraliondF51 [SS]
13.00
8.39
34.90
101.36
70.30
16.00
9.19
1.80
19:41
6.50
12.24
PoinCCintras: NoueNee [78]
12,59
8.52
20.12
11.16
25.83
15.49
9.55
1.57
18,44
5.23
20.06
10.19
Poncal-NoiNNe11 [75]
13.81
7.57
21.38
10:11
24.65
14.75
9:16
1.68
15.65
4.38
30.3 35
10.25
Meu Det3D [34]
10.28
4.03
23.42
10.64
25.65
13.85
5.21
1.32
15.44
4.37
22.13
7,09
Prootypccal VouNNer (90)
15.
9.25
31.25
16.01
32.25
19:52
11
2.21
21.14
6.39
28.55
12:25
Prontypical [18]
15.00
31.27
10.00
31.70
36.75
6.37
25.03
13.35
Ades = 1.0. We present the training and inference process in Algorithm and 2. All experiments utilize eight NVIDIA RTX A6000 Ada GPUs.
4.1 General 3-D object detection
Datasets and evaluation metrics We evaluate the proposed method on two standard indoor benchmarks: SUN RGB-D and ScanNet. SUN RGB-D includes 5,285 training scenes along with corresponding validation scenes, while ScanNet comprises 1,201 training and 312 validation scenes reconstructed from 2.5 million RGB-D frames. Following prior works [17, 10, 50, 61], we evaluate on the 10 most common object classes for SUN RGB-D and 18 semantic classes for ScanNet. Performance measured using mean Average Precision (mAP) at IoU thresholds of 0.25 and 0.5. All results are averaged over three random splits, and we report both the mean and standard deviation.
Results Table presents the results on the general 3-D object detection task. The proposed method achieves approximately 5% improvement in mAP ScanNetV2 compared to other diffusion- detectors, and it demonstrates competitive performance on SUN RGB-D compared to state-of-the-art approaches. Note that while V-DETR uses a three-layer cascade structure, we use only a single layer. Additionally, for fair comparison, we also report results under CaTFree3D's experimental setting.
4.2 Few-shot 3-D object detection
Datasets and evaluation metric We test our method on two few-shot 3-D object detection bench- marks: FS-SUNNGBD and FS-ScanNet [90]. FS-SUNRGBD contains 5,000 point-cloud scenes spanning 10 object categories, while FS-ScanNet includes 1,513 scenes across 18 categories. The base/novel splits are 6/4 for FS-SUNRGBD and 12/6 for FS-ScanNet. Following standard proto- cols [68, 90], we report mAp at IoU thresholds of 0.25 and 0.5 under varying shot settings.
Results As described in Section 3.4, following [68, 90], we adopt an episodic training strategy. To adapt the scenario, we leverage support sets consisting of annotated point cloud features and text prompts as conditional inputs the latent diffusion model. The results on FS-SUNRGBD and FS FS-ScanNet under 1-shot, 3-shot, and 5-shot settings are summarized in Tables 2 and 3. Across all settings, the proposed method achieves approximately a 4% improvement in mAP, demonstrating its superior performance and strong generalization capability in the few-shot regime.
4.3 Grounding 3-D object detection
Datasets and evaluation metrics We evaluate the proposed method on three benchmarks for 3-D visual grounding: ScanRefer [5], Multi3DRefer [89], and ViGiL3D [72]. Specifically, ScanRefer
8

---

Table 4: Grounding object detection on ScanRefer, Multi3DRefer, and ViGiL3D.
Table 5: Zero-shot grounding object de-
tection on the open-vocabulary benchmark OpenI
Method
ScanReftr [5]
Mutt3DRefer [89]
VIGILDD (72)
Acc
Acc1550
116225
F1030
Asc
Acc9050
Oyeeles1D [3]
Scankefer [5]
Replica
Scashetee
RMDD
MuletDRefer
51.9
ConcreoNett (69)
46.5
OpenMall (61)
21.5
15.1
9.8
42
8.7
53
D-LIAA [86]
ConcrnKingaa [1]]
11.3
24
Chat-Some [2]]
52.4
11.00
PQ3D [96)
51.2
50.1
10.8
Ours
59.5
99.4
53.8
15.7
Results produced by our evaluations with the provided code.
focuses on single-object grounding, Multi3DRefer involves multi-object grounding per query, and ViGiL3D serves as a diagnostic benchmark featuring a mixture of single, multiple, and no-target queries. Following their respective evaluation protocols, we report Acc 25 and Acc@50 50 for ScanRefer, and Fl @25 and FI @50 for both Multi3DRefer and ViGiL3D. During training, we adopt DETR-style matching strategy using Hungarian bipartite assignment. At inference time, we retain high-confidence predictions via non-maximum suppression (NMS), which is consistently applied across all tasks. Notably, while Hungarian matching is suited for single-object settings such as ScanRefer, cannot directly support multi-object queries. Therefore, for Multi3DRefer and ViGiL3D, follow prior work [89] and apply a multi-match evaluation strategy based on IoU thresholds and label agreement to compute precision, recall, and Fl scores. We further evaluate our method in zero-shot setting the open- vocabulary benchmark OpenLex3D [28]. Since the dataset provides only semantic segmentation masks and synonym lists, we construct axis-aligned bounding boxes (AABB) from the segmentation results and use the first synonym in each list the reference label. Accuracy is reported at IoU thresholds of 0.25 and 0.5.
Results Table 4 shows that our method achieves strong performance across all three grounding benchmarks, demonstrating robustness in both single- and multi-object scenarios. In Table 5, also achieves competitive results on OpenLex3D under a zero-shot setting, highlighting strong generalization in open-vocabulary 3D understanding. While OpenMask3D and ConceptGraphs report slightly higher scores on some subsets, they rely on complex multi-stage pipelines involving multi- view fusion, explicit mask decoding, or large language models like CLIP, LLaVA, and GPT-4. In contrast, our approach leverages CLIP2Point for visual-language alignment and achieves comparable or better performance without external LLMs, handcrafted scene graphs, or post-procet pesst-pocessing.. This underscores the efficiency of promptable latent diffusion for open- vocabulary 3-D object grounding.
4.4 Ablation study and discussion
Genaral (ScanNett
As shown in Figure 4, we perform three ablation studies across eight 3-D benchmarks to evaluate three key components of our method.
68.3
Grounding
Gmena (UNGGB-D)
10 ve)0)
30.2
Language-guides object anchors Replacing the cross-attended anchors with purely visual features leads to consistent performance drops, particu- larly on open-vocabulary benchmarks such as ViGiL3D (-2.9) and OpenLex3D (-2.7), where semantic alignment crucial. The degradation on Multi3DRefer (-4.4) and ScanRefer (-5.2) fur- ther highlights the importance of tangguag--guider anchoring for accurate grounding.
0.6
47.2
88.5
49.1
Grounding (VIGLL50)
13.5
10.4
5.1
43.5
R2
10.1
12.3
136
I-shot (1S-ScannNet
3.5
39.2
13
47.4
4544
96.
47.5
1-shet
Grounding DCuRtIDRdie)
(FS-SUNRG
52.7
Full evadd
Grounding (Scanil eler)
N/ Dilicioo
Langugg-Andhee
Latent diffusion refinement Substituting our diffusion module with direct regression signifi- cantly degrades few-shot performance, with FS- ScanNet dropping from 13.64 to 8.24 (-5.4) and FS- OS-SUNRGBD from 6.52 to 4.32 (-2.2). Ground- ing accuracy also declines, e.g., ViGiL3D: (-4.1), Multi3DRefer: (-6.4), indicating the value of iter- ative refinement under limited data.
Frongt
Figure 4: Ablation study eight 3-D detec- tion and grounding benchmarks, comparing the full model with versions without latent diffusion, language-guiie
anchors, or prompt condition- ing. Each component contributes significantly, especially in open-vocabulary grounding tasks.

---

Table 6: Summary of key characteristics of diffusion-basse approaches versus the proposed method.
Method
Prompt modality
Detection task
Representative score
Def [7]
2-D detection
COCO: 45.8 mAP@
GroundingDINO [38]
text
2-D detection
LVIS :32.5 mAP@50
Diffusion-SSaD [17]
3-D detection
ScanNerV2: 43.2 mAP@
DifTDETR [10]
3-D detection
ScanNetV2: 44,9 mA
Ours
text, image
3-D detection, Few-shot, Grounding
datasets prompt 3% (1)
Promptable conditioning Removing prompt conditioning causes the most severe degradation across grounding tasks. Performance on ScanRefer drops from 52.7 to 39.2 (-13.5), Multi3D from 53.8 to 39.7 (-14.1), and ViGiL3D from 13.3 to 6.3 (-7.0). Even in one-shot settings, e-g., FS-SUNRGBID: (-3.1), performance declines, confirming the necessity of prompts for semantic guidance, particularly in ambiguous or data-scarce conditions.
Overall, the ablation results are consistent with our design goals: promptable conditioning is critical for language-driven tasks, diffusion enables robust learning in low-data regimes, and language-guides anchors improve semantic grounding, particularly in open-vocabulary settings.
Core contributions of the proposed method As shown in Table 6, our diffusion-basse detector is built upon the concept of a noisc-to-box, reflecting the recent advancements of diffusion models across computer vision, including both discriminative tasks and dense prediction scenarios. Recent methods such Marigold [29] for depth estimation further highlight the flexibility of diffusion- models. However, existing diffusion-based detectors, such as DiffusionDet [7] and Diffusion-SS3DD [17], typically diffuse only raw box coordinates. This design restricts their ability to incorporate arbitrary language inputs or exemplar-based prompts directly, thereby limiting their generalization capacity. Our method addresses these limitations by leveraging latent diffusion models, which can seamlessly integrate diverse multimodal inputs. By embedding an aligned foundational model into the diffusion process, our approach achieves high degree of flexibility and adaptability across a wide range of tasks and modalities. Unlike standard detectors, our framework natively supports various input types, allowing for flexible and precise prediction control via prompt-based conditioning. This feature significantly broadens the practical utility of diffusion bbsed detection frameworks. The main novelty of our method is the integration of prompt conditioning into the diffusion- based detector through a noise-to-box paradigm, as emphasized in the Introduction and Conclusion. Our approach draws inspiration from recent findings, such as the work "Multimodality Helps Few-shot 3-D Point Cloud Semantic Segmentation". which demonstrates the benefits of leveraging multiple modalities [1]. We specifically employ latent diffusion model due to its intrinsic ability to fuse multimodal information effectively. While existing diffusion- based detectors are usually restricted to a single task or modality (e.g., DiffusionDet is limited to 2-D detection, Diffusion-SS closed-set 3D detection without prompt conditioning, and GroundingDINO [38]. although promptable, is confined to 2D detection tasks), our proposed framework overcomes these barriers. By utilizing a latent box VAE, our method enables a single trained model to flexibly support closed-set detection, few-shot adaptation, and language-drive 3D detection, all by simply modifying the prompts (e.g., class names, visual exemplars, or natural language deseriptions).
and Diff3DETR [10] only address
5 Conclusion
We propose a unified and flexible framework for 3-D object detection that reformulates the diffusson process conditioned noise-to-box transformation. Unlike prior diffusion-basee detectors, our method introduces a conditional latent diffusion model that enables promptable box generation within a latent space. By incorporating task sseciiec prompts, our approach seamlessly adapts to diverse 3-D detection settings, including general, few-shot, and grounding based scenarios. To enhance detection quality, we adopt DETR-style architecture to generate semantically rich object anchors, which serve as informative inputs to the diffusion process. Extensive experiments across various benchmarks demonstrate the versatility and strong performance our method, highlighting both its generalization capabilities and the potential of diffusion-based modeling in 3-D object detection.
10

---

This work was supported in part by NSTC grants 113-2221-E-00100000 --001-0100-YY3 and 113-2634-F-007-002 of Taiwan. We thank National Center for Herf-perfoe Computing for providing computing resources.
Acknowledgements.
References
Zhaochong An, Guolei Sun, Yun Liu, Runjia Li, Min Wu, Ming-Ming Cheng, Ender Konukoglu, and Serge J. Belongie. Multimodality helps few-shot 3d point cloud semantic segmentation. In ICLR, 2025. 10
[2] Sergio Arnaud, Paul Mayay, Ada Martin, Arjun Majumdar, Krishna Murthy Jatavallabhula, Phillip Thomas. Ruslan Partsey, Daniel Dugas, Abha Gejji, Alexander Sax, et al. Locate 3d: Real-world object localization via self-supervised learning in 3d. arXiv preprint arXiv:2504. 14151, 2025. 3
[3] Wenjing Bian, Zirui Wang, and Andrea Vedaldi. Catfree3d: Category-agnostte 3d object detection with diffusion. CoRR, abs/2408. 12747, 2024. 3,4,
[4] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Volcti, Adam Letts, Varun Jampani, and Robin Rombach. Stable vidco diffusion: Scaling latent video diffusion models to large datasets. CoRR, abs/231 2023.
[5] Dave Zhenyu Chen, Angel X. Chang, and Matthias Nießner. Scanrefer: 3d object localization RGB-D scans using natural language. In ECCV, 2020. 3, 8, 9, 18, 19
[6] Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma. Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by CLIP. In CVPR, 2023.
[7] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In ICCV, 2023. 3,44,,11 10
[8] Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang. Zhongang Cai, Lei Yang, Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable editing with gaussian splatting. In CVPR, 2024. 2
[9] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias Nießiner Scannet: Richly-annotace 3d reconstructions of indoor scenes. In CVPR, 2017. 4, 8, 18
[10] Jiacheng Deng, Jiahao Lu, and Tianzhu Zhang. Diffãdetr: Agent-based diffusion model for semi-supervised 3d object detection. In ECCV, 2024. 2, 3, 4, 7, 10,20 20
[11] Runyu Ding, Jihan Yang, Chubui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. PLA: language-driven
open-vocabulary 3d scene understanding. In CVPR, 2023. 3
[12] Mingtao Feng, Zhen Li, Qi Li, Liang Zhang, Xiangdong Zhang, Guangming Zhu, Hui Zhang, Yaonan Wang. and Ajmal Mian. Free-form description guided 3d visual graph network for object grounding in point cloud. In ICCV, 2021. 3
[13] Runyang Feng. Yixing Gao, Tze Ho Elden Tse, Xueqing Ma, and Hyung Jin Chang. Diffpose: Spatiotem- poral diffusion model for video-based human pose estimation. In ICCV, 2023. 3
[14] Jia Gong, Lin Geng Foo, Zhipeng Fan, Qiuhong Ke, Hossein Rahmani, and Jun Liu. Diffpose: Toward more refiable 3d posc estimation. In CVPR, 2023.
[15] Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, Chuang Gan, Celso Miguel de Melo, Joshua Tenenhaum, Antonio Torralba, Florian Shkurti, and Liam Paull. Conceptgraphs: Open- vocabulary 3d scene graphs for perception and planning. In ICRA, 2024. 9
[16] Wenxuan Guo, Xiuwei Xu, Ziwei Wang, Jianjiang Feng, Jie Zhou, and Jiwen Lu. Text-guided sparse voxel pruning for efficient 3d visual grounding. CoRR, abs/2501.01.. 2025. 3
[17] Cheng-Ju Ju Ho, then-Hsuan Tai, Yen- Yu Lin, Ming-Hsuan Yang. and Yi-Hsuan Tsai. Diffusion-ss3dl Diffusion model for semi-supervised 3d object detection. In NeurIPS, 2023. 4, 10, 20
[18] Jonathan Ho and Tim Salimans. Classificr-free diffusion guidanee. CoRR, abs/2207. 2022, 6, 17
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NewrIPS, 2020.

---

[20] Karl Holmquist and Bastian Wandt. Diffpose: Multi-hypotheiis human pose estimation using diffusion models. In ICCV, 2023.
[21] Basile Van Hoorick, Rundi Wu, Ege Ozguroglu, Kyle Sargent, Ruoshi Liu, Pavel Tokmakov, Achal Dave, Changxi Zheng, and Carl Vondrick, Generative camera dolly: Extreme monocular dynamic novel view synthesis. In ECCV, 2024. 2
[22] Haifeng Huang, Yilun Chen, Zehan Wang, Rongjee Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, and Zhou Zhao. Chat-scene: Bridging 3d scene and large language models with object identifiers. In NeurIPS, 2024. 3,9
[23] Pin-Hao Huang, Han-Hung Lee, Hwann- Tzong Chen, and Tyng- Lug Liu. Text-guided graph neural networks for referring 3d instance segmentation. In AAAI, 2021. 3
[24] Shijia Huang. Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual grounding. In CVPR. 2022. 3
[25] Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson W. H. Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer CLIP to point cloud classification with image-depth pre-training. In ICCV, 2023, 16, 18, 19
[26] Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In ECCV, 2022. 3
[27] Li Jiang, Shaoshuai Shi, and Bernt Schiele. Open-vocabulary 3d semantic segmentation with foundation models. In CVPR, 2024. 3
[28] Christina Kassab, Sacha Morin, Martin Büchner. Matías Mattamala, Kumaraditya Gupta, Abhinav Valada, Liam Paull, and Maurice Fallon. Openlex3d: A new evaluation benchmark for open-vocabulary 3d
scene representations. CoRR, 19972503.19764, 2025. 3, 9, 18
[29] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- based image generators for monocular depth estimation. In CVPR, 2024. 10
[30] Se-Ho Kim, Inyong Koo, Inyoung Lee, Byeongjun Park, and Changick Kim. Diffreffd:: A diffusion-based proposal refinement framework for 3d object detection. CoRR, 16,429,,,,6349, 2023.
[31] Ruihuang Li, Zhengqiang Zhang, Chenhang Hc, Zhiyuan Ma, Vishal M. Patel, and Lei Zhang. Dense multimodal alignment for open-vocabulary 3d scene understanding. In ECCV, 2024. 3
[32] Xuejing Li, Weijia Zhang, and Chao Ma. Cp-votenet: Contrastive prototypical votenet for few-shot point cloud object detection. In PRCV, 2024. 3
[33] Tsung-Yi Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. IJC,, 2017.
[34] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. In ICML, 2023. 6
[35] Jiawei Liu, Xingping Dong. Sanyuan Zhao, and Jianbing Shen. Generalized few-shot 3d object detection of lidar point cloud for autonomous driving. CoRR, abs/2302.03914, 2023. 3,8
[36] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T., Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh 45 seconds without per-shape optimization. In NeurlPS, 2023.
[37] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to- 1- Zero-shot one image to 3d object. In /CCV, 2023. 2
[38] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng L.i, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang, Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. ECCV, 2024. 10
[39] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel CNN for efficient 3d deep learning. In NewrIPS, 2019. 4
[40] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In ICCV, 2021. 8
12

---

[41] Xiaoxiao Long. Yuan-Chen Guo, Cheng Lin, Yuan Liu. Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song- Hai Zhang, Mare Habermann, Christian Theobalt, and Wenping Wang. Wonder3d: Single image 3d using cross-domain diffusion. CVPR, 2024.
[42] Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haihing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single-stage visual grounding via referred point progressive selection. In CVPR, 2022. 3
[43] Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao, Lem-lora: A universal stable-diffusion acceleration module. CoRR, 0tss/2311.05556
2023. 19, 20
[44] Luke Melas- Parlas-Kyriazi,, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion 360 reconstruction of any object from a single image. In CVPR, 2023. 2
[45] Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to end transformer model for 3d object detection. In ICCV, 2021. 8
[46] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. CoRR, abs/2212.08851,, 2022. 2
[47] Songyou Peng. Kyle Genova, Chiyu Max Jiang, Andrea Tagliasacchi, Mare Pollefeys, and Thomas A. Funkhouser. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023. 3
[48] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion.
In ICLR, 2023. 2
[49] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Lconidas J. Guibas. Pointnet++: Deep hierarchical fcature learning point sets in metric space. In NeurlPS, pages 5099-5108, 2017. 4
[50] Charles R. Qi, Or Litany, Kaiming He, and Leonidas J. Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV, 2019. 3,
[Si] Zhipeng Qian, Yiwei Ma. Zhekai Lin. Jiayi Ji, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Multi-brande collaborative learning network for 3d visual grounding. In ECCV, 2024. 3
[52] Yasiru Ranasinghe, Deepti Hegde, and Vishal M. Patel. Monodiff: Monocular 3d object detection and pose estimation with diffusion models. In CVPR, 2024. 3
[53] Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ- VAE-2. In NewrlPS, 2019.
[54] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time ohject detection. In CVPR, 2016.
[55] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. NewrIPS, 2015.
[56] Tal Ridnik, Emanuel Ben Baruch, Nadav Zamir, Asaf Noy. Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor, Asymmetric loss for multi-label classification. In ICCV, 2021. 6
[57] Junha Roh, Karthik Desingh, Ali Farhadi, and Dieter Fox. Languagerefer: Spatial- language model for 3d visual grounding. In CoRL, 2021.
[58] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022.
[59] Etai Sella, Gal Fiehelman, Peter Hedman, and Hadar Averhuch-Flor, Vox-e: Text-guided voxel editing of 3d objects. In ICCV, 2023. 2
[60] Sitian Shen, Zilin Zhu, Linqian Fan, Harry Zhang, and Xinxiao Wu. Diffelip: Leveraging stable diffusion for language grounded 3d classification. In WACV, 2024. 2
[61] Yichao Shen, Zigang Geng, Yuhui Yuan, Yutong Lin, Liu, Chunyu Wang. Han Hu, Nanning Zheng, and Baining Guo. V-DETR: DETR with vertex relative position encoding for 3d object detection. In ICLR, 2024. 2, 5,8, 8, 16
[62] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent -iewi-view diffuusion base model. CoRR. Abs/2310.15110 2023. 2
13

---

[63] Xiangxi Shi, Zhonghua Wu, and Stefan Lee. Viewpoint-aware visual grounding in 3d scenes, In CVPR. 2024. 3
[64] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021. 4,
17
[65] Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao. SUN RGB-D: A RGB-D scene understanding benchmark suite. In CVPR, 2015. 4, 8, 18, 19, 20
[66] Ximeng Sun, Ping Hu, and Kate Sacnko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. In NeurlPS, 2022. 6
[67] Ayça Takmaz, Flisabetta Fedele, Robert W. Sumner, Mare Pollefeys, Federico Tombari, and Francis Engelmann. Openmask3d: Open-vocahulary 3d instance segmentation. In NeurlPS, 2023. 3,
[68] Weiliang Tang, Biqi Yang. Xianzhi Li, Yun-Hui Liu, Pheng-Am Ann Heng, and Chi- Miig Fu. Prototypical variational autoencoder for 3d sew-shot object detection. In NeurIPS, 2023, 3, 7, 8
[69] Ozan Unal, Christos Sakaridis, Suman Saha, and Lue Van Gool. Four ways improve verbo-visual fusion for dense 3d visual grounding. ECCV, 2024.
[70] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurIPS, 2017.
[71] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017. 5, 6
[72] Austin T. Wang. ZeMing Gong, and Angel X. Chang, Vigil3d: A linguistically diverse dataset for 3d visual grounding. CoRR, abs/2501.01 2025. 3, 9,
[73] Pengfei Wang, Yuxi Wang, Shuai Li, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Open vocabulary 3d scene understanding via geometry guided self-distillation. In ECCV, 2024. 3
[74] Yuan Wang, Yali Li, and Shengjin Wang. G Marrying hyperbolic alignment with explicit semantic- geometric modeling for 3d visual grounding. In CVPR, 2024. 3
[75] Zhenyu Wang, Ya-Li Li, Xi Chen, Hengshuang Zhao, and Shengjin Wang. Uni3detr: Unified 3d detection transformer. In NewrlPS, 2023. 2,8 8
[76] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurlPS, 2023, 2
[77] Yanmin Wa, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. EDA: explicit text-decoupling and dense alignment for 3d visual grounding. In CVPR, 2023.
[78] Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas J. Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for point cloud understanding. In ECCV, 2020. 8
[79] Jianing Yang. Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar. David F. Fouhey, and Joyce Chai -lmouuuudee:: apccbvoccy visual grounding with large language model an agent. In ICRA, 2024. 3
[80] Jihan Yang, Runyu Ding, Weipeng Deng, Zhe Wang, and Xiaojuan Qi. Regionple: Regional point-language contrastive learning for opeenworld 3d scene understanding. CVPR, 2024.
[81] Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. SAT: 2d semantics assisted training for 3d visual grounding. In ICCV, 2021. 3
[82] Taoran Yi, Jiemin Fang. Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text 3d gaussians by bridging 2d and 3d diffusion models. In CVPR, 2024. 2
[83] Hong- Xing Yu, Jiajun Wu, and Li Yi. Rotationally cquivariant 3d object detection. In CVPR, 2022. 4
[84] Shuaihang Yuan, Xiang Li, Hao Huang, and Yi Fang. Meta-det3d: Learn to learn few-shot 3d object
detection. In ACCV, 2022. 3,
[85] Zhihao Yuan. Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. In- stancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In ICCV, 2021. 3
14

---

[86] Zhihao Yuan, Jinke Ren, Chun-Mei Feng, Hengshuang Zhao, Shuguang Cui, and Zhen Li. Visual programming for zero-shot open-vocabulary 3d visual grounding. In CVPR, 2024. 3
[87] Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung.
Zhen Yang, Xiaodan Liang, and Hang Xu. Clip?: Contrastive language-imagepoone pretraining from real-world point cloud data. In CVPR, 2023.
[88] Haomeng Zhang, Chiao-An Yang. and Raymond A. Yeh. Multi-object 3d grounding with dynamic modules and language-informed spatial attention. In NeurlPS, 2024. 3,
[89] Yiming Zhang, ZeMing Gong, and Angel X. Chang. Multi3drefer: Grounding text description to multiple objects, ICCV, 2023. 8,9, 9,
[90] Shizhen Zhao and Xiaojuan Qi. Prototypical votenet for few-shot 3d point cloud object detection. In NeurIPS, 2022. 3, 18, 19
[91] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from CLIP. In ECCV, 2022. 3
[92] Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation scale. In ICLR, 2024. 16, 18, 19
[93] Xin Zhou, Jinghua Hou, Tingting Yao, Dingkang Liang, 7he Liu, Zhikang 7ou, Xiaoqing Ye, Jianwei Cheng. and Xiang Bai. Diffusion-based 3d object detection with random boxes. In PRCV, 2023.
[94] Xiaoyu Zhu, Hao Zhou, Pengfei Xing, Long Zhao, Hao Xu, Junwei Liang, Alexander Hauptmann, Ting Liu, and Andrew C. Gallagher. Open-vocabulary 3d semantic segmentation with text-to-image diffusion models. In ECCV, 2024.
[95] Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text aligmment. In ICCV, 2023.
[96] Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang. and Qing Li. Unifying 3d vision-languag understanding via promptable queries. ECCV, 2024.
3,
15

---

A Additional implementation details
Support Cf
Prompt Encoder
Vision Encoder
Rol MaxPooling
MLP
Set
Support Set
Prompt Encoder Few--Shot Object Detection
p' by Annolated Object
lignment
Query
Hungarian Tipartit Marthing
Incaliation Regression
Language Encoder
Cross
Category Name
Key, Value
Attention
Object Detection
Box Decoder
Prompt: A nnotated objects and category name
Share Weight
MLP
C2
Latent Diffusion Model
(ii))
\~2
Vision Encoder
Gaussian Noise {z,) the
Encoder
Global Visual Feature
Q uerr Set
{p)) 2-1 3-D Somee
Aligmeent
Query
Language Encoder
2 1
Cross
Categories Names
of (categontes)
Key,
A Attention
MLP
[o Object Anchors
b
11 1=
Initial Boxes
Figure 5: Model architecture for the few-shot setting.
Architecture for few-Shot object detection We adopt an episodic training paradigm to address few-shot object detection and introduce architectural modifications tailored to the FS3D setting. Unlike general grounding-base based 3-D detection, FS3D framework employs dual-branch architecture that separately processes a query point cloud and a small set annotated support scenes. Each query input is complete 3-D scene represented as point cloud {p, E IR Ji=1, which is processed by backbone network to produce per-point features (z; E Rd J==1 where d denotes the feature dimension.
For the support set, each support sample is also a full 3-D scene ((pf E R However, only the points enclosed within ground-truth bounding boxes are used to represent the relevant object instances. Specifically, for each bounding box (b" (z, h, w, ()), we extract the subset of points within the region of interest (Rol) and obtain their corresponding features via the backbone network. These features are then aggregated using max pooling to produce an instance-level representation of the object:
(f (b))))))
(13)
In the FS3D prompt encoder, these instance-level features are passed through an additional MLP and cross-attention module to produce class-aware and instance-aware latent representations. These latents are subsequently used to condition the diffusion-basee detector. This design differs from prior FS3D approaches that typically fuse support and query features via cross-attention at detection time. Instead, the proposed method treats the support set as an explicit conditioning signal for the generative latent process, promoting flexibility and compatibility with generalizable multimodal prompts.
Language-guided object anchor features For object anchor generation, we employ vision- language pretrained models, CLIP2Point [25] and Uni3D [92]. whose vision encoders align with the CLIP encoder, facilitating seamless multimodal integration. As detailed in Section 3.2, generate = 128 object anchors ) initializing the corresponding bounding boxes {b, using Farthest Point Sampling (FPS) for centroids. The dimensions of the box are initialized with the average dimensions of the dataset or randomly within a normalized range [0, 1].
Box representation For the Box VAE, we utilize a pretrained one-layer V-DETR module [61], omitting the ordinal prediction branch and adapting the global feature from the visual encoder (n2 = 10224). Note that the original V-DETR employs three-stage cascade architecture, whereas approach leverages simplified, single-layer adaptation.
16

---

Algorithm 1: Training
Algorithm 2: Inference
def train(pe, gt.b,, gt.1, clan, cond, T): Extract
def inference (pc, clan, cond, I, stepa):
3-D scene features pts, foundation. ienndationneecoderr cnc.oer...nddd.. v(pc)
pts, Foundation. encoder. (ppc)
2t foundation encoder. sendderrtccllee ((cle))
foundation wtwpe/ttnnencndcttcccc.
Computa conditional anheddings pronpt.encoder encoder (cond)
Compute conditional enbeddings
prompt proopt.encoder" encoder (cond)
Generate object
features bo sross_attention(2r, zt)
object
bo aioe__aitentoon(w,, 2t)
Initialize bounding boxes
# Initialize noisy latent boxes bb_init init_ (bo)
bb_init init_boxes
Encode to latent space db_latent bax_vae.co wncooeerrbbbbeineeee
# Eacode to latent space bb_latent bex_vae. wnc.oe. init, bo)
Sample randon diffusion timestep randint(1,
randon noise in latent donain ege bornal(neane std-1, size=bb_latent shape) bb_noisy (obrupt(bb_latttt,
Add
noise-epa)
normal (mean=0, std-1) ab_noisy corrupt latent, t, ops)
Prepare sampling time_ time_pointa 1inspace(-1, stepa) tines reversed(tine_ pointte) pairs List times
Predict noise with diffusion model spa_pred Idmbho_noia,,
Compute diffusion I_ diff mse (eps_pree,, egs)
t_cur, t_next in pairs: # Predict noise at current step apa_pred 1dn(bb_notay, cz, t_cur)
Decode latent to boxes bb_prad box_ box_vae.dee dacoder (bb_latattt
DDIM update latent
bb_ncisy ddin(bb_noigg epe_pred, nextt)
Compute detection loss __det detection_ ltspion__ssstbb_rreeeee (be_p,ed, gt_b, gt_l) loss L_diff + L_det
# Decode final latent to boxes bi_final box vas, dacodar (bb_notay)
apdata loss)
return losg
return binfinal
corrupt(x, t, ops) inqrt( (t)pa_
linapace:generate evanly spaced
sqrt(1 alpha_cumprod(t)) apa
cuppo_(cuprod(()): Ii o.
Box Refinement via Conditional Latent Diffusion We the DDIM [64] noise scheduler with a maximum of 1000 timesteps, initializing the latent diffusion model from pre-trained stable video diffusion weights [4]. Classifier-free guidance [18] is applied with guidance scale of 3.5. For the prompt encoder, we adopt the CLIP text encoder for general and grounding object detection tasks, benefiting from its well-established alignment and widespread use in latent diffusion models.
B More experimental results
B.1 Performance consistency and stability
As mentioned above, due to space constraints, only the mean performance is reported in the main paper. To support the reported main results in the paper, include the performance over three independent runs with different random seeds for each benchmark. As shown Table 7, the proposed method achieves consistent results with low variance general, few shot, and grounding 3-D object detection tasks. This demonstrates the stability and robustness of the approach despite the diverse task settings.
B.2 Ablation study
In addition the core components discussed in the main paper, we further investigate impact of different box initialization strategies. Specifically, we compare random initialization with a strategy that uses the average object size computed from the training set as prior. shown in Table using average size initialization consistently improves performance across all three benchmarks:
17

---

Table Performance of the proposed method across different 3-D detection and grounding bench- marks. We report the mean and standard deviation over three independent runs with different random seeds, illustrating the consistency and robustness the method.
mean Average Precision
Accuracy
FI FI-Score
Dataset
m5
mAP@50
Acc
6c
F1 6-
FI 69
General 3-D object detection
SUN RGB-D [9]
67.4 + 09
50.2 0.5 60.3 + 0.7
ScanNetV2 [65]
72.8 +: 1.3
Few-shot 3-D object detection
FS-SUNRGBD (1-shot) [90]
20.69 + 1.32
6,52 + 062
FS-SUNRGBD (3-shot) [90]
34.72 + 1.11
13.52 0.3
FS- "S-SUNRGBD (5-shot) [90]
40.52 +: 145
20.25 0.5
FS-ScanNet Split 1 (1-shot) [90]
20.34 + 150
13.64 0.54
FS-ScanNet Split 1 (3-shot) [90]
36.75 +
24.42 063
FS "S-ScanNet Split 1 (5-shot) [90]
37.45 17 1.72
26.54 097
FS -S-ScanNet Split 2 (1-shot) [90]
17.23 1 194
6,37 1.B
FS- NS-ScanNet Split 2 (3-shot) [90]
25.63 + 1.53
13.54
+ 0.55
FS "S-ScaNNet Split 2 (5-shot) [90] 41,36 + 1.45
19.75+ 0.56
Grounding 3-D object detection
ScanRefer [5]
59.5 1
$2.7 + 0.9
Multi3DRefer [89]
19.4
53.8 -
ViGiL [72]
15.7 +
13.3 +
OpenLex3D (Replica) [28]
19.5 0.
17.9 + 06
OpenLex3D (ScanNet++) [28]
11.3
5.4
OpenIee33 (HM3D) [28]
9.9 +
7.6 10
Table 8: Ablation study on three 3-D detection and grounding benchmarks comparing box initial- ization strategies. Using average box size prior leads to consistent improvements over random initialization.
ScanNetV2 [65]
FS-SUNRGED (1-shot) [90]
ScanRefer [S]
Method
mAP@25
mAP@50
mAP@25
mAP@50
Acc@25
Acc@50
Method w/ random initial
71.2 = 166
59.5
19.52 133
6.49 + oms
57.2
51.4 + 1.1
Method w/ average size boxes
72.8 + 13
60.3 1: 07
20.69 132
6.52 1 0.2
59.5 1 12
52.7 + 0s
Scan [65], FS-SUNRGBI [90], and ScanRefer [5]. This confirms that incorporating simple geometric priors leads to more stable and accurate predictions both detection and grounding tasks.
Box initialization strategy We adopt a strategy that uses the average size of bounding boxes in the dataset to provide strong prior for initialization. As shown in Table 8, this approach outperforms random initialization across all benchmarks, confirming our hypothesis that informed priors lead to more stable and accurate performance. We also observe that the variance in performance is notably higher under random initialization, which is intuitive since randomly sampled box sizes introduce greater variability in the optimization process.
Foundation models For extracting object query features, we leverage two vision vision-language foun- dation models: CLIP2Point [25] and Uni3D [92]. As shown in Table 9, the overall performance is slightly better when using Uni3D across all three benchmarks. While CLIP2Point and Uni3D differ more significantly in zero-shot 3-D scene recognition settings, their impact on object query features is more subtle. This is likely because the downstream detection performance relies more heavily on the diffusion process and latent refinement, rather than the initial feature representation alone.
Number of object anchors Following prior diffusion- based detectors, we adopt 128 initial object anchors as the default setting for fair comparison. As shown in Table 10, increasing the number of anchors from 64 to 256 leads to improved performance in both general and few-shot 3-D object detection tasks. However, the gains are marginal for grounding tasks such as ScanRefer. This is expected, as ScanRefer typically involves a single target object per query, and 128 anchors already provide sufficient coverage for localizing bounding box.
18

---

Table Ablation study comparing different foundation models for extracting object query features. Using Uni3D leads to slightly better performance than CLIP2Point across three 3-D detection and grounding benchmarks.
Sc [65]
ES-SUNRGBD (1-shot) [90]
ScanRefer [5]
Method
mAP@25
mAP@50
mAP@25
mAP@50
Acc@25
Acc@ 50
Method w/ CLIP2Point [25]
72.2 + 13
59.0 + 08
19.11 + 133
6.34 + 0.61
58.7 + 1.7
51.6 t: 0.9
Method w/ Uni3D [92]
72.8 + 13
60.3 = 0.7
20.69 + 1.32
6.52 + 0.62
59.5 t 17
52.7 :: 0.9
Table 10: Ablation study on the number of object anchors used in the detection pipeline. While increasing the number of anchors improves general and few-shot detection performance, the effect is limited in single-object grounding scenarios such as SeanRefer.
ScanNetV2 [65]
FS-SUNRGED (I-shot) [90]
ScanRefer [5]
Method
mAP@25
mAP@50 50
mAP@25
mAP@50
Acc@25
Acc@50
Method w/ 64
69.3 + 13
56.7 + 0.9
17.23 + 1.31
5.74 + 0.62
57.4 + 1.7
51.6 + 09
Method w/ 128
72.8 +
60.3 +
20.69 + 1.32
6.52 + 0.62
59.5 + 1.7
52.7 : 09
Method w/ 256
73.2 =
61.2 07
21.45 1.27
6.67 + 0.60
59.7 +
52.8 + 0.8
Table 11: Ablation study on loss weighting strategies. The proposed scheduled weights improve
stability and generalization across detection tasks, compared to fixed or single-objective settings.
ScanNetV2 [65]
FS-SUNRGED (I-shot) [90]
ScanRefer [5]
Loss Weights (Aais Ades)
mAP@50 m5P@25
mAP@25
mAP@50
Acc@25
Acc 50
(1.0, 0.2) - (0.5, 1.0)
72.8 = 13
60.3 + 07
20.69 t 1.32
6.52 + 0.62
59,5 t 1.7
52.7 0.
(1.0, 1.0) - (1.0, 1.0)
71.2 = 14
59.1
+ 0.
18.92 + 1.1
6.07 + 0.60
60.1 t 1.5
51.2 + 08
(1.0, 0.2) - (1.0,
70.3 = 1.5
58.4 1:
17.53
+ L.
5,71
+ 0.77
18.4 15
50.7
+ 0.9
(0.0, 1.0) - (0.0, 1.0)
66.1 = 17
53.6 1.2
12.41 + 1.6
4.38 1 0.82
54.2 +: 19
47.3 + 1.1
Loss weight scheduling We study the impact of different loss weighting strategies between the latent space denoising loss Laitt and the detection loss Last. As shown in Table 11, our proposed scheduling scheme starting from (Adift, Ades) ) = (1.0, 0.2) and gradually transitioning to (0.5, 1.0) achieves the best overall performance in general, few-shot and grounding benchmarks.
Using equal weights (1.0, 1.0) improves grounding slightly but performs worse in low-data scenarios such PS-SUNRGBD, suggesting overemphasis on detection loss early in training. Keeping fixed low weight on detection loss without scheduling, as in (1.0, 0.2), leads to underfitting and degraded overall accuracy. Complete removal of diffusion loss (0.0, 1.0) results in significant performance drops in all tasks, confirming that the latent denoising objective essential for effective and robust localization.
Accelerated variants for efficiency For completeness, we report two accelerated variants of our method: 4-step and 2-step LCM-LLRA [43] model. Due the limited timeframe for the rebuttal, the newly introduced hyperparameters, such as learning-rate warm-up and cosine decay schedules for the LoRA adapter, have not yet been fully optimized. We anticipate further improvements with more extensive tuning. As shown in Table 12, although our base model is relatively large, its end-to-end runtime matches or exceeds the FPS throughput of competing methods, while consistently achieving higher accuracy. The promptable design also naturally extends to few-shot and grounding tasks without requiring retraining, which highlights the versatility of our framework. In addition, standard engineering optimizations, including mixed precision. quantization, and channel scaling or sparse coding approaches such as Matryoshka representations, could further enhance the efficiency of our method. Nevertheless, we wish to emphasize that the primary focus of this paper is to introduce the promptable concept within diffusion-based detection framework and explore the broader potential of diffusion models.
19

---

Table 12: Performance of the proposed method and its accelerated variants on general 3D object detection, evaluated on ScanNetV2.
ScanNetV2 [65]
Method
mAP@50 (t)
Model Parameters ())
Lateney/scene (L)
FPS (L)
Diffusion-SS3D [17]
64.1
30.07
Diff3DETR [10]
65.7
Ours (DDIM-8)
72.8
1.03B 1.03B + 70M LoRA
0.28s
3.60
Ours (LCM-LoRA-4 [43])
72.3
0.05s
22.23
Ours (LCM-LoRA-2 [43))
71.9
1.03B + 70M LoRA
0.03s
30.07
B.3 Qualitative results
Figure 6 presents qualitative examples from the ScanNetV22 and Multi3DRefer datasets. The proposed method produces accurate and well-localized 3-D bounding boxes in both general object detection and grounding tasks, with predicted boxes closely matching the ground truth. These results highlight the model's robustness and adaptability across different types of 3-D scenes.
C Limitations
While the proposed method demonstrates strong versatility across various 3-D detection tasks, the denoising process inherent to latent diffusion models relies on iterative refinement, which can be computationally expensive. Future work may explore integrating one-step alternatives (e.g. Consistency models Rectified flow models) to improve efficiency. Moreover, although the proposed framework supports promptable conditioning, our experiments primarily focus on text and visual modalities. Leveraging more complex prompts (e.g. audio video) remains open direction for future research.
Broader impacts
The proposed method has the potential to benefit range of applications embodied AI, robotics, and interactive world models, especially in scenarios where labeled 3-D data is limited. However, the approach builds upon foundation models for vision-language alignment, may inherit biases present in those models. This could lead to skewed predictions when deployed in real world settings.
20

---

Input
Gramd Truth
Ours
3-Der
+
the table is white.
Referring Expression
3-D Scene
+
the chair nearest to the window faces away from the table.
Referting Expression
3DDee
+
P
P
the brown, circular table positioned the center of the couches
Referring Expression
Figure 6: Qualitative results on general 3-D object detection and grounding-baan based object detection.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The authors discuss the limitations of the work in the main content
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
This paper is not applicable to theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [No]
Justification: The paper discloses all the implementation details and information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/guides/Code wtwpeeee/oooeSuumiiiiinnneee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n /nipp..ccccc httpsscccppblccccccccccccccc gwwge//deeeeeeee /eddeC/Saubb ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide experiment details our paper necessary understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the statistical significance of our experiments in the supplementary materials. We present only the mean performance in the main paper due to space limitations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources needed to repro- duce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't ' make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurfaa http://wurrpsscpppllieelece cc/publicc/thhcscuuleeee ines?
Answer: [Yes]
Justification: The authors have carefully reviewed the NeurIPS Code of Ethics and con- formed with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts of the work in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited the creators or original owners of assets used in the paper and explicitly mentioned and properly respected the license and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have not yet released new assets in our paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing experiments nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
27

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https //tt:r/isn..////eeee Ceeeurips.cc/ooffeee /LLM/ for what should or should be described.
28


Reviews:
[{"review_id": "8FoOWMktos", "reviewer": "Reviewer_G49r", "summary": "This paper introduces a novel framework for 3D object detection by leveraging conditional latent diffusion models, addressing key limitations in existing methods. The primary contribution is the development of a \"promptable\" 3D object detection approach, where the refinement of bounding box predictions is guided by textual prompts, enabling flexibility across a range of downstream tasks. \nThe proposed framework integrates both visual and semantic cues, facilitating more robust performance in various detection scenarios, including general 3D object detection, few-shot detection, and 3D object grounding. Extensive experimental evaluations on standard 3D object detection benchmarks demonstrate that the method outperforms existing state-of-the-art approaches, showcasing its versatility and effectiveness.", "questions": "1. What specific novel contributions does your framework bring to the field beyond applying existing techniques in a new context? Please clarify any novel theoretical insights or design principles that distinguish your approach from prior work.\n2. The paper lacks a detailed discussion of the computational complexity of the proposed method. Could you provide an analysis of the computational cost in terms of runtime and scalability?\n3. Could you provide a more detailed discussion of how your approach compares to other models, especially in terms of flexibility, ability to condition on textual prompts, and how these features lead to improved performance across different tasks?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces a novel framework for 3D object detection using conditional latent diffusion models. The concept of \"promptable\" detection represents a innovation in integrating vision and language for object detection tasks.\n2. The proposed framework is highly flexible, able to handle multiple tasks including general 3D object detection, few-shot learning, and grounding-based tasks. This ability to perform well across various types of tasks without major modifications or retraining makes the approach highly adaptable.\n3. The paper provides comprehensive experimental results that demonstrate the effectiveness of the proposed method. It shows competitive performance against state-of-the-art approaches on well-established 3D object detection benchmarks, validating the robustness and generalizability of the model.\n\nWeaknesses:\n1. While the paper presents an interesting approach by combining latent diffusion models with text-based conditioning for 3D object detection, the framework lacks significant originality.\n2. It lacks a detailed discussion of the computational complexity of the proposed method. \n3. There is a need for a more explicit comparison with related work, particularly in terms of the model's flexibility, ability to condition on textual prompts, and how these factors lead to improved performance across tasks.", "was_revised": true, "final_justification": "In light of the rebuttal, which has addressed most of my concerns, I will maintain my original score.", "rebuttal": "We appreciate the reviewers’ valuable feedback. Below, we provide detailed responses to each of the main points raised.\n\n---\n\n**Comment**: \"... lacks significant originality ... What specific novel contributions does your framework bring to the field beyond applying existing techniques in a new context ... clarify any novel theoretical insights or design principles that distinguish your approach from prior work. ...''\n\n**Response:**  Departing from the limitations of “noise-to-box” methods that operate directly on raw coordinates, our approach reformulates the problem within a learned latent space. Each bounding box is encoded into a rich representation that encapsulates both its geometric structure and the surrounding visual context, enabling a powerful and promptable latent diffusion model. As demonstrated by our experiments, this framework offers a unified and versatile solution for 3-D object detection, consistently performing well across a wide range of challenging scenarios.\n\n**Prompt-conditioned box latent representation.** As detailed in Section 3.2, existing diffusion-based detectors (e.g., DiffusionDet and Diffusion-SS3D) operate by diffusing raw box coordinates and thus cannot effectively incorporate arbitrary language inputs or exemplar prompts. In contrast, we encode 6-DoF boxes into a compact 256-dimensional latent space via a Variational Autoencoder (VAE) and introduce cross-modal context through a shared token interface. This strategy explicitly decouples spatial noise from semantic control. \n\n**Unified “promptable” anchor design.** A unified mechanism of cross-attention transforms various modalities of prompts: class lists, few-shot exemplars, and free-form referring phrases into updates of the latent space score. This design allows our single checkpoint to seamlessly perform generic detection, few-shot transfer, and 3-D grounding tasks without head replacements or additional fine-tuning, a capability unmatched by existing 3-D detectors. \n\n**Multimodal classifier-free guidance (CFG) for 3-D boxes.** We generalize CFG to accommodate joint vision-language scoring, thereby enabling prompt-specific trade-offs between precision and recall. This feature is absent in current DiffusionDet variants and latent 3-D generation frameworks.\n\n**Task-agnostic training objective.** Unlike prior approaches, some necessitate task-specific control tokens; our training approach treats prompts solely as inputs. Consequently, switching between datasets like SUNRGB-D (closed vocabulary), ScanRefer (open vocabulary), or FS-ScanNet (few-shot learning) requires no additional or specialized loss functions. Additionally, we note that, as illustrated in Fig.~1 and supported by our experimental results, the proposed formulation is capable of handling input conditions beyond text prompts when annotated objects are provided.\n\n---\n\n**Comment**: \"...  lacks a detailed discussion of the computational ... provide an analysis of the computational cost in terms of runtime and scalability? ...''\n\n**Response:**  Thank you for highlighting the need for a clearer analysis of computational complexity. We have now included comprehensive 3-D detection results on ScanNetV2 under the general object detection setting. Additionally, following reviewer n1sa's suggestion and as mentioned in the limitation section, we report two accelerated variants of our method (a 4-step and a 2-step LCM-LoRA model) for completeness. Given the tight timeframe for the rebuttal, the newly introduced hyperparameters (e.g., learning-rate warm-up and cosine decay schedules for the LoRA adapter) may not yet be fully optimized, and we anticipate further improvements with more thorough tuning. As demonstrated in the provided table, although our base model is relatively large, its end-to-end runtime matches or exceeds the FPS throughput of competing methods while consistently achieving higher accuracy. Moreover, the promptable design seamlessly extends to few-shot and grounding tasks without requiring retraining, underscoring the versatility of our proposed framework. Furthermore, existing engineering techniques, e.g., mixed precision, quantization, and channel scaling/sparse coding approaches like Matryoshka representations, could further enhance the efficiency of our method. However, we wish to emphasize that the primary focus of this paper is to introduce the promptable concept within a diffusion-based detection framework, thereby exploring the potential of diffusion models. Comprehensive details on latency, parameter count, accuracy, and discussions regarding scalability across sampling steps and GPU configurations have been incorporated into the revised manuscript.\n\n\n| Method            | mAP@50(%) (↑) |Model Parameters| Latency/scene (↓) | FPS (↓) |\n|-------------------|:-------------:|:-----------------:|:-----------------:|:-------:|\n| Diffusion-SS3D    |      64.1     |         -         |         -         |  30.07  |\n| Diff3DETR         |      65.7     |         -         |         -         |    -    |\n| **Ours (DDIM‑8)**|      **72.8**     |       1.03B       |       0.28s       |   3.60  |\n| **Ours (LCM‑LoRA‑4)**|      72.3     | 1.03B + 70M LoRA |       0.05s       |  22.23  |\n| **Ours (LCM‑LoRA‑2)**|      71.9     | 1.03B + 70M LoRA |       **0.03s**       |  **30.32**  |\n\n---\n\n**Comment**: \"... more explicit comparison with related work ... more detailed discussion of how your approach compares to other models, especially in terms of flexibility, ability to condition on textual prompts, and how these features lead to improved performance across different tasks?''\n\n**Response:**  Thank you for the opportunity to clarify the motivation behind our work. Our diffusion-based detector builds upon the noise-to-box concept. Similar to recent approaches, such as Marigold (a diffusion-based depth estimator), diffusion models have demonstrated considerable potential across a variety of computer vision tasks, including both discriminative applications and dense prediction scenarios. Existing diffusion-based detectors (e.g., DiffusionDet and Diffusion-SS3D) typically diffuse raw box coordinates, which limits their capability to directly incorporate arbitrary language inputs or exemplar prompts, thus restricting their generalization capabilities. In contrast, our method leverages the inherent advantages of latent diffusion models, particularly their ability to seamlessly fuse diverse multimodal inputs. By integrating an aligned foundational model within the diffusion framework, our approach achieves remarkable versatility across numerous applications. Compared to standard detectors, our proposed method readily incorporates multiple modalities, providing flexible and precise control over predictions. This significantly broadens the practical applicability of diffusion-based detection frameworks. As emphasized in both the Introduction and Conclusion sections of our paper, our method uniquely integrates prompt conditioning into the diffusion-based detector through the noise-to-box paradigm. Our approach is inspired by recent findings, such as the work \"Multimodality Helps Few-shot 3-D Point Cloud Semantic Segmentation,\" which highlights performance gains achievable through leveraging multiple modalities. Specifically, our work employs a latent diffusion model due to its intrinsic capability to effectively integrate multimodal data. In contrast, existing diffusion-based detectors usually address singular tasks or modalities: DiffusionDet focuses exclusively on 2-D detection; Diffusion-SS3D and Diff3DETR are restricted to closed-set 3-D detection without prompt-based conditioning; and GroundingDINO, although promptable, is limited strictly to 2-D detection tasks. Our proposed framework overcomes these limitations by utilizing a latent box VAE, enabling a single trained model to flexibly perform closed-set detection, few-shot transfer, and language-based 3-D detection tasks merely by altering the prompts (e.g., class names, visual exemplars, or natural language descriptions). Additionally, we have prepared a comprehensive comparison table, which we will incorporate into the revised manuscript to further illustrate these distinctions.\n\n\n| Method                          | Prompt modality                | Detection task                              | Representative score                         |\n|------------------------------------------------------|----------------------------------|----------------------------------------------|------------------------------------------|\n| DiffusionDet (2-D)| –                              | 2-D detection                               | COCO: 45.8 AP$_{50}$                  |\n| GroundingDINO (2-D)  | text               | 2-D detection                               | LVIS : 32.5 AP$_{50}$                  |\n| Diffusion-SS3D (3-D)       | –                                | 3-D detection                               | ScanNetV2: 43.2 AP$_{50}$             |\n| Diff3DETR (3-D)               | –                               | 3-D detection                               | ScanNetV2 : 44.9 AP$_{50}$            |\n| **Ours**                                | **text, image**| **3-D detection, Few-shot, Grounding**    | **8 datasets w/ prompt + 3 % (↑)** |"}, {"review_id": "27psq9wUUL", "reviewer": "Reviewer_21Ug", "summary": "The paper proposes Promptable 3D Object Localization with Latent Diffusion Models, a unified detection framework that encodes 3D bounding boxes into a latent space with a VAE style box encoder, then refines them through a conditional latent diffusion model, and finally drives the diffusion with prompts including category lists, few‑shot exemplars, and natural language referring expressions. Across eight public benchmarks, the method achieves very competitive performances. In addition to experiments, an extensive ablation study shows that each of the three components, language‑guided anchors, latent diffusion, and prompt conditioning, contributes to accuracy.", "questions": "What is the inference latency per scene for each different task?\n\nIf provided with the correct ground truth orientation, could the latent representation be extended to SE(3), so that the full 6-DoF bounding box can be predicted? What else do the authors think is the bottleneck to learning orientation?\n\nFor the grounding part, how robust is the model against paraphrase shift or some ambiguous spatial propositions?", "limitations": "See weakness and question part.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength:**\nThe author redesigns the ‘noise-to-box’ diffusion used in previous works into denoising in latent space, so that it could be conditioned on arbitrary prompts, enabling a unification of detection, few-shot transfer, and grounding into one single architecture\n\nExperiments span SUN RGB‑D, ScanNetV2, FS‑SUNRGBD, FS‑ScanNet, ScanRefer, Multi3DRefer, ViGiL3D, and OpenLex3D, covering both closed and open vocabulary scenarios. Results are reported at two IoU thresholds with standard deviations.\n\nAblation study in Figure 4 isolates the impact of each component across eight tasks, supporting the claims of versatility and robustness.\n\n\n**Weakness:**\nAll datasets are scanned rooms, it is unclear whether the method generalizes to outdoor LiDAR, autonomous driving ranges, or partial-view robotics data.\n\nInitialising from a video StableDiffusion backbone hints at high memory, training time, and inference, yet the paper provides neither a resource table nor an efficiency report.\n\nAlthough mentioned at the beginning that the orientation ground truth is often not correct, such work might benefit the robotic community where the pose of the bounding box is essential. It would be better if the author could report the orientation result.", "was_revised": true, "final_justification": "The authors addressed my concerns. Experiments on outdoor scenes and efficiency evaluation look good. I raise my score.", "rebuttal": "We appreciate the reviewers’ valuable feedback. Below, we provide detailed responses to each of the main points raised.\n\n---\n\n**Comment**: \"... it is unclear whether the method generalizes to outdoor LiDAR, autonomous driving ranges, or partial-view robotics data. ... could report the orientation result. If provided with the correct ground truth orientation, could the latent representation be extended to SE(3), so that the full 6-DoF bounding box can be predicted? What else do the authors think is the bottleneck to learning orientation? ...''\n\n**Response:**  We sincerely appreciate the reviewer's valuable feedback. Similar to most existing diffusion-based detection methods, our primary focus is on indoor datasets, with only a few diffusion-based approaches addressing outdoor scenarios. Regarding the orientation issue, unlike previous approaches such as Diffusion-SS3D and Diff3DETR, where the search space can become excessively large, our method operates directly within the latent space. This design inherently simplifies orientation consideration, particularly because the box-VAE we utilize includes an angle prediction head. Consequently, orientation can be seamlessly integrated into our framework. Additionally, when accurate ground truth orientations are available, we can effectively initialize the angle prediction head, analogous to our approach of leveraging average box sizes for initialization. Following the reviewer's suggestion, we conducted additional experiments on the KITTI dataset and report the corresponding AP$_{3D}$ results on the validation set. We adopted the general object detection setting, augmented with the angle residual regression loss as described in V-DETR. As shown in the provided table, our method achieves competitive performance compared to MonoDiff (CVPR'24). Given the tight rebuttal timeline, the newly introduced hyperparameters have not yet undergone exhaustive optimization. We anticipate further performance improvements with additional tuning.\n\n| Method  |AP$_{3D}$ (Easy) | AP$_{3D}$ (Moderate) | AP$_{3D}$ (Hard) |\n|:-------:|:----------------------:|:--------------------------:|:----------------------:|\n| MonoDiff | 32.18 | 22.02 | 19.84 |\n| **Ours** | **33.32** | **23.89** | **21.14** |\n\n---\n\n**Comment**: \"...  Initialising from a video StableDiffusion backbone hints at high memory, training time, and inference, yet the paper provides neither a resource table nor an efficiency report. What is the inference latency per scene for each different task? ...''\n\n**Response:**  Thank you for highlighting the need for a clearer analysis of computational complexity. We have now included comprehensive 3-D detection results on ScanNetV2 under the general object detection setting. Additionally, following reviewer n1sa's suggestion and as mentioned in the limitation section, we report two accelerated variants of our method (a 4-step and a 2-step LCM-LoRA model) for completeness. Given the tight timeframe for the rebuttal, the newly introduced hyperparameters (e.g., learning-rate warm-up and cosine decay schedules for the LoRA adapter) may not yet be fully optimized, and we anticipate further improvements with more thorough tuning. As demonstrated in the provided table, although our base model is relatively large, its end-to-end runtime matches or exceeds the FPS throughput of competing methods while consistently achieving higher accuracy. Moreover, the promptable design seamlessly extends to few-shot and grounding tasks without requiring retraining, underscoring the versatility of our proposed framework. Furthermore, existing engineering techniques, e.g., mixed precision, quantization, and channel scaling/sparse coding approaches like Matryoshka representations, could further enhance the efficiency of our method. However, we wish to emphasize that the primary focus of this paper is to introduce the promptable concept within a diffusion-based detection framework, thereby exploring the potential of diffusion models. Comprehensive details on latency, parameter count, accuracy, and discussions regarding scalability across sampling steps and GPU configurations have been incorporated into the revised manuscript.\n\n| Method            | mAP@50(%) (↑) |Model Parameters| Latency/scene (↓) | FPS (↓) |\n|-------------------|:-------------:|:-----------------:|:-----------------:|:-------:|\n| Diffusion-SS3D    |      64.1     |         -         |         -         |  30.07  |\n| Diff3DETR         |      65.7     |         -         |         -         |    -    |\n| **Ours (DDIM‑8)**|      **72.8**     |       1.03B       |       0.28s       |   3.60  |\n| **Ours (LCM‑LoRA‑4)**|      72.3     | 1.03B + 70M LoRA |       0.05s       |  22.23  |\n| **Ours (LCM‑LoRA‑2)**|      71.9     | 1.03B + 70M LoRA |       **0.03s**       |  **30.32**  |\n\n---\n\n**Comment**: \"...  For the grounding part, how robust is the model against paraphrase shift or some ambiguous spatial propositions? ...''\n\n**Response:**  From our perspective, the CLIP ViT-L/14 text encoder naturally clusters paraphrases, and our classifier-free guidance (CFG) effectively translates small embedding shifts into subtle changes in the score field. Additionally, OpenLex3D intentionally provides multiple synonyms per object. Previously, Table 5 reported results using only the first synonym; however, we have now re-evaluated using all synonyms for the Replica split, resulting in a minor change from 17.9% to 17.1\\% (-0.8%). This modest reduction highlights the robustness of our method. With performance variations below 1% for both synonym expansion and fuzzy spatial terminology, our promptable latent diffusion method demonstrates consistent robustness without relying on external LLMs, handcrafted scene graphs, or multi-view fusion. We will include this detailed analysis in our revised manuscript."}, {"review_id": "zy716QnWMS", "reviewer": "Reviewer_D3tk", "summary": "This paper investigates a promptable latent diffusion model for 3D object detection.  It is based on some prior designs (like diffusiondet) that conducts the diffusion process on noisy bounding boxes for training and inference, while some new design is introduced in this paper. (1)  foundation vision and language models serve as feature extractor and a cross-attention mechanism that fuse the two feature to get anchor feature. (2) a pretrained VAE that is designed to encode bounding box into latent space. (3) The promptable design itself that support grounding and open vocabulary detection flexibly. Experiments are conducted on generic 3d detection benchmarks (SUN RGB-D and ScanNet), few-shot 3d detection (FS-SUNRGBD and FS-ScanNet), and Grounding 3-D object detection (: ScanRefer, Multi3DRefer, and ViGiL3D), showing promising improvement.", "questions": "1. Could you list the significance or any merits of using diffusion-based detector rather than using an equally computationally heavy but DETR based detector? In other words, why we should choose the diffusion-based detectors? \n2. Could you emphasize the novel of the architecture design?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "[Strength]\n1. The paper is clear and easy to follow.\n2. Using diffusion model for 3d object detection - despite whether practical - is underexplored and might have the potential to achieve better performance than mainstream methods due to its heavy compution per se.  From that perspective, it does not bother the community to see works in this area.\n3. The method is verified on three tasks, which makes it empirically solid. \n4. The promptable design makes the model general and fit few-shot, generic, and grounding tasks, expanding its ability. \n\n[Weakness]\n1. As claimed, I doubt the practical usage of diffusion based object detection.  I do not really think that detection (categorization + box regression) needs complex modeling that can only be handled by costly diffusion process.  Especially for bounding box regression, it should be an easy and solved problem in object detection; from this perspective, encoding bounding box into latent space remains questionable.  I think the effectiveness is from the strong VAE encoder that better extract ROI features, i.e., the learnable initial anchors gain merits from the better feature representation.  \n2. Furthermore, this diffusion design is time-consuming. The model requires language query as input, thus it seems to detect one class per run.  The diffusion process can also be slow.  These factors make the model an impractical use. \n3. The overall architecture looks like a combination of multiple pretrained powerful models.", "was_revised": true, "final_justification": "The paper overall is solid work, and the rebuttal partially resolved my concern.  Therefore, I will keep my rating.", "rebuttal": "We appreciate the reviewers’ valuable feedback. Below, we provide detailed responses to each of the main points raised.\n\n---\n\n**Comment**: \"... the practical usage of diffusion based object detection ... list the significance or any merits of using diffusion-based detector rather than using an equally computationally heavy but DETR based detector? ... why should we choose the diffusion-based detectors?. ...''\n\n**Response:**  Thank you for the opportunity to clarify the motivation behind our work. Our diffusion-based detector is based on the concept of box-to-noise separation. Similarly to recent approaches such as Marigold (a diffusion-based depth estimator), diffusion models have shown strong potential in various computer vision tasks, not only in discriminative applications but also in dense prediction scenarios. Existing diffusion-based detectors (e.g., DiffusionDet and Diffusion-SS3D) typically diffuse raw box coordinates. This limits their capability to directly incorporate arbitrary language inputs or exemplar prompts, constraining their generalization. In contrast, our method leverages the intrinsic advantages of latent diffusion models, specifically their ability to seamlessly fuse diverse multimodal inputs. By utilizing an aligned foundational model within a diffusion framework, our approach achieves remarkable versatility across numerous applications. Compared to standard detectors, our proposed method readily integrates various modalities, allowing flexible control over predictions. This significantly broadens the practical applicability of diffusion-based detection frameworks. As highlighted in our Introduction and Conclusion, our method uniquely integrates prompt conditioning into the diffusion-based detector by employing a noise-to-box approach. Inspired by recent research, i.e., ``Multimodality Helps Few-shot 3-D Point Cloud Semantic Segmentation'' which illustrates the performance gains achievable through leveraging multiple modalities. The proposed work specifically utilizes a latent diffusion model for its inherent capacity to integrate multimodal data effectively. In contrast, existing diffusion-based detectors typically address singular tasks or modalities: DiffusionDet focuses solely on 2-D detection, Diffusion-SS3D and Diff3DETR are restricted to closed-set 3-D detection without prompt-based conditioning, and GroundingDINO, though promptable, is limited exclusively to 2-D detection. Our proposed framework addresses these limitations by leveraging a latent box VAE, enabling a single trained model to flexibly handle closed-set detection, few-shot transfer, and language-based 3-D detection tasks simply by altering prompts (e.g., class names, visual exemplars, or natural language descriptions). Additionally, we have prepared a comprehensive comparison table that we will incorporate into the revised manuscript to further illustrate these distinctions.\n| Method                          | Prompt modality                | Detection task                              | Representative score                         |\n|------------------------------------------------------|----------------------------------|----------------------------------------------|------------------------------------------|\n| DiffusionDet (2-D)| –                              | 2-D detection                               | COCO: 45.8 AP$_{50}$                  |\n| GroundingDINO (2-D)  | text               | 2-D detection                               | LVIS : 32.5 AP$_{50}$                  |\n| Diffusion-SS3D (3-D)       | –                                | 3-D detection                               | ScanNetV2: 43.2 AP$_{50}$             |\n| Diff3DETR (3-D)               | –                               | 3-D detection                               | ScanNetV2 : 44.9 AP$_{50}$            |\n| **Ours**                                | **text, image**| **3-D detection, Few-shot, Grounding**    | **8 datasets w/ prompt + 3 % (↑)** |\n\n---\n\n**Comment**: \"... this diffusion design is time-consuming ... it seems to detect one class per run ... make the model an impractical use.''\n\n**Response:**  Thank you for highlighting the need for a clearer analysis of computational complexity. First, we would like to clarify the concern regarding whether our method detects only one class per run in general object detection. Notice that our implementation indeed performs single-round multi-class inference: for each ScanNet or SUN RGB-D scene, all dataset class names are passed simultaneously within a single prompt (e.g., by concatenating all labels into one CLIP sentence). As a result, the model is not restricted to one class per run. Multi-class prompting has been supported by our implementation from the outset. The implementation has supported multi‑class prompts from the start. Furthermore, we have now included comprehensive 3-D detection results on ScanNetV2 under the general object detection setting. Additionally, following reviewer n1sa's suggestion and as mentioned in the limitation section, we report two accelerated variants of our method (a 4-step and a 2-step LCM-LoRA model) for completeness. Given the tight timeframe for the rebuttal, the newly introduced hyperparameters (e.g., learning-rate warm-up and cosine decay schedules for the LoRA adapter) may not yet be fully optimized, and we anticipate further improvements with more thorough tuning. As demonstrated in the provided table, although our base model is relatively large, its end-to-end runtime matches or exceeds the FPS throughput of competing methods while consistently achieving higher accuracy. Moreover, the promptable design seamlessly extends to few-shot and grounding tasks without requiring retraining, underscoring the versatility of our proposed framework. Furthermore, existing engineering techniques, e.g., mixed precision, quantization, and channel scaling/sparse coding approaches like Matryoshka representations, could further enhance the efficiency of our method. However, we wish to emphasize that the primary focus of this paper is to introduce the promptable concept within a diffusion-based detection framework, thereby exploring the potential of diffusion models. Comprehensive details on latency, parameter count, accuracy, and discussions regarding scalability across sampling steps and GPU configurations have been incorporated into the revised manuscript.\n\n| Method            | mAP@50(%) (↑) |Model Parameters| Latency/scene (↓) | FPS (↓) |\n|-------------------|:-------------:|:-----------------:|:-----------------:|:-------:|\n| Diffusion-SS3D    |      64.1     |         -         |         -         |  30.07  |\n| Diff3DETR         |      65.7     |         -         |         -         |    -    |\n| **Ours (DDIM‑8)**|      **72.8**     |       1.03B       |       0.28s       |   3.60  |\n| **Ours (LCM‑LoRA‑4)**|      72.3     | 1.03B + 70M LoRA |       0.05s       |  22.23  |\n| **Ours (LCM‑LoRA‑2)**|      71.9     | 1.03B + 70M LoRA |       **0.03s**       |  **30.32**  |\n\n---\n\n**Comment**: \".... architecture looks like a combination of multiple pretrained powerful models. ... emphasize the novel of the architecture design? ...''\n\n**Response:**  From our perspective, the CLIP ViT-L/14 text encoder naturally clusters paraphrases, and our classifier-free guidance (CFG) effectively translates small embedding shifts into subtle changes in the score field. Additionally, OpenLex3D intentionally provides multiple synonyms per object. Previously, Table 5 reported results using only the first synonym; however, we have now re-evaluated using all synonyms for the Replica split, resulting in a minor change from 17.9% to 17.1\\% (-0.8%). This modest reduction highlights the robustness of our method. With performance variations below 1% for both synonym expansion and fuzzy spatial terminology, our promptable latent diffusion method demonstrates consistent robustness without relying on external LLMs, handcrafted scene graphs, or multi-view fusion. We will include this detailed analysis in our revised manuscript. We appreciate the reviewer’s request for a clearer articulation of novelty. Below, we explicitly enumerate our key contributions that distinctly extend beyond merely applying existing techniques in a new context. \n\n**Prompt-conditioned latent boxes:** As detailed in Section 3.2, existing diffusion-based detectors (e.g., DiffusionDet and Diffusion-ss3d) operate by diffusing raw box coordinates and thus cannot directly incorporate arbitrary language inputs or exemplar prompts. In contrast, we encode 6-DoF boxes into a compact 512-dimensional latent space via a Variational Autoencoder (VAE) and introduce cross-modal context through a shared token interface. This strategy explicitly decouples spatial noise from semantic control. \n\n**Unified “promptable” anchor design.** A unified mechanism of cross-attention transforms various modalities of prompts: class lists, few-shot exemplars, and free-form referring phrases into updates of the latent space score. This design allows our single checkpoint to seamlessly perform generic detection, few-shot transfer, and 3-D grounding tasks without head replacements or additional fine-tuning, a capability unmatched by existing 3-D detectors.\n\n**Multimodal classifier-free guidance (CFG) for 3-D boxes.** We generalize CFG to accommodate joint vision-language scoring, thereby enabling prompt-specific trade-offs between precision and recall. This feature is absent in current DiffusionDet variants and latent 3-D generation frameworks.\n\n**Task-agnostic training objective.** Unlike prior approaches, some necessitate task-specific control tokens; our training approach treats prompts solely as inputs. Consequently, switching between datasets like SUNRGB-D (closed vocabulary), ScanRefer (open vocabulary), or FS-ScanNet (few-shot learning) requires no additional or specialized loss functions."}, {"review_id": "ZFpMc6LUeT", "reviewer": "Reviewer_n1sa", "summary": "This paper proposes a \"promptable noise-to-box\" conversion paradigm, which improves the diffusion process from directly acting on the box coordinates to diffusing the latent representation of the box. This paper introduces a conditional latent diffusion framework that enables flexible and conditionally guided 3D object detection. In addition, it naturally extends to localizing object detection through natural language descriptions, and effectively generalizes to few-shot learning by using annotated examples as visual prompts.", "questions": "In addition to the weaknesses mentioned, please solve the following confusion:\n1. The boundary of generalization ability. Through the prompt mechanism, can the proposed method be generalized to object detection tasks of unseen categories. Similarly, can the proposed method be generalized to object detection tasks of different datasets, or even generalized across tasks?\n2. What are the advantages of the proposed diffusion process in the latent representation space compared to the diffusion model that directly acts on the box coordinates? Are there rigorous ablation experiments to support this?\n3. How many steps does the model denoise during inference? Can fewer steps be considered to increase the inference speed?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths** :\n1. A \"promptable noise-box\" paradigm is proposed to achieve flexible and condition-guided 3D object detection\n2. The diffusion process is applied to the latent representation space, and multiple object detection tasks can be completed by adjusting the conditional parameters only.\n3. Extensive experiments and ablation experiments have been conducted to verify the feasibility and effectiveness of the proposed method. It has shown excellent performance in various 3D object detection tasks.\n\n**Weaknesses**:\n1. Lack of model complexity comparison. In the main experimental results, I prefer to see the comparative analysis of different methods in terms of model parameter quantity, computational complexity and inference speed.\n2. Lack of visual comparison experiments. The paper lacks more intuitive visual comparison experiments with other models. Please add more detailed visual comparison experiments to more intuitively observe the detection performance between different methods.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers’ valuable feedback. Below, we provide detailed responses to each of the main points raised.\n\n---\n\n**Comment**: \"... the comparative analysis of different methods in terms of model parameter quantity, computational complexity and inference speed ...''\n\n**Response:**  Thank you for highlighting the need for a clearer analysis of computational complexity. We have now included comprehensive 3-D detection results on ScanNetV2 under the general object detection setting. Additionally, following reviewer n1sa's suggestion and as mentioned in the limitation section, we report two accelerated variants of our method (a 4-step and a 2-step LCM-LoRA model) for completeness. Given the tight timeframe for the rebuttal, the newly introduced hyperparameters (e.g., learning-rate warm-up and cosine decay schedules for the LoRA adapter) may not yet be fully optimized, and we anticipate further improvements with more thorough tuning. As demonstrated in the provided table, although our base model is relatively large, its end-to-end runtime matches or exceeds the FPS throughput of competing methods while consistently achieving higher accuracy. Moreover, the promptable design seamlessly extends to few-shot and grounding tasks without requiring retraining, underscoring the versatility of our proposed framework. Furthermore, existing engineering techniques, e.g., mixed precision, quantization, and channel scaling/sparse coding approaches like Matryoshka representations, could further enhance the efficiency of our method. However, we wish to emphasize that the primary focus of this paper is to introduce the promptable concept within a diffusion-based detection framework, thereby exploring the potential of diffusion models. Comprehensive details on latency, parameter count, accuracy, and discussions regarding scalability across sampling steps and GPU configurations have been incorporated into the revised manuscript.\n\n| Method            | mAP@50(%) (↑) |Model Parameters| Latency/scene (↓) | FPS (↓) |\n|-------------------|:-------------:|:-----------------:|:-----------------:|:-------:|\n| Diffusion-SS3D    |      64.1     |         -         |         -         |  30.07  |\n| Diff3DETR         |      65.7     |         -         |         -         |    -    |\n| **Ours (DDIM‑8)**|      **72.8**     |       1.03B       |       0.28s       |   3.60  |\n| **Ours (LCM‑LoRA‑4)**|      72.3     | 1.03B + 70M LoRA |       0.05s       |  22.23  |\n| **Ours (LCM‑LoRA‑2)**|      71.9     | 1.03B + 70M LoRA |       **0.03s**       |  **30.32**  |\n\n---\n\n**Comment**: \"... visual comparison experiments to more intuitively observe the detection performance between different methods.''\n\n**Response:**  Thank you for your comment! Due to conference policy, we are unable to provide additional qualitative results at this stage; however, we have already included qualitative examples in Figure 6 of the supplementary material. It is worth noting that the second and third examples illustrate the same scenes evaluated by Diffusion-SS3D (Figure 3 in the main paper and Figure 2 in the supplementary material). In the former scene, both methods yield comparable detection results for general object prediction, but our proposed approach further demonstrates the capability of predicting based on referring descriptions. In the latter scene, our approach achieves more accurate and flexible object detection, underscoring its versatility. We will explicitly highlight this comparison in the revised manuscript.\n\n---\n\n**Comment**: \".... The boundary of generalization ability. Through the prompt mechanism, can the proposed method be generalized to object detection tasks of unseen categories. Similarly, can the proposed method be generalized to object detection tasks of different datasets, or even generalized across tasks? ...''\n\n**Response:** We thank the reviewer for highlighting the importance of generalization. For unseen categories, our grounding branch is trained exclusively on ScanRefer, Multi3DRefer, and ViGiL3D, and is directly evaluated in a zero-shot manner on OpenLex3D, without any fine-tuning. As shown in Table 5, for the cross-dataset setting, our model achieves 19.5 / 17.9 (Acc\\@0.25/0.5) on Replica, 11.3 / 5.4 on ScanNet++, and 9.9 / 7.6 on HM3D, performing on par with more complex open-vocabulary baselines despite the domain gap. We specifically use ScanRefer prompts for evaluation on OpenLex3D, as ScanRefer provides natural language queries that are well aligned with our promptable latent diffusion approach. This setup offers the most challenging yet realistic scenario for evaluating out-of-vocabulary transfer. We will add these clarifications to the revised manuscript.\n\n---\n\n**Comment**: \"....What are the advantages of the proposed diffusion process in the latent representation space compared to the diffusion model that directly acts on the box coordinates? Are there rigorous ablation experiments to support this? ...''\n\n**Response:**  Thank you for the opportunity to clarify the motivation behind our work. Departing from the limitations of “noise-to-box” methods that operate directly on raw coordinates, our approach reformulates the problem within a learned latent space. Each bounding box is encoded into a rich representation that encapsulates both its geometric structure and the surrounding visual context, enabling a powerful and promptable latent diffusion model. As demonstrated by our experiments, this framework offers a unified and versatile solution for 3-D object detection, consistently performing well across a wide range of challenging scenarios.\n\nDiffusion-based detectors that perturb box coordinates function similarly to the original DDPM variant, which predicts $x$. These methods operate in a low-dimensional geometric space and are therefore hard to incorporate semantic prompts. Drawing inspiration from Conditional Latent Diffusion Models, we first encode each box with a VAE, where the box anchor is produced by fusing global scene information for greater robustness. This process ensures that the latent vector combines both geometric information and a CLIP2Point semantic anchor. The model then predicts noise $\\epsilon$ within this latent space. This redesign allows our approach to accept arbitrary language or exemplar prompts via classifier-free guidance and also removes issues related to scale and origin sensitivity. We provide two lines of empirical evidence in the paper. For cross-model comparison, under the standard ScanNetV2 protocol, our latent-diffusion detector achieves 60.3 mAP\\@0.5, outperforming the coordinate-diffusion baselines Diffusion-SS3D (43.2) and Diff3DETR (44.9) by a substantial margin of 16 to 17 percentage points. For within-model ablation, Figure 4 demonstrates that disabling latent diffusion (“w/o Diffusion”), while keeping the backbone, losses, and training schedule unchanged, results in a performance drop of 5.4 mAP on FS-ScanNet 1-shot and 6.4 F1 on Multi3DRefer. These controlled experiments isolate the specific contribution of latent-space diffusion. In summary, diffusing in latent space is not a superficial modification; rather, it is the essential factor that enables promptable and zero-shot generalization across datasets and tasks. In contrast, coordinate diffusion remains single-modal and less accurate."}]

Author rebuttal:

Meta-review: This submission proposes a promptable latent diffusion framework for 3D object detection. By encoding bounding boxes into a latent space and conditioning diffusion with multimodal prompts (text, exemplars, category lists), the approach unifies general detection, few-shot transfer, and language-guided grounding within a single model.

Reviewers agreed the paper is technically solid and novel in its promptable design, highlighting flexibility and robust evaluation. However, they also rasied concerns regarding practicality, including that the approach is computationally heavy, builds on large pretrained components, and its advantages over non-diffusion alternatives remain unclear. The rebuttal and discussions addressed efficiency, generalization, and robustness, convincing reviewers to maintain or raise borderline-accept scores. 

Overall, while efficiency and deployment issues remain, the work’s novel formulation and comprehensive results make it a valuable contribution likely to inspire future research in 3D detection.
Decision: Accept (poster)