Interactive Cross-modal Learning for Text-3D Scene Retrieval
Authors: Yanglin Feng
Yongxiang Li
Yuan Sun
Yang Qin
Dezhong Peng
Peng Hu
Venue: NeurIPS 2025 oral ()
Keywords: Cross-modal retrieval
interactive retrieval
large language model
Primary area: applications
TL;DR: We propose an Interactive Text-to-3D Scene Retrieval Method to handle inherent query limitations.

Abstract: Text-3D Scene Retrieval (T3SR) aims to retrieve relevant scenes using linguistic queries. Although traditional T3SR methods have made significant progress in capturing fine-grained associations, they implicitly assume that query descriptions are information-complete. In practical deployments, however, limited by the capabilities of users and models, it is difficult or even impossible to directly obtain a perfect textual query suiting the entire scene and model, thereby leading to performance degradation. To address this issue, we propose a novel Interactive Text-3D Scene Retrieval Method (IDeal), which promotes the enhancement of the alignment between texts and 3D scenes through continuous interaction. To achieve this, we present an Interactive Retrieval Refinement framework (IRR), which employs a questioner to pose contextually relevant questions to an answerer in successive rounds that either promote detailed probing or encourage exploratory divergence within scenes. Upon the iterative responses received from the answerer, IRR adopts a retriever to perform both feature-level and semantic-level information fusion, facilitating scene-level interaction and understanding for more precise re-rankings. To bridge the domain gap between queries and interactive texts, we propose an Interaction Adaptation Tuning strategy (IAT). IAT mitigates the discriminability and diversity risks among augmented text features that approximate the interaction text domain, achieving contrastive domain adaptation for our retriever. Extensive experimental results on three datasets demonstrate the superiority of IDeal. Code is available at https://github.com/Yangl1nFeng/IDeal.

Interactive Cross-modal Learning for Text-3D Scene Retrieval
Yanglin Feng', Yongxiang Li', Yuan Sun', Yang Qin', Dezhong Peng Peng Hu'
'College of Computer Science, Sichuan University, Chengdu, China.
"National Key Laboratory of Fundamental Algorithms and Models for Engineering Numerical Simulation, Sichuan University, Chengdu, China. 'Tianfu Jincheng Laboratory, Chengdu, China.
fcyzfy10163. com, aaythmli.scu@@maill com, Sanyuan_work@lee com, qinyang gg@ggail.. com, pengdz@scu. edu. en, penghu m1 mw0mmmall...mmmmm com
Abstract
Text-3D Scene Retrieval (T3SR) aims to retrieve relevant scenes using linguistic queries. Although traditional T3SR methods have made significant progress in capturing fine-grate -rainnd associations, they implicitly assume that query descriptions are information-complete. In practical deployments, however, limited by the capa- bilities of users and models, it is difficult or even impossible to directly obtain a perfect textual query suiting the entire scene and model, thereby leading to perfor- mance degradation. To address this issue, we propose novel Interactive Text-3D Scene Retrieval Method (IDeal), which promotes the enhancement of the alignment between texts and 3D scenes through continuous interaction. To achieve this, we present an Interactive Retrieval Refinement framework (IRR), which employs a questioner to pose contextually relevant questions rounds that either promote detailed probing or encourage exploratory divergence within scenes. Upon the iterative responses received from the answerer, IRR adopts a retriever to perform both feature-level and semantic-level information fusion, facilitating scene-level interaction and understanding for more precise re-rankings. To bridge the domain gap between queries and interactive texts, we propose an Interaction Adaptation Tuning strategy (IAT). IAT mitigates the discriminabilitty and diversity risks among augmented text features that approximate the interaction domain, achieving contrastive domain adaptation for our retriever. Extensive experimental results on three datasets demonstrate the superiority of IDeal. Code is available https: / thub. com/Yang] 1nF /Fnng//an IDeal.
answerer in successive
Introduction
Recent years have witnessed natural language interfaces to embodied intelligence systems [1, 3, 4, 5] become increasingly prevalent in our daily lives. This opens up further opportunities for natural language-based interaction with intelligent agents, such user verbally instructing agents to perform tasks in a specific scene. Before executing any task, an agent must first retrieve the scene relevant the user's intent. This requirement has spurred recent works on Text-3D Scene Retrieval (T3SR) [6. 7], which enables the retrieval of 3D point-cloud scenes using linguistic queries. Such a language-scene alignment capability lays critical foundation for enabling agents to generalize across scenes and environments.
Although existing dedicated methods [6] achieve promising performance for T3SR by facilitating fine- fine-grained query text and scene understanding, such success often relies on the assumption that
*Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025)

---

Raund 0
Intial
Interaction
Tuo long serieene Never seen mattress, dumbbeak
What color bed?
Quectioney
brown bad.
Answerer
Retrieven
Bed
FIND bed my
white matties ess dumbbell..
a
What is /
square
ScaneDa5s
Answerer Dus
Retriever
Chollenge I: Lock holistic interoction perspectives.
Challenge Domain between queries and interaction texts. 100
Uniformtttvve
Round i+1
on brown bed
white
Informatvve rerough
Round I+2
Round i+2
Bed Dumbial
with dumbbell
gloced
Provide mone detalls bbout bed.
What object opposita side of bed??
Questioner
Answerer
Questioner
C
Arsswerer
without
opposite the bed
Retriever
Scene0073
Figure 1: Overview of interactive text-3D scene retrieval. The above gray part illustrates the two specific challenges, while the below white part shows the illustrations of our interactive framework.
the queries provided are information-complett However, such an assumption is often violated in real-world scenarios due to the inherent limitations of text inputs and models, such as incomplete one-shot descriptions of user intent [8], ambiguous descriptions [6], domain shifts [9], and limited generalization of the models. As a result, the performance and robustness the models remain persistently constrained, and relying solely on limited internal knowledge is insufficient to overcome this inherent bottleneck.
To break through the bottleneck, recent studies [8, 10, 11] have explored integrating external knowl- edge from Large Language Models (LLMs) to enhance their understanding and alignment abilities. However, such approaches typically require prohibitively expensive fine-tuning or retraining of offline models [10, 12]. Some other attempts [8, 13, 14] have proposed interactive cross-modal retrieval frameworks that incorporate LLMs and Vision-Language Models (V (VLMs) to facilitate more fin-grained understanding and alignment, thereby iteratively evolving the retrieval performance. Although these methods have demonstrated remarkable effectiveness in image-text matching [8] and video-text retrieval [15, 16], they still face two tricky challenges in the T3SR setting, as shown in Figure 1. Firstly, since the scale and complexity of 3D scenes, these methods lack holistic perspec- tives beyond a localized focus during interaction, e.g., the LLMs tend to focus on the salient objects in the scene and ignore the fine ane-grained details scene-level perspective, limiting the depth and breadth of LLM interaction, as demonstrated in Table 1. Secondly, existing retrieval models exhibit limited generalization ability biased text domains, limiting their effectiveness in handling realistic interaction texts that exhibit domain gaps.
To address the aforementioned challenges, this paper proposes novel Interactive Text-3D Scene Retrieval method (IDeal) to conduct continuous interaction between the T3SR models and external users (e.g., LLMs), achieving the active alignment between text queries and 3D scenes, as depicted in Figure 1. Our IDeal consists of two components: an Interactive Retrieval Refinement framework (IRR) and an Interaction Adaptation Tuning strategy (IAT), as illustrated in Figure 2. More specifically, IRR coordinates three specialized agents (i.e., questioner, answerer, and retriever) to perform multi- round interaction. First, the questioner adaptively determines whether to continue probing object details or to pursue divergence by exploring the broader scene, based on the assessment of the current round's description. Based this, it continuously formulates context-relevant questions to the answerer. After receiving responses, the retriever iteratively integrates information at both the feature and semantic levels, facilitating comprehensive scene-level understanding for progressively precise re-rankings. To mitigate the domain shift between training queries and interactive texts, IAT proposes adapting the reiriever toward the interaction text domain. Specifically, IAT leverages LLMs to generate more realistic augmented texts that closely resemble the interaction text domain. Subsequently. IAT robustly mitigates the discriminability and diversity theoretical risks in the features of the augmented texts for domain gap bridging, thereby ensuring an unbiased understanding of the interaction texts by the retriever. The contributions of this paper are as follows:
We propose a novel Interactive Text-3D Scene Retrieval Method (IDeal), which actively enhances alignment between text queries and 3D scenes through ongoing interactions.
An Interactive Retrieval Refinement framework (IRR) is presented to enable a deep interaction for comprehensive scene exploration, leading to progressively improved retrieval.
An Interaction Adaptation Tuning strategy (IAT) is proposed, which facilitates the transfer of the retriever to the interaction text domain, promoting improved interaction.
We conduct extensive comparison experiments on text-3D scene datasets. Our IDeal remarkably outperforms the existing methods, demonstrating its superiority.

---

2 Related Work
Cross-Modal Retrieval. Cross-Modal Retrieval (CMR) [17, 18, 19, 20, 21, 22] aims to match corresponding results across modalities for a given query, bridging the gap caused by modal het- erogencity. In recent years, CMR has garnered significant attention in fields such as Image-Texx Text Retrieval [14, 23, 24, 25]. Video-Text Retrieval [26, 27], 2D-3D Retrieval [28, 291, Pointcloud- Text Matching [6]. The primary challenge of CMR lies effectively aligning multimodal data. To address this issue, most existing works could be broadly categorized into two groups: 1) Coarse-graines retrieval [30, 31, 32] directly maps multimodal data into a shared space, aiming for a more straight- forward and computationally efficient alignment. 2) Fine-graine retrieval [33, 34] seeks to establish local associations between fine-grained features across modalities (e.g., regions in images, words in texts). These local associations are then progressively integrated form precise cross-modal correspondences. This paper focuses on a more challenging CMR task, i.e. Text-3D Scene Retrieval, involving obscure spatial cues and sophisticated 3D scenes (including issues such as viewpoints and occlusions [35, 36]). Although prior work [6] performs well with comprehensive descriptions, it struggles with online and blurty queries. To this end, we propose an Interactive T3SR solution that iteratively incorporates online feedback to achieve more precise and practical scene retrieval.
Interactive Learning. Unlike traditional learning paradigms [6, 37], interactive learning emphasizes the continuous improvement of a model's s behavior through ongoing interactions with the environment or users. Specifically, several pioneering works [38, 39] leverage simple forms of user feedback (e.g.. preferred sample selection and relevance scoring) to iteratively achieve improved training quality or better satisfy user-specific requirements during testing. With the development of Large Language interactions
Models (LLMMs) other studies [8, 40, 41] have begun exploring question-answere through free-form text dialogue, closely replicating natural human communication. For example, several methods leverage iterative interactions to continuously refine the retrieval query for better reranking. Recently, more advanced methods such as PlugIR [13], MERLIN [16], ICL [42], and LLaVA-ReID [43] have integrated LLMs for context-aware question generation, mining more visual details. However, these methods cannot be effectively generalized T3SR due to the differences in tasks and data domains shown in Figure 1. In this paper, we develop an interactive framework tailored for T3SR to help the offline models adapt to complex scene perception.
3. Method
3.1 Problem Formulation
Given a text query set T (4))21 and a 3D scene gallery C = {cz}}=1, where 4, and 6j represent i-th text and j-th scene, n, = |T| and n. = [C] means the sample number, and I denotes the volume of set. The purpose of T3SR is to use the text query to match the ideal 3D scenes from the gallery, where there exists correspondence Yi.. E {0, 1}. indicating whether the points are matched (i.e., Biy = 1), unmatched (i.e., Vij = 0). Existing methods [6, 37] typically train an offline model to achieve encoding of multimodal data, followed by meticulous single-turn retrieval. They assume that user-prout wer-pproidee text queries information complete, overlooking the practical fact that queries are often partial, ambiguous, or even exhibit domain shift.
To address these issues, an interactive Text-3D scene retrieval method, i.e., IDeal, is proposed to bridge the interaction between the retrieval models and external agents, progressively overcoming the aforementioned query limitations and achieving improved alignment between texts and 3D scenes. More specifically, IDeal asks question of (l E (1. ,)} about i-th sample based on the users' previous response a !-1 (af = tz), where r is the upper limit of rounds. Subsequently, the external agents recall details and answer a' of the target 3D scene e (21 = 1), forming a dialogue context pairs (g. a;). Both the j-round response text and 3D scenes are projected into a shared feature space by trained retrieval model, which can be: u. (a 0) and v. fr 0), where 0 denotes the learnable parameters.
D. = {aL. (g. a.) ...
.. (of a composed of question-
3

---

Retriever
PD Poirr cicud Encodor
Shored space
Reature-level Fusion
Mhetbomatioe /epomstte
fourday, Sit
Amsd tros/ Itrgett
Covin' set
Init (tsl: There t.s
Project
Combised
+>
Weighted
White
Ita hece
ao
Teat Frooddr
Semontioirnen Fuxion
barkk utthoue
Centrol sppreyooos
Asorage fustos
Initial Query and r- Round Answers
Semontic
Questioner
Rosder
In
Ot
Object
Probe for detoils
Diverge to explore
Memorgg the bed dbrnelainng the wall aithaate ra
Objects and ctok, cotinet nutals hit durdlas
OLs Thene is bed here
Scenee 4-LLM Reconstructio
right corear pui on
9r
10
Answerer (LLM / User)
Wt what color ore bed anylhng on the bes?
Anay from ancohbig
dhaaney, dioi...
aloog the et nights the
with whit calend t4a.
orptaseer lg)
locng wwwyy from
@+++
ihe Iesl
Motching
Figure 2: (Left) Pipeline of the proposed IDeal. A questioner employs router to adaptively pose either probe or divergent questions, which require responses from an answerer. After receiving iterative responses, retriever projects the multimodal data into shared feature space, performing feature/seemantic-evell fusion (Right) to enable progressively precise scene retrieval.
3.2 Interactive Retrieval Refinement Framework
In this section, we introduce the Interactive Retrieval Refinement framework (IRR), which coordinates three agents (i.e., questioner, answerer, and retriever) to enable iterative interaction. In the following sections, we will elaborate on them by introducing their interaction process.
3.2.1 Adaptive Questioning
To achieve a deep interaction for T3SR, we present an adaptive questioner, which enables pertinent questioning the answerer for both detailed and comprehensive exploration of complex 3D scenes. Firstly, a question router is employed, which determines the focus of questioning based on the feature distribution in the shared space. More specifically, the router assesses whether the previous-round description is informative by computing Cross-modal Affinity Entropy (E), formulated as:
E(u = Σ
p(u ,,,) log 11 ,1y),
JENx(u
where ( means the k-nearest-neighhor index of ut- and affinity probability p(u] U,) formulated as:
exp(S(u,
p p(u 0.)
(2)
EiEN((uE-)) exp(S(r, ..)/7)'
where S represents the computation of similarity between features, and 7 a temperature parameter.
However, the distribution of scene features extracted by the trained retrieval models is fixed and inherently non-uniform, with regions of over-density and under-density introducing structural density bias [44]. To mitigate the bias, we introduce a Density Compensated Factor for each scene feature, which is formulated as: p(v.) (I/R) [sE p(e.,,))+e' where D represents the distance
calculation and e a minimal constant for numerical stability. Based on this, we try to approximately correct the original similarity score S(u, (1;) by incorporating the Density Compensated Factor p(ex), which could be written as: S(u (r-1) vy) = S(u (r-1) v)),))(),) Subsequently, this corrected similarity is brought back into Equations (1) and (2) obtain a Density Compensated Affinity Entropy, denoted as E. This process approximates a Bayesian Correction leveraging prior density estimation mitigating the impact of the inherent bias in scene feature distribution.
Leveraging the fairer metric E, our questioner categorize descriptions with E 3 as uninformative. prompting an LLM to generate questions for detail probe (i.e., Q1) for attribute and spatial relationship detail refinements within the described area. Conversely, when E < B. the descriptions are considered informative, triggering the adopting of questions for diverge exploration (i.e., Q2) that inquire about object arrangements not previously discussed in the dialogue.
2 Pleesse refer to our Supplemental Material for further discussion.

---

3.2.2 Iterative Retrieval
After completing the questioning, an LLM is employed to simulate the external user acting as an answerer to answer the questions, following existing interactive approaches [8, 13]. It receives multi-round questions and provides responses based on its memory. In this paper, we adopt text modality to simulate the memory, which better approximates how humans recall information mind.
Upon receiving the response descriptions from the answerer, we construct a relriever that can be seamlessly integrated with existing cross-modal models [37, 45], enabling iterative scene retrieval. It can obtain the scene retrieval predictions for any given text within the shared feature space. Specifically, given text feature u, the prediction can be formulated as:
p(u,) = ip(u; 11), p(u; u2),..., p((u, vn.)]l
where p(u (,) exp(S uu, u;))/ Ei exp(S(u;, vi)) denotes the probability that the i-th text retrieves the j-th scene. Accordingly, our retriever first utilizes the initial query to compute initial retrieval prediction P1 (.;)) Subsequently, the feature-level and semantic-level fusion of interactive responses is conducted to achieve more precise scene retrieval.
For the fusion of interactive response feature, on one hand, considering responses to Q1 are re- finements of the previous-round descriptions, supplementary information. This strategy enables the preservation of core semantic cues from previ- ous rounds while emphasizing newly introduced details: u au? + (1 aa where of E Qi, is a trade-off weight. On the other hand, benefiting from 22, the other response features and the aforementioned fused features capture variations across different regions of the scene. To fuse them into a comprehensive feature, inspired by [46], we model their distribution around the target scene by encapsulating them within a minimum enclosing hypersphere:
apply weighted linear fusion to incorporate
(o), R = arg min R,
{ Rs w.oo S S.,, vi}.
(4)
where of and R are the center and radius of the hypersphere, respectively. Based on this, features near the hypersphere boundary are grouped into a boundary set while the remainder constitute the central set 212 To balance fusion robustness and feature discrimination, potentially noisy boundary features in 21 aggregated at the hypersphere center and averaged with cleaner features in 21 to
yield the final fused response feature: ú; = of + मी Enfeup ut] This fused feature is then input into Equation (3) to obtain an interactive feature prediction P2(ū;)
However, the aforementioned feature-level fusion responses. To address this limitation, we leverage an LLM to reconstruct the 3D) scene from all re- sponses in textual space. More specifically, inspired by Chain-of-Thoone Thought (CoT) [47], decompose this process into object extraction and scene reconstruction for a more stable and comprehensive scene summary. The LLM first identifies the scene objects across multi-round responses and then summarizes an object-centric scene reconstruction text. Finally, the texts are encoded into feature 8,, from which a interactive semantic prediction pa(s), (8:) computed using Equation (3).
fully capture the holistic semantics of the
Finally, the initial and interactive predictions are combined through weighted fusion to obtain the final scene retrieval prediction follows:
Pa (u:) X (u.) + Aap2 (ū:) + Aapa(s:),
where (u(u) is the final retrieval prediction, \1. X2, and A are trade-off parameters. Benefiting from the adaptive questioning and the comprehensive retrieval information fusion, our IDeal can alleviate the limitations of initial queries through progressive interaction.
3.3 Interaction Adaptation Tuning
Although IRR can exploit interactions to promote retrieval quality, the limited text domain of the retriever remains a bottleneck that restricts further performance improvements. To overcome this limitation, we propose Interaction Adaptation Tuning strategy (IAT), which enhances texts to approximate the domain of interaction texts.
We begin by integrating information and descriptions of the same scenes from the training data to construct simulated memory for text augmentation. Following IRR, we first provide a training- based answerer (i.e., LLM) with the constructed memory and initial queries. We then simulate the
5

---

IRR interaction process by iteratively posing fixed number of Q1 and Q2 questions. The response descriptions yield augmented texts that closely approximate the interaction scenario.
After obtaining the enriched augmented texts, inspired by the contrastive domain adaptation paradigm [48, 49], we try to minimize the theoretical risk R(0) associated with our retriever among the augmented text features, as formulated below. It facilitates model adaptation of the retriever to the augmented text domain without requiring access its implementation details.
R(00- Rdio (0) + (o) Bu (5:1((25 1(-Ea+ {S(ū, ū.))) + (Eg- { (6)
where the two components R.tis (0) and R.tiv (0) respectively reflect discrininability and diversity risks, E denotes expectation, Eyy is taken with respect to the distribution for the target domain features (i.e., augmented text features U = {u, ) and ü+ and ū is the distribution for the corresponding positive features र. and negative features ü, respectively.
On the one hand, minimizing the discriminability risk R.dis (0) requires encouraging the augmented text features to align closely with those belonging to the same scenes. However, the scale and complexity of scenes often lead substantial variability even among features corresponding the same scenes. This introduces significant uncertainty in the selection of positive samples, complicating the risk optimization process.' To handle this, we adopt the corresponding 3D scene features as substitutes for the text features to construct positive pairs. This is based on the assumption that the scene features encoded by the well-trained model are more stably located near the center of the corresponding description distribution. Finally, we mitigate the aforementioned discriminability risk RRis(O) by minimizing a negative log-based proxy loss term Lais, which could be written as follows:
b
Lase ΣV
Yi log S(üi, Uz),
i=1
where is the size of the mini-batch. On the other hand, motivated by [50], we attempt approximate the minimization of the divergence risk (aiio()) by minimizing its upper bound, i.e.,
sup (Rdiv (0)) ~ {Ea- (S(ū, ū ū) Va-nu- -ū- (S(ū, u }
(8)
where sup(-) means the upper bound and V is the variance. We can obviously see that the divergence risk is affected by the mean and variance the selected negative samples. Yet existing methods [48] negative samples for contrastive learning, thereby
usually treat others within the same mini- batch minimizing the expectation term. Due to the stochasticity of mini-batch sampling, similar samples may be mistakenly chosen negatives, which increases the variance of negative samples, enlarging the upper bound the divergence risk.
To tackle it, we propose weighted complementary contrastive loss as surrogate objective to achieve divergence risk optimization more robustly, which can be formulated as:
Ldiv
exp
max (0, S(,,, úy) - )) log (1 - Sūū ú,))
im
Weighting
Complementary contrastive
where y is a threshold, above which samples are assigned lower weights. Minimizing the complemen- tary contrastive term optimizes the expectation can mitigate the impact of hirh-variance -eegatve samples.
negative pairs, while the weighting component
Finally, we combine both terms to obtain our loss for domain adaptation tuning, as follows:
L = ALdia + (1 - Ld
(10)
where À is hyperparameter to control the contribution of each component. Minimizing this proxy loss facilitates the reduction of domain adaptation risk, thereby enabling the retriever to better adapt to the domain of interaction text.
The analysis can be found in Supplemental Material.

---

Table I: Performance comparison ScanRefer, Nr3D, and Sr3D in terms of @@) R@5, R@ and their sum (Rsum). denotes the use of coarse-graines descriptions as memory.
Methods
SeanRefer
Nr
3r
R@D R@S R@ 10 Rsum R@I R@5
R@ 10
Rsum R@ I R@S R@ 10 Rsum
w/ coarse-graineee grained descriptions:
VSEco CVPR'2
9.7
33.1
50.2
93.0
5.8
21.5
32.5
59.8
5.5
18.9
27.4
51.8
CHAN (CVPR 23)
9.4
32.3
52.1
93.8
7.5
15.0
32.1
54.6
5.4
21.3
34.8
61.5
HREM (CVPR
10.2
34.0
51.4
95.6
7.3
18.1
31.9
57.3
5.4
21.9
33.6
60.9
CRCL [NurlPS2223
10.3
32.4
49.8
92.5
8.1
22.5
33.2
63.8
4.9
19.5
31.9
56.3
RoMa (TMM'25)
11.4
34.8
54.4
100.6
6.5
24.8
37.6
68.9
7.8
27.3
39.3
74.4
IDeal
16.0
42.7
59.8
118.5
11.7
34.8
50.4
96.9
10.3
30.2
48.5
89.0
w/ fine- line-grained descriptions:
ChatlR (NcurIPS'22
21.8
55.4
73.1
150.3
15.6
40.3
58.1
114.0
12.1
35.9
50.3
98.3
Rewrite I (ICMR '24)
17.4
47.0
63.7
128.1
17.1
31.4
45.8
94.3
12.4
28.1
40.4
80.9
MERLIN' (EMNLP'24)
31.1
68.8
83.8
183.7
21.8
55.0
71.8
148.6
14.2
42.0
60.3
116.5
BASELINE: IR
29.9
68.0
83.3
181.2
18.0
51.6
60.2
129.8
14.6
39.2
55.3
109.1
BASELINE SUM
34.4
69.5
85.1
189.0
22.5
55.2
67.5
145.2
16.4
41.5
62.1
120.0
IDeal
37.8
71.8
86.4
196.0
26.4
62.7
78.7
167.8
20.2
43.1
63.1
126.4
4
Experiments
4.1
Experimental Setting
Datasets, baselines, and evaluation metries: We adopt the ScanNet 3D scene set along with several description sets (i.e., ScanRefer [51], Nr3D [52], Sr3D [52], and Scene experiments, where ScanRefer, Nr3D, and Sr3D are employed as query sets, and SceneDepict-3D 3D2 employed simulate fine-grained memory. To verify the superiority of our IDeal, introduce eleven comparative baseline methods: five conventional offline cross-modal matching methods (i.e., VSExo [45], CHAN [53], HREM [54], CRCL [37], and RoMa [6]), three interactive cross-modal retrieval methods (i.e., ChatlR [8], Rewrite [41]. and MERLIN [16]), and two additional strong interactive baselines (IR and SUM). More specifically, the Iterative Reranking (IR) involves multi- round interaction, where the results are iteratively re-ranked based on the response of each round. The Summary reranking (SUM) also involves interaction, but ultimately aggregates all answers into comprehensive description for matching.
[6]) to conduct
In addition, follow [55, 56] to re- port R@I, R 5, @ 10, and their summation (Rsum) as the evaluation metrics. Due to the space limitation, more details of datasets, prompts. and additional experiments are provided in the Supplemental Material.
Table 2: Performance comparison on ScanRefer and Nr3D in terms of R@ 1, R@ R,5, R@ 10, and their sum. +IDeal indicates plugging the model into our IDeal. t denotes the use of fine-grained descriptions memory.
Methods
ScanRefer
Nr3D
R@1
R@5
R 10
Rsum R@
R@5
R@
10
Rsum
VSEcc
9.7 33.1
50.2
93.0
5.8
21.5
32.5
59.8
All meth- ods are implemented in PyTorch and carried out on GeForce RTX 3090 GPUs We adhere to the experimen- tal settings of [6] for all method im- plementations. We adopt widely-used DGCNN [57] and BERT [58] to ob- tain fine- tiee-graindd features for 3D point clouds and texts, respectively. To im- plement interaction, we explore two approaches to constructing memory:
Implementation details:
+1Deal
13.3
38.9
57.6
109.8
8.7
27.5
42.1
78.3
VSEx
14.9
42.3
61.5
118.7
16.4
47.5
55.2
119.1
+IDeal
35.8
70.6
85.0
191.4
21.2
52.1
68.4
141.7
CRCL.
10.3 32.4
49.8
92.5
8.1
22.5
33.2
63.8
+IDeal
13.4 35.5
56.1
105.0
7.4
25.4
38.3
71.1
CRCL
17.5
45.1
58.3
120.9
13.4
44.5
51.5
109.4
+IDeal
31.7
66.9
83.5
182.1
15.8
50.4
64.4
130.6
RoMa
9.7
33.1
50.2
93.0
8.3
27.9
37.2
73.4
+IDeal
16.0
42.7
59.8
118.5
11.7
34,8
50.4
96.9
RoMa
16.7
44.8
61.6
123.1
17.4
48.5
57.5
123.4
+IDeal
37.8
71.8
86.4
196.0
25.4
60.7
75.7
161.8
1) Coarse grained description: We leverage an LLM to generate rich expansions of queries, serving as memory without introducing any additional information leakage. 2) FFin-ggrain description: In line with existing interactive methods [43, 13), we simulate the user's memory in real-world scenarios using fine-grained scene descriptions, albeit with access to partial additional information. In our experiments, we utilize Qwen- Syven-7B-Instrue [59] as our investigated L.LM for the interaction experiments.
7

---

4.2 Comparison on Text-3D Scene Retrieval
Table presents a comparison between our IDeal and conventional and interactive cross-modal matching methods under two memory settings. Table 2 further demonstrates the performance gains brought by integrating our interactive framework into conventional single-round methods. These results could yield the following observations: 1) Even without additional fine-grained information, our Ieall achieves competitive performance, highlighting its ability to uncover complementary information implicitly embedded in the queries, gradually alleviating inherent query limitations. 2) Compared to existing interactive methods, our IDeal also achieves superior performance with access to fine- geee-gra descriptions. This demonstrates that our interactive questioning and retrieval strategies enable an ongoing understanding complex scenes. 3) Our IDeal can be seamlessly integrated into conventional cross-modal retrieval methods and yields substantial performance gains under both memory settings. This suggests that, beyond equipping offline models with interactive capabilities, IDeal empowers them more effectively comprehend complex descriptions through interaction.
user requirements and support comprehensive interpretation of
4.3 Ablation Study
In this section, conduct ablation study to evaluate the contribution each proposed compo- nent IDeal. Specifically, first ablate the router in the questioner, restricting to ask either Qi or Q2 continuously. In addition, remove each of the three retrieval prediction strategies, and we examine removing CoT prompting the construction in the proposed retriever. Finally, we investigate the effect of using the IAT strategy for domain adaptation and sequentially ablate its two loss terms. The results in Table lead to the following observation: 1) Removing replacing any component from IDeal results performance degradation, highlighting the contribution of each component. Specifically, the adaptive questions generated by questioner facilitate meticulous and comprehensive understanding of scenes. The various feature aggregation strategies in the retriever contribute to precise scene matching. 2) Removing or substituting IAT components consistently leads to performance degradation, underscoring the necessity of text domain alignment and adaptation risk minimization in our IAT.
Table 3: Ablation studies for components of our IDeal ScanRefer. RSum is the sum of R@ R@5, R@ 10. w/o stands for without use.
Configurations
SeanRefer
R@1
R@5 R 10 Rsum
Questioner
w/o 8:
36.0
71.2
86.7
193.9
w/o
26.3
61.5
77.3
165.1
w/o p (u.)
35.2
70.1
86.5
191.8
Retriever
w/o Pa u.
28.1
63.3
80.6
172.0
w/o Pa (x:)
31.8
67.8
84.2
183.8
wo CoT
35.7
71.5
85.9
193.1
w/o LAT
16.6
48.4
64.4
129.4
Adaptation
w/o Ldis
34.9
69.5
84,4
188.8
w/ Liie
35.1
69.4
84.1
191.7
Full IDeal
37.8
71.8
86.4
196.0
16.3
16
17.4
16.0
25
03
26.3 600
68
4
25
25
25
602
(a) 0 retriever.
(b) A1. λ2. As in retriever.
(c) ls, questioner.
(d) X, in
Figure 3: PTM performance in terms of R @ versus different values of the parameters of our IDeal on ScanRefer. (a) and (b) display a, A1. Ag. and Ag in our retriever. (c) shows and in our questioner. (d) shows À and y in IAT.
4.4 Parameter Analysis
To evaluate the sensitivity of our IDeal to different hyperparameter settings, we plot the retrieval performance versus different values on ScanRefer, as shown in Figure 3. The experimental results lead to the following observation: 1) For our retriever, tuning greater weights to interactive and reconstruction predictions helps achieve a well-balanced trade-off that fully leverages the interactive responses. Additionally, a higher feature fusion weight (e.g., 0.75) represents emphasis

---

25
30
R191
20
10
0
5
6
10
0
Round
Round
Round
Round
(a) ScanRefer
(b) Nr3D
Figure 4: Performance (R@ I) versus rounds on two datasets. Round indicates the setting without interaction. The green and purple bars represent the cases with coarse saanse-raaieed and fine-grained memory descriptions, respectively. The lighter bars indicate no performance gain.
the integration of discriminative features from the refined descriptions, leading to more effective interaction feature fusion. 2) For our proposed questioner, using reasonable and moderate settings of k and (e.g.. k 20, 2.0) enables accurate identification of informative descriptions, thereby supporting reasonable decisions on question types in the next round. 3) During domain adaptation tuning, relatively wide range of and y values in IAT (i.e., À [0.2, 0.5) and y E [0.1, 0.8) ensures effective contrastive adaptation and mitigates the impact of false negatives.
Inttial Query
[Ronk
Cose
Initial Query
(Ronk 23
theve cussas be other montor abo just of and dirgenaly
close act
Round Answer
Round 2 Answerr
[Ronk 8]
with othen the right
brownpaten rosis in duvere oner whith it wey cher the wall mhe sinpe med i plone dgtt the well lnderttoon any seldiau doar bellng (tall) men ehoggite
sumounded by oddiieel maston jver and elogreely
thove actorr aloued rdde by alle prosidin emple wortapeer.
[Ronk
Round 2 Answer [Ronk 2)
Case
Acswer
Ours
initial Query
Ground Trutht
Round Answer [Ronk 5)
Merlin
Round Answer
|Rank 4)
desccletio me none wher tren the
gray
thov o reterpaler
pidge heve a bachh
[Rank 1]
ther of ehe
woll thm
spperr to or iuprasventortt yarer abutrctt deniede
The paliting
charr loble rosd rockese
[Rank 4) scurel char...
ois restisee that stere oroy tark can rggtt rukle addnonslly
Round' 5 Answer
UpU
orr uuemal musiiigg chais with troen cenaaninn the Jonc.
traat right pointing.
with somee crettt betteen Hem
ohe that mose conncco
solecing we talin shout the other
rgtt sovitig.
color othewe mf voee owver in cnstter maney tosed
Round Answer
The round buskea Sonc. dN surronsaing aetordlctat..
[Rank 4]
[Rank
Figure 5: Case illustrations of interactive cross- scene matching process of IDeal on ScanRefer. Cases 1-2 and are with coarse coarse-grained grained and fine- graingd memory descriptions, respectively.
4.5 Visualization Analysis
To provide comprehensive analysis of IDeal, we conduct series of visualization experiments. Specifically, we first present the changes in retrieval performance of IDeal across multiple rounds of interaction, shown in Figure 4, to analyze the incremental gains brought by each interaction. In addition, we visualize several representative cases, as illustrated in Figure 5. The observations can be drawn from the results: 1) Interaction consistently improves performance over the first several rounds (five rounds with fine-grained and three rounds with marse--grinne descriptions). Although redundant interactions may inevitably cause performance to saturate even slightly degrade due to LLM hallucinations or excessively long texts, these results indicate that a reasonable number of interaction steps can effectively enhance the query and improve retrieval performance. 2) Under the setting with coarse-grained memory texts, IDeal can infer and decompose object attributes and relationships within the queries through interaction, leading improved retrieval performance. Moreover, with the integration of fine-grained memory, IDeal leverages targeted and switchable questioning to elicit informative responses, continually improving retrieval precision. In contrast, MERLIN [16] frequently generates redundant descriptions confined to local details of scenes.
5
Conclusion
In this paper, we propose a novel Interactive Text-3D Scene Retrieval Method, namely IDeal, to address the Text-3D Scene Retrieval (T3SR). Our IDeal integrates two components: the Interac- tive Retrieval Refinement Framework (IRR) and the Interaction Adaptation Tuning strategy (IAT). Specifically, IRR continuously conducts adaptive questioning and comprchensive response fusion,

---

enabling holistic exploration of 3D scenes for more precise retrieval. IAT performs contrastive domain adaptation for the retriever toward realistic texts, overcoming the performance bottleneck during interaction. Extensive experiments demonstrate the superiority of our IDeal in T3SR task.
Limitations and Potential Impact Statement: Although our work has taken the initial step forward in interactive T3SR, there are some limitations and potential impacts that should be acknowledged First, the performance of the methods is relatively low. Second, we employ LLMs, and more stable and unbiased LLMs and interaction approaches merit further exploration in the future.
Acknowledgments
This work was supported in part by NSFC under Grant 62472295 and 62372315; in part by the Funda- mental Research Funds for the Central Universities under Grant C1202403; part by Sichuan Science and Technology Planning Project under Grant 24NSFTD0130 20247IDZX0004, 2024NSFTD0049, and in part by the Chengdu Science and Technology Project under Grant 2023-XT00000004--GX
References
[1] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d ssiio-language-acttin generative world model. arXiv preprint arXiv:2403.09631 2024.
[2] Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911-2921, 2023.
[3] Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang. Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. In European Conference on Computer Vision, pages 289-310. Springer, 2024.
[4] Huajian Huang, Changkun Liu, Yipeng Zhu, Hui Cheng, Tristan Braud, and Sii-Kit Yeung. 360loc: A dataset and benchmark for omnidirectional visual localization with cross-device queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 22314 223244,, 2024.
[5] Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023,
[6] Yanglin Feng. Yang Qin, Dezhong Peng, Hongyuan Zhu, Xi Peng, and Peng Hu. Pointcloud-text matching: Benchmark dataset and baseline. IEEE Transactions Multimedia, 2025.
[7] Jiaqi Chen, Daniel Barath, Iro Armeni, Marc Pollefeys, and Hermann Blum. "where am i?" scene retrieval with language. In European Conference on Computer Vision, pages 201-220. Springer, 2024.
[8] Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski. Chatting makes perfect: Chat- based image retrieval. Advances in Neural Information Processing Systems, 36:61437-61449, 2023.
[9] Haobin Li, Peng Hu, Qianjun Zhang, Xi Peng, Xiting Liu, and Mouxing Yang. Test-time adaptation for cross-man moosl retrieval with query shift. arXiv preprint arXiv:2410.1500 2024.
[10] Yunfan Li, Peng Hu, Dezhong Peng, Jiancheng Lv, Jianping Fan, and Xi Peng. Image clustering with external guidance. In International Conference on Machine Learning, pages 27890-27902 PMLR, 2024.
[11] Kaiqu Liang and Samuel Albanie. Simple baselines for interactive video retrieval with questions and answers. Proceedings of the IEFF/CVE International Conference on Computer Vision pages 11091-11101, 2023.
10

---

[12] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages
PmL., 2021.
[13] Sachyung Lee, Sangwon Yu, Junsung Park, Jihun Yi, and Sungroh Yoon. Interactive text-to- image retrieval with large language models: plug-and-plas approach. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties ( (Volume 1: Long Papers), pages 791-809, 2024.
[14] Haochen Han, Qinghua Zheng, Guang Dai, Minnan Luo, and Jingdong Wang. Learning to rematch mismatched pairs for robust cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26679 266888 2024.
[15] Avinash Madasu, Junier Oliva, and Gedas Bertasius. Learning to retrieve videos by asking questions. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2022.
[16] Donghoon Han, Eunhwan Park, Gisang Lee, Adam Lee, and Nojun Kwak. Merlin: Multimodal embedding refinement via Ilm-based iterative navigation for text-video retrieval-rerank pipeline. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 547-562, 2024.
[17] Junsheng Wang, Tiantian Gong, Zhixiong Zeng, Changchang Sun, and Yan Yan. C3emr: Cross-modality cross-instance contrastive learning for cross-media retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4300 13000-430,, 2022.
[18] Zhenqiu Shu, Yibing Bai, Kailing Yong, and Zhengtao Yu. Deep cross-modal hashing with ranking learning for noisy labels. IEEE Transactions on Big Data, 2024.
[19] Quanxing Zha, Xin Liu, Yiu-ming Cheung, Xing Xu, Nannan Wang, and Jianjia Cao. Ugnel: Uncertainty-gunt guided noisy correspondence learning for efficient cross-modal matching. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 852-861, 2024.
[20] Chao Su, Huiming Zheng, Dezhong Peng, and Xu Wang. Dica: Disambiguated contrastive alignment for cross-modal retrieval with partial labels. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 20610-20618. 2025.
[21] Ruitao Pu, Yang Qin, Xiaomin Song. Dezhong Peng. Zhenwen Ren, and Yuan Sun. She: Streaming-mest media hashing retrieval. In Forty-second International Conference Machine Learning.
[22] Yuan Sun, Jian Dai, Zhenwen Ren, Yingke Chen, Dezhong Peng, and Peng Hu. Dual self- paced cross-modal hashing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15184-11 55184-1512,, 2024.
[23] Khoi Pham, Chuong Huynh, Ser-Nam Lim, and Abhinav Shrivastava. Composing object relations and attributes for image-text matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14354-1436 2024.
[24] Yang Qin, Dezhong Peng, Xi Peng, Xu Wang, and Peng Hu. Deep evidential learning with noisy correspondence for cross cross-modal retrieval. In Proceedings of the 30th ACM International Conference Multimedia, pages 4948 4956.99566.22 2022.
[25] Ruitao Pu, Yuan Sun, Yang Qin, Zhenwen Ren, Xiaomin Song, Huiming Zheng, and Dezhong Peng. Robust self-paced hashing for cross-modal retrieval with noisy labels. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 19969-19977, 2025.
[26] Chengzhi Lin, Ancong Wu, Junwei Liang, Jun Zhang, Wenhang Ge, Wei-Shi Zheng, and Chunhua Shen. Text-adaptive multiple visual prototype matching for video-text retrieval. Advances in neural information processing systems, 15:38655-38666 2022.
11

---

[27] Weike Jin, Zhou Zhao, Pengcheng Zhang, Jieming Zhu, Xiuqiang He, and Yueting Zhuang. Hierarchical cross-moda graph consistency learning for video-text retrieval. In Proceedings of the 44th International ACM SIGIR Conference research and development in information retrieval, pages 1114-1124, 2021.
[28] Yanglin Feng, Hongyuan Zhu, Dezhong Peng. Xi Peng, and Peng Hu. Rono: robust discrimina- tive learning with noisy labels for 2d-3d cross-modal retrieval. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11610-11619 2023.
[29] Yongxiang Li, Yang Qin, Yuan Sun, Dezhong Peng, Xi Peng, and Peng Hu. Romo: Robust unsupervised multimodal learning with noisy pseudo labels. IEEE Transactions on Image Processing, 2024.
[30] Hongguang Zhu, Chunjie Zhang, Yunchao Wei, Shujuan Huang, and Yao Zhao. Esa: External space attention aggregation for image-text retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[3]] Zheng Li, Caili Guo, Zerun Feng, Jenq-Neng Hwang, and Xijun Xue. Multi-view visual semantic embedding. In IJCAI, volume 2, page 7, 2022.
[32] Yuan Sun, Zhenwen Ren, Peng Hu, Dezhong Peng, and Xu Wang. Hierarchical consensus hashing for cross-modal retrieval. IEEE Transactions on Multimedia, 26:824-836, 2023.
[33] Haiwen Diao, Ying Zhang, Lin Ma, and Huchuan Lu. Similarity reasoning and filtration for image-text matching. In Proceedings of the AAAI conference on artificial intelligence. volume 35, pages 1218-1226, 2021.
[34] Dongwon Kim, Namyup Kim, and Suha Kwak. Improving cross-modal retrieval with set of diverse embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23422-23431, 2023.
[35] Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15524-15533, 2022.
[36] Changkun Liu, Shuai Chen, Yash Sanjay Bhalgat, Siyan Hu, Ming Cheng, Zirui Wang, Vic- tor Adrian Prisacariu, and Tristan Braud. Gs-epr: Efficient camera pose refinement via 3d gaussian splatting. In The Thirteenth International Conference on Learning Representations, 2024.
[37] Yang Qin, Yuan Sun, Dezhong Peng, Jocy Tianyi Zhou, Xi Peng, and Peng Hu. Cross- modal active complementary learning with self-refining correspondence. Advances in Neural Information Processing Systems, 36, 2024.
[38] Ryoya Nara, Yu-Chich Lin, Yuji Nozawa, Youyang Ng, Goh Itoh, Osamu Torii, and Yusuke Matsui. Revisiting relevance feedback for clip-based interactive image retrieval. In European Conference on Computer Vision, pages 1-16. Springer, 2024.
[39] Honglin Liu, Peng Hu, Changqing Zhang, Yunfan Li, and Xi Peng. Interactive deep clustering via value mining. Advances in Neural Information Processing Systems, 37:42369-4999 42387, 2025.
[40] Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. Large language models are zero-shot rankers for recommender systems. In European Conference on Information Retrieval, pages 364-381. Springer, 2024.
[41] Hongyi Zhu, Jia-Hong Huang, Stevan Rudinac, and Evangelos Kanoulas. Enhancing interactive image retrieval with query rewriting using large language models and vision language models. In Proceedings of the 2024 International Conference Multimedia Retrieval, pages 978-987, 2024.
[42] Yang Qin, Chao Chen, Zhihang Fu, Dezhong Peng, Xi Peng, and Peng Hu. Human-centered interactive learning via milms for text-to-image person re-identification. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14390-14399, 2025.
12

---

[43] Yiding Lu, Mouxing Yang, Dezhong Peng, Peng Hu, Yijie Lin, and Xi Peng. Llava-reid: Selective multi-image questioner for interactive person re-identification. arXiv preprint arXiv:2504. 10174, 2025.
[44] Christopher R Palmer and Christos Faloutsos. Density biased sampling: An improved method for data mining and clustering. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 82-92, 2000.
[45] Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15789-15798, 2021.
[46] David MJ Tax and Robert PW Duin. Support vector data description. Machine learning, 54:45-66, 2004.
[47] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 0-22422-24487, 2022.
[48] Shiqi Yang. Shangling Jui, Joost Van De Weijer, et al. Attracting and dispersing: A simple approach for source-free domain adaptation. Advances in Neural Information Processing Systems, 15:5802--8155 2022.
[49] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. Advances in neural information processing systems, 34:3635- 3649,,6494604 2021.
[50] Gezheng Xu, Hui Guo, Li Yi, Charles Ling, Boyu Wang, and Grace Yi. Revisiting source-free domain adaptation: a new perspective via uncertainty control. In The Thirteenth International Conference on Learning Representations.
[51] Dave Zhenyu Chen, Aneel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202-221. Springer, 2020.
[52] Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 422-440. Springer, 2020.
[53] Zhengxin Pan, Fangyu Wu, and Bailing Zhang. Fine-grained image-text matching by cross- modal hard aligning network. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19275-19284, 2023.
[54] Zheren Fu, Zhendong Mao, Yan Song, and Yongdong Zhang. Learning semantic relationship among instances for image-text matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15168, June 2023.
[55] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV), pages 201-216, 2018
[56] Zhenyu Huang, Guocheng Niu. Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng. Learning with noisy correspondence for cross-modal matching. In M. Ranzato, A. Beygelzimer, Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances Neural Information Processing Systems, volume 34, pages 29406-200 29419. Curran Associates, Inc., 2021.
[57] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38 2019.
[58] Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04803 2018.
13

---

[59] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2399.1660, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the Conclusion Section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
The Analysis and proof are provided in the supplementary material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The implementation and training details are clearly described for reproduction in main paper and supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
The code will released publicly after in-peer review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/
mublic/guides Seriidde//odeeummisssisis.. wwwssssoooooooo oooicllcyyy)oo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //tpps.ccppublll//// ctcccc/pp. wwt.ccccpppccc.eee.ee..ee...ceee ppbliccguides. /ttgg/////eeeuubbiiiiiieneee
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experiment settings are clearly presented in the paper and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive for experiments involving LLMs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources are reported the experiment settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https //tee///neuipeee htcp/pbbicc/thhccccieeeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conform with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts are discussed with the limitations.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: All datasets and models used in this paper are publicly available.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Proper citations are provided throughout the document and the licenses will be included with the code when it is released.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The document will accompany the code upon its release.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We have fully disclosed the details of the use of the adopted LLMs in our supplementary material.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https http///ips.ce/ooneeeeeeeeeeeeeee ccc////oooeerrccce/22/////// for what should or should be described.
21


Reviews:
[{"review_id": "NeQbKp7uwO", "reviewer": "Reviewer_3WWu", "summary": "This paper addresses the challenge of retrieving 3D scenes from textual queries, emphasizing scenarios where the queries are incomplete or otherwise imperfect—a realistic concern in practical deployments. The proposed method, IDeal, introduces an interactive retrieval refinement (IRR) framework, in which a questioner, answerer (simulated via LLMs), and retriever iteratively interact to enrich scene understanding and align multi-turn text with scene features through both feature-level and semantic-level fusion. To reduce the domain gap between initial queries and dialog-augmented texts, the Interaction Adaptation Tuning (IAT) strategy is developed, leveraging contrastive domain adaptation principles. Experiments on several benchmarks show improved performance over both traditional and interactive baselines, with ablations evidencing the contributions of each proposed component.", "questions": "1: How does IDeal handle interaction rounds when the LLM answerer \"hallucinates\" or provides inconsistent information?\n\n2: Would performance differ substantially if the \"answerer\" were replaced with real end-users?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1: The paper identifies a genuine gap in existing text-3D scene retrieval systems: the assumption of information-complete queries, which rarely holds in real-world settings. By focusing on interactive, user-in-the-loop refinement, it realistically addresses this shortcoming.\n\n2: The Interactive Retrieval Refinement (IRR) framework is conceptually well-motivated. It employs a router in the questionnaire that adaptively selects between in-depth, detail-probing questions and divergent, exploratory questions. This is supported by a formalization using a corrected affinity entropy that accounts for density bias in feature space, which is a thoughtful design.\n\n3: The paper evaluates IDeal extensively on relevant datasets (ScanRefer, Nr3D, Sr3D), using standard metrics (R@1, R@5, R@10, Rsum) and compares not only to standard retrieval baselines but also to state-of-the-art interactive approaches, as detailed in Table 1. Ablation studies and parameter analyses provide further depth.\n\nWeaknesses:\n\n1: **Dependency on Quality of LLM-generated Responses.** The framework’s core relies on the LLM’s ability to answer questions accurately. When the LLM fails or hallucinates (as hinted at in the discussion of Figure 4), retrieval performance can degrade or plateau. This dependency may limit the method's practical robustness, particularly in domains where LLMs have limited training data or understanding. It would be better if the authors could provide an analysis of the generated content from the LLM to convince the audience. \n\n2: **Evaluation with More Types of LLMs**. The authors have only benchmarked their method against academic baselines and failed to include any evaluation with industry-grade vision-language models (VLMs), such as GPT-4o, Gemini 2.5 Flash.\n\n3: Can the authors clarify the process for constructing and updating the questioner’s router, especially with regard to parameter sensitivity? While Figure 3c explores $k$ and $\\beta$, it’s not clear how sensitive the system is to these parameters in practice, or how robust the metric is to different gallery/scene densities.", "was_revised": true, "final_justification": "Most of my concerns have been addressed in the replies. Therefore, I decide to maintain my current rating.", "rebuttal": "Thanks for your valuable comments and insightful suggestions. We have carefully looked into all the comments and suggestions. Attached is our point-by-point response.\n\n**Q1: 1) The framework’s core relies on the LLM’s ability to answer questions accurately. When the Large Language Model (LLM) fails or hallucinates, retrieval performance can degrade. This dependency may limit the method's practical robustness, particularly in domains where LLMs have limited training data or understanding. 2) It would be better if the authors could provide an analysis of the generated content from the LLM to convince the audience.**\n\n**R1**: **1)** We acknowledge that our framework depends on the LLM’s ability to some extent. However, unlike most of the interactive methods [A], our proposed method does not strongly rely on the LLM’s reasoning ability over questions or response texts, but merely on its ability to generate or answer questions based on the previous-round conversation or memory text. This is a straightforward task for LLMs and thus does not pose significant risks of hallucination. Moreover, to enhance its robustness, we have i) constrained the LLM’s output with task-specific, well-defined prompts and ii) limited free-form generation by using structured inputs and outputs, thereby reducing hallucination risks. **2)** Thank you again for your constructive suggestions. We have already presented and analyzed cases of generated question and answer contents in **Section 4.5 and Figure 5 of our main text**, as well as in **Section B.6 and Figure 1 of our Supplementary Material**. In future versions, we will provide additional supplementary material with a more comprehensive case analysis focusing on hallucinations.\n\n**Q2: The authors have only benchmarked their method against academic baselines and failed to include any evaluation with industry-grade LLMs, such as GPT-4o, Gemini 2.5 Flash.**\n\n**R2**: Thank you for your suggestion on the completeness of the method evaluation. Due to time constraints, we are unable to provide evaluation results for all methods under industry-grade LLMs within a short period. Nevertheless, we attempted to evaluate our IDeal using industry-grade models. Our experiments show that using larger models, such as Qwen2.5-32B and GPT-4o, do not improve performance. We speculate that this is because, as noted in **R1**, generating questions and answers based on text is a relatively straightforward task for LLMs. Thus, academic models are not inferior to large-scale industry-grade LLMs for our task evaluation.\n\n**Q3: 1) Can the authors clarify the process for constructing and updating the questioner’s router, especially with regard to parameter sensitivity? 2) While Fig. 3c explores $k$ and $\\beta$, it’s not clear how sensitive the system is to these parameters in practice, or how robust the metric is to different gallery/scene densities.**\n\n**R3**: **1)** As shown in the blue part in the middle of **Figure 2**, the Router functions as a conditional switcher. Specifically, it takes a user query and computes its Density Compensated Factor (see **Eq. 1 and Eq. 2**), which measures whether the query is sufficiently informative. Based on whether it exceeds a predefined threshold, the Router guides the user either to further clarify ambiguous parts or to explore new scenarios. As analyzed in **Section 4.4**, the involved hyperparameters $k$ and $\\beta$ are not sensitive: setting $k$ to 10–20 and $\\beta$ to 2–2.5 effectively filters out uninformative queries for next-round clarification while enabling divergent exploration for the remaining. **2)** To further address your potential concerns, we also conducted a sensitivity analysis of $k$ and $\\beta$ on Nr3D and Sr3D datasets with varying scene densities compared with ScanRefer. The same settings ($k$=10–20, $\\beta$=2–2.5) consistently yielded stable optimal performance. As we are currently unable to include the figure in the rebuttal stage, we will provide the corresponding results in our future versions.\n\n**Q4: Would performance differ substantially if the \"answerer\" were replaced with real end-users?**\n\n**R4:** No. Although this paper uses LLMs instead of real end-users to improve the evaluability and reproducibility of the experiments, we have made every effort to ensure that the LLM answerer closely resembles real end-users, following the newest interactive work [B]. More specifically, on the one hand, as described in the experiment settings (i.e., **Section 4.1**) and **Section B.2 2) of Supplementary Material**, we simulate user memory using fine-grained descriptions. Compared to other existing interactive works [A, C, D] that directly use ground truth, our approach better reflects the real-world scenario where users may possess richer, latent knowledge about the scene that has yet to be explicitly expressed. On the other hand, we carefully design prompts in the style of the query dataset (i.e., ScanRefer, Nr3D, etc.) to guide the LLM answerer in generating more realistic responses that better resemble those of real users in the data context. Please refer to **Section A.3** of our Supplementary Material for details. Naturally, complex interaction challenges in real-world settings may impact performance, further motivating our continued exploration of this topic in future research.\n\n**References:**\n\n[A] Han, Donghoon, et al. \"MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline.\" EMNLP (Industry Track). 2024.\n\n[B] Lu, Yiding, et al. \"LLaVA-ReID: Selective Multi-image Questioner for Interactive Person Re-Identification.\" Forty-second International Conference on Machine Learning.\n\n[C] Levy, Matan, et al. \"Chatting makes perfect: Chat-based image retrieval.\" Advances in Neural Information Processing Systems 36 (2023): 61437-61449.\n\n[D] Zhu, Hongyi, et al. \"Enhancing interactive image retrieval with query rewriting using large language models and vision language models.\" Proceedings of the 2024 International Conference on Multimedia Retrieval. 2024."}, {"review_id": "y9tphKGuko", "reviewer": "Reviewer_av9S", "summary": "This paper discusses an interactive approach to solving Text-3D Scene Retrieval, which serves as the foundation for 3D scene grounding or reasoning. Specifically, the authors propose an interactive Text-3D Scene Retrieval method called IDEAL, which continuously optimizes the alignment between text and the target scene through interaction. It consists of two modules: IRR, which addresses the problem of lack of holistic interaction perspectives, and IAT, which tackles the challenge of the domain gap between queries and interaction texts. This method achieves performance improvements on three common indoor scene multimodal datasets under two different experimental settings.", "questions": "1. More discussion of existing interactive methods?\n\n2. More detailed explanation of methods?\n\n3. Further discussion of experimental settings?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n\nThe paper proposes a novel interactive solution to address the inherent challenges of T3SR. I believe the motivation is clear, the problem-solving approach is well-defined, and the work demonstrates strong originality. The authors provide solutions to issues such as the domain shift in their method, supported by sound theoretical foundations and presented in a well-organized and readable manner. The evaluation under two different memory in the experimental section is comprehensive.\n\nCons:\n\n1. The explanations for Fig. 1 and 2 in the paper are insufficient, which leads to some misunderstandings, especially in the upper right part of Fig. 2.\n\n2. Several parts of method section is difficult to understand. For example, between lines 166 and 188, there are too many inline equations and the explanations are not clear enough. What is the difference between two feature fusion approaches?\n\n3. The authors emphasize that their approach addresses the lack of holistic perspective issue present in current interactive methods, but there is a lack of discussion and comparison with related methods.\n\n4. Need to provide a more in-depth discussion of the two memory settings. Does the fine-grained memory setting risk label leakage?", "was_revised": true, "final_justification": "The authors have addressed the technical details I was concerned about, and I am willing to maintain my original score.", "rebuttal": "Thank you for your valuable comments and insightful feedback. Attached is our point-by-point response.\n\n**Q1: The explanations for Figure 1 and Figure 2 in the paper are insufficient.**\n\n**R1:** Thank you for your suggestion. **Figure 1** and **Figure 2** correspond to the motivation and methodology of our paper, respectively. In addition to the figure captions, we also provided comprehensive explanations and discussions in our Introduction and Method Section (i.e., **Section 1** and **Section 3**). To avoid ambiguity, we will include more comprehensive descriptions in the figure and its captions in the next version.\n\n**Q2: 1) Several parts of the method section are difficult to understand. For example, between lines 166 and 188, there are too many inline equations, and the explanations are not clear enough. 2) What is the difference between the two feature fusion approaches?**\n\n**R2:** **1)** Thank you for your feedback. We will carefully re-examine the readability and completeness of the Method Section (i.e., **Section 3**). In the next version, we will integrate inline equations, improve the coherence of the displayed equations, and provide clearer and more detailed explanations to enhance overall clarity. **2)** There are two types of feature fusion strategies. More specifically, the first fusion approach targets the answer features to the detail probe $\\mathcal{Q}_1$. It performs a weighted linear fusion of current and previous answer features, enhancing the discriminability of fine-grained scene details. In contrast, the second approach focuses on the answers to the divergent exploration $\\mathcal{Q}_2$. The answer text features are modeled as a hypersphere, and the fusion is achieved by averaging its center with the internal features, yielding a comprehensive representation of the scene's divergent characteristics.\n\n**Q3: The authors claim their approach addresses the lack of a holistic perspective in current interactive methods, but offer limited discussion or comparison with related work. More discussion of existing interactive methods.**\n\n**R3:** We have provided both empirical and experimental comparisons of the questioning perspective between existing interactive retrieval methods and our proposed IDeal in **Section 1**, **Section 2.2**, and **Section 4.2**. Specifically, existing interactive methods are primarily designed for image data, which carries significantly less information than 3D scene data. These methods typically refine the query based on the previous response, guiding the user to progressively provide discriminative answers. However, 3D scenes contain numerous and spatially scattered details, making such approaches prone to over-focusing on local regions while overlooking information from other parts of the scenes, as illustrated in **the top-left part of Figure 1** and **the lower part of Figure 5**. To address this issue, we introduce the Adaptive Questioning module, which explicitly encourages attention to different regions of the scene from a holistic perspective. This design achieves improved performance in both quantitative (i.e., **Table 1**) and qualitative (i.e., **Figure 5 and Figure 1 in the Supplementary Material**) evaluations. In future versions, we will consolidate these discussions into a more comprehensive analysis.\n\n**Q4: 1) Need to provide a more in-depth discussion of the two memory settings. 2) Does the fine-grained memory setting risk label leakage?**\n\n**R4:** **1)** We provide a comprehensive explanation and discussion of the two memory settings and their practical significance in **Section B.2 of the Supplementary Material**. **2)** No. As discussed in **Section B.2 of the Supplementary Material**, although this setting involves certain information leakage, it is significantly more restrained compared to prior interactive works [A, B], which expose ground-truth labels. Our fine-grained descriptions do not directly leak such labels, making the setting more realistic for evaluating models under vague or implicit user intent.\n\n**References:**\n\n[A] Levy, Matan, et al. \"Chatting makes perfect: Chat-based image retrieval.\" Advances in Neural Information Processing Systems 36 (2023): 61437-61449.\n\n[B] Han, Donghoon, et al. \"MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline.\" EMNLP (Industry Track). 2024."}, {"review_id": "yDvil7o4M2", "reviewer": "Reviewer_iGFD", "summary": "This manuscript attempts to address the challenges in the T3SR task by proposing IDeal, a new interactive retrieval framework. This framework targets and tackles two major challenges faced by existing interactive approaches in T3SR. Specifically, it consists of two key components: IRR, which refines retrieval results through multi-round Q&A between a questioner and an answerer, and IAT, which adapts the model to the interactive text domain via contrastive learning. Extensive experiments on three public datasets demonstrate that IDeal outperforms existing baselines in both fine-grained and coarse-grained memory settings.", "questions": "My main concern is on whether the seemingly complex methods introduced by the author bring any clear benefits, and how these complexities can be clearly explained. If the author can address the issues I mentioned in the weaknesses section reasonably, I would consider raising my score.", "limitations": "Yes, the authors have discussed.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "This manuscript is well-motivated, addressing the issue of query bias in real-world applications of the T3SR task. The manuscript is well-structured and clearly written, making it easy to follow. In the methodology section, the paper introduces IRR, which implements an interactive framework tailored for the T3SR task, and employs IAT to facilitate effective interactions that enhance retrieval performance. The experimental setup is thorough and convincingly demonstrates the effectiveness and superiority of the proposed method.\n\nThe manuscript still has some weaknesses. First, while the authors mention issues with the retrieval queries, it is unclear whether these actually impact retrieval performance. Additionally, the description of the retriever and the IAT process lacks clarity. The added complex operations may increase model intricacy without a clear demonstration of performance gains. For example, it is not evident why IAT is preferred over directly retraining the model. Furthermore, two kinds memory experiments are confusing, with different baselines and memory data. The ablation study seems to be incomplete, missing the experiments with coarse-grained memory.", "was_revised": true, "final_justification": "Thanks for the response, which has addressed my concerns. I support accepting this paper!", "rebuttal": "Thanks for your valuable comments and insightful suggestions. Attached is our point-by-point response.\n\n**Q1: It is unclear whether query issues actually impact retrieval performance.**\n\n**R1:** As demonstrated in existing work [A] discussed in **lines 30-37 of Section 1**, query issues have a direct impact on retrieval performance. In particular, **Table 2** shows that under the coarse-grained memory setting, the existing cross-modal retrieval method combined with our interactive framework IDeal, which progressively mitigates query issues, significantly outperforms the baseline that does not address these issues across all three datasets. In addition to comparing the inherent query issues in ScanRefer and Nr3D/Sr3D, we conducted supplementary experiments in **Section B.3 of our Supplementary Material** by introducing real-world query issues, including sentence-level, word-level, and character-level corruptions, as shown in **Table 1** of the Supplementary Material. These experiments reveal that the presence of query issues results in different degrees of performance degradation.\n\n**Q2: The description of the retriever and the IAT process lacks clarity.**\n\n**R2:** Thank you for your feedback. We will carefully re-examine the readability and completeness of the Retriever and IAT Section (i.e., **Section 3.2.2** and **Section 3.3**). In the next version, we will integrate inline equations, providing more coherent and clearer transition descriptions and process explanations to enhance overall clarity.\n\n**Q3: The added complex operations may increase model intricacy without a clear demonstration of performance gains. For example, it is not evident why IAT is preferred over directly retraining the model.**\n\n**R3:** As stated in **Section 3.3** of our paper, IAT facilitates retrieval models adapting to the interaction text domain without requiring access to their implementation details, making it more convenient for improving interactive performance. To address the constructive concerns raised by the reviewer, we conducted comparative experiments between IAT and direct retraining on the ScanRefer dataset. The results are shown in the table below. In addition, based on the results and the results summarized in **Table 3** of our ablation study, removing or substituting components of IAT consistently leads to a noticeable performance degradation, highlighting the necessity of our proposed IAT.\n\n| ScanRefer     | R@1  | R@5  | R@10 | Rsum  |\n|---------------|------|------|------|-------|\n| w/ Retraining | 34.5 | 68.8 | 85.1 | 188.4 |\n| w/ IAT        | 37.8 | 71.8 | 86.4 | 196.0 |\n\n**Q4: Two kinds of memory experiments are confusing, with different memory data and baselines.**\n\n**R4:** This paper designs two different memory settings corresponding to two types of memory data and baselines. Specifically, the memory data in the coarse-grained memory setting consists of query expansions without additional information. The fine-grained memory data consists of external detailed descriptions (from the SceneDepict-3D2T text set) that simulate latent memories in the human mind, which are not explicitly expressed in the original query. A detailed discussion of these data can be found in **Section B.2 of our Supplementary Material**. In addition, under the coarse-grained memory setting, since no additional information is available, comparisons are made using general cross-modal matching or 3D-text matching baselines. Under the fine-grained memory setting, the user’s memory is simulated by external scene descriptions, and the method iteratively interacts with and learns from these fine-grained memories. Therefore, comparisons are made against interactive cross-modal retrieval baselines to ensure fairness.\n\n**Q5: The ablation study seems to be incomplete, missing the experiments with coarse-grained memory.**\n\n**R5:** Due to page limitations, the ablation study with coarse-grained memory setting has been included in our **Supplementary Material (Section B.5)**. The experiments demonstrate that under the coarse-grained memory setting, all components of our proposed IDeal contribute to the performance.\n\n**References:**\n\n[A] Li, Haobin, et al. \"Test-time Adaptation for Cross-modal Retrieval with Query Shift.\" The Thirteenth International Conference on Learning Representations."}, {"review_id": "OWWsr0aqep", "reviewer": "Reviewer_zoJY", "summary": "This paper targets Text-to-3D Scene Retrieval (T3SR), and proposes a novel interactive paradigm to address it. Specifically, the authors introduce the IDeal method, which includes the Interactive Retrieval Refinement (IRR) framework and the Interaction Adaptation Tuning (IAT) strategy. IRR facilitates interaction between a questioner and answerer to enrich scene understanding, while IAT bridges the domain gap between static queries and interactive texts for robust interactions. Extensive experiments demonstrate the effectiveness of the proposed method under two realistic settings.", "questions": "Please refer to **Weaknesses**. The authors need to consider elaborating on Section 3.2.2 of the methodology to clarify the necessity of the current iterative feature fusion approach. In addition, the rationale behind the chosen experimental settings could be better justified, particularly in terms of whether they effectively address the query issues highlighted earlier in the paper.", "limitations": "Yes, the authors have thoroughly discussed the limitations and potential impact.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths\n1. The paper is well-motivated, and the proposed method appears effective based on both the methodological and experimental sections. The integration with an interactive LLM-based scheme enhances both the novelty and performance of the approach.\n2. The methodology is sound and reproducible. The implementation details and parameter specifications are explained in a comprehensive manner.\n3. The figures and tables in the paper are aesthetically pleasing and informative, and the visualization in Figure 5 is interesting.\n4. The theoretical foundations used in the paper are reasonable, and the proofs appear sufficiently complete.\n\n### Weaknesses\n1. The authors propose different feature fusion strategies for two types of interaction responses, but the rationale behind this design is not clearly explained. It is unclear why the better-quality response features are not simply given greater weight in the fusion process, and the current approach appears somewhat complex.\n2. While the paper discusses how deficiencies in the current query texts may impact the method's performance, the experiments still rely on the original queries from ScanRefer and Nr3d datasets. This point would benefit from further clarification.\n3. There are instances of imprecise wording that may lead to ambiguity in the presentation, such as in lines 46 and 131.\n4. The performance of the current T3SR method appears to be relatively low.", "was_revised": true, "final_justification": "I appreciate the authors' detailed responses, which have satisfactorily addressed all my questions. I have also read other reviews and the authors' responses to them. I agree with other reviewers that this is a good paper and will inspire further research. Therefore, I recommend accepting the paper. Congrats!", "rebuttal": "Thanks for your valuable comments and insightful suggestions. We have carefully looked into all the comments and suggestions. Attached is our point-by-point response.\n\n**Q1: 1) The rationale behind the feature fusion strategy design is not clearly explained. 2) The necessity of the iterative fusion approach remains insufficiently justified. 3) It is unclear why the better-quality response features are not simply given greater weight in the fusion process.**\n\n**R1:**  **1)** The two proposed feature fusion strategies are designed for different types of answer features. More specifically, the first fusion approach targets the answer features to the detail probe $\\mathcal{Q}_1$. It performs a weighted linear fusion of current and previous answer features, enhancing the discriminability of fine-grained scene details. In contrast, the second approach focuses on the answers to the divergent exploration $\\mathcal{Q}_2$. The answer text features are modeled as a hypersphere, and the fusion is achieved by averaging its center with the internal features, yielding a comprehensive representation of the scene's divergent characteristics.  **2)** To demonstrate the necessity of the proposed fusion strategy, we compare it against four baseline strategies on the ScanRefer dataset with fine-grained memory setting: using the First Answer (FA), the Best Answer (BA), the Last Answer (LA), and an Average Fusion of all answers (AF). **The retrieval performance in terms of RSum (i.e., R@1 + R@5 + R@10) is shown in the table below.** These results verify that, compared to non-fusion or naive averaging methods, our proposed fusion strategy enables adaptive integration of both fine-grained details and divergent information, making it more suitable for complex and large-scale 3D scene data. **3)** During the interactive process, the extracted features are diverse. However, high feature quality based on a specific metric (i.e., Density Compensated Affinity Entropy in our proposed Adaptive Questioning module) only indicates better discriminability, but it does not guarantee it is retrieval-relevant. As demonstrated by the results in the parameter analysis (i.e., **Figure 3**) and ablation studies (i.e., **Table 3**), our feature fusion strategy, which integrates all responses, captures scene-level details more comprehensively and effectively prevents information loss.\n\n| Method | FA   | BA   | LA   | AF   | Ours |\n|--------|------|------|------|------|------|\n| Rsum   | 26.5 | 29.1 | 30.7 | 34.6 | 35.4 |\n\n**Q2: While the paper discusses how deficiencies in the current query issues may impact the method's performance, the experiments still rely on the original queries from ScanRefer and Nr3d datasets. The rationale behind the chosen experimental settings could be better justified.**\n\n**R2:** In our experiments, we use the original queries from the ScanRefer and Nr3D datasets as the initial queries, which themselves contain query issues such as incomplete one-shot descriptions of user intent, ambiguous descriptions, etc. These are used to analyze whether existing methods and our IDeal can address these issues during the interaction or retrieval process, in order to achieve more robust and applicable cross-modal alignment and retrieval. A more detailed rationale for the experimental setup is provided in **Section 4.1** of our main text and **Section B.2 of our Supplementary Material**.\n\n**Q3: There are instances of imprecise wording that may lead to ambiguity in the presentation, such as in lines 46 and 131.**\n\n**R3:** Thank you for pointing that out. In the next version, we will review and reorganize these statements to ensure greater clarity.\n\n**Q4: The performance of the current T3SR method appears to be relatively low.**\n\n**R4:** The T3SR task is an emerging realistic task, and it still relies on object-level queries from 3D visual grounding to achieve scene-level alignments, which naturally limits its performance. This work specifically targets the query issues and explores leveraging external LLMs or user knowledge, leading to notable improvements in both the average performance and practical applicability of T3SR. Moving forward, we will continue to focus on advancing both T3SR models and datasets to enhance their real-world performance further."}]

Author rebuttal:

Meta-review: This paper proposes an interactive framework for text-to-3D scene retrieval that effectively addresses the real-world challenges of incomplete or ambiguous queries through iterative refinement and domain adaptation. The reviewers unanimously recognize the novelty, solid theoretical grounding, and comprehensive experimental validation of the work, and the rebuttal successfully clarified all concerns regarding methodological details and experimental settings. With strong support from the reviewers, this paper stands out as a timely and impactful contribution to interactive multimodal retrieval. I recommend Accept.
Decision: Accept (oral)