Title: Interactive Cross-modal Learning for Text-3D Scene Retrieval
Venue: NeurIPS 2025
Accepted: yes
Keywords: Cross-modal retrieval

interactive retrieval

large language model
Abstract: Text-3D Scene Retrieval (T3SR) aims to retrieve relevant scenes using linguistic queries. Although traditional T3SR methods have made significant progress in capturing fine-grained associations, they implicitly assume that query descriptions are information-complete. In practical deployments, however, limited by the capabilities of users and models, it is difficult or even impossible to directly obtain a perfect textual query suiting the entire scene and model, thereby leading to performance degradation. To address this issue, we propose a novel Interactive Text-3D Scene Retrieval Method (IDeal), which promotes the enhancement of the alignment between texts and 3D scenes through continuous interaction. To achieve this, we present an Interactive Retrieval Refinement framework (IRR), which employs a questioner to pose contextually relevant questions to an answerer in successive rounds that either promote detailed probing or encourage exploratory divergence within scenes. Upon the iterative responses received from the answerer, IRR adopts a retriever to perform both feature-level and semantic-level information fusion, facilitating scene-level interaction and understanding for more precise re-rankings. To bridge the domain gap between queries and interactive texts, we propose an Interaction Adaptation Tuning strategy (IAT). IAT mitigates the discriminability and diversity risks among augmented text features that approximate the interaction text domain, achieving contrastive domain adaptation for our retriever. Extensive experimental results on three datasets demonstrate the superiority of IDeal. Code is available at https://github.com/Yangl1nFeng/IDeal.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper targets Text-to-3D Scene Retrieval (T3SR), and proposes a novel interactive paradigm to address it. Specifically, the authors introduce the IDeal method, which includes the Interactive Retrieval Refinement (IRR) framework and the Interaction Adaptation Tuning (IAT) strategy. IRR facilitates interaction between a questioner and answerer to enrich scene understanding, while IAT bridges the domain gap between static queries and interactive texts for robust interactions. Extensive experiments demonstrate the effectiveness of the proposed method under two realistic settings.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The paper is well-motivated, and the proposed method appears effective based on both the methodological and experimental sections. The integration with an interactive LLM-based scheme enhances both the novelty and performance of the approach.\n2. The methodology is sound and reproducible. The implementation details and parameter specifications are explained in a comprehensive manner.\n3. The figures and tables in the paper are aesthetically pleasing and informative, and the visualization in Figure 5 is interesting.\n4. The theoretical foundations used in the paper are reasonable, and the proofs appear sufficiently complete.\n\n### Weaknesses\n1. The authors propose different feature fusion strategies for two types of interaction responses, but the rationale behind this design is not clearly explained. It is unclear why the better-quality response features are not simply given greater weight in the fusion process, and the current approach appears somewhat complex.\n2. While the paper discusses how deficiencies in the current query texts may impact the method's performance, the experiments still rely on the original queries from ScanRefer and Nr3d datasets. This point would benefit from further clarification.\n3. There are instances of imprecise wording that may lead to ambiguity in the presentation, such as in lines 46 and 131.\n4. The performance of the current T3SR method appears to be relatively low.\n\nQUESTIONS:\nPlease refer to **Weaknesses**. The authors need to consider elaborating on Section 3.2.2 of the methodology to clarify the necessity of the current iterative feature fusion approach. In addition, the rationale behind the chosen experimental settings could be better justified, particularly in terms of whether they effectively address the query issues highlighted earlier in the paper.\n\nLIMITATIONS:\nYes, the authors have thoroughly discussed the limitations and potential impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript attempts to address the challenges in the T3SR task by proposing IDeal, a new interactive retrieval framework. This framework targets and tackles two major challenges faced by existing interactive approaches in T3SR. Specifically, it consists of two key components: IRR, which refines retrieval results through multi-round Q&A between a questioner and an answerer, and IAT, which adapts the model to the interactive text domain via contrastive learning. Extensive experiments on three public datasets demonstrate that IDeal outperforms existing baselines in both fine-grained and coarse-grained memory settings.\n\nSTRENGTHS AND WEAKNESSES:\nThis manuscript is well-motivated, addressing the issue of query bias in real-world applications of the T3SR task. The manuscript is well-structured and clearly written, making it easy to follow. In the methodology section, the paper introduces IRR, which implements an interactive framework tailored for the T3SR task, and employs IAT to facilitate effective interactions that enhance retrieval performance. The experimental setup is thorough and convincingly demonstrates the effectiveness and superiority of the proposed method.\n\nThe manuscript still has some weaknesses. First, while the authors mention issues with the retrieval queries, it is unclear whether these actually impact retrieval performance. Additionally, the description of the retriever and the IAT process lacks clarity. The added complex operations may increase model intricacy without a clear demonstration of performance gains. For example, it is not evident why IAT is preferred over directly retraining the model. Furthermore, two kinds memory experiments are confusing, with different baselines and memory data. The ablation study seems to be incomplete, missing the experiments with coarse-grained memory.\n\nQUESTIONS:\nMy main concern is on whether the seemingly complex methods introduced by the author bring any clear benefits, and how these complexities can be clearly explained. If the author can address the issues I mentioned in the weaknesses section reasonably, I would consider raising my score.\n\nLIMITATIONS:\nYes, the authors have discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses an interactive approach to solving Text-3D Scene Retrieval, which serves as the foundation for 3D scene grounding or reasoning. Specifically, the authors propose an interactive Text-3D Scene Retrieval method called IDEAL, which continuously optimizes the alignment between text and the target scene through interaction. It consists of two modules: IRR, which addresses the problem of lack of holistic interaction perspectives, and IAT, which tackles the challenge of the domain gap between queries and interaction texts. This method achieves performance improvements on three common indoor scene multimodal datasets under two different experimental settings.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\nThe paper proposes a novel interactive solution to address the inherent challenges of T3SR. I believe the motivation is clear, the problem-solving approach is well-defined, and the work demonstrates strong originality. The authors provide solutions to issues such as the domain shift in their method, supported by sound theoretical foundations and presented in a well-organized and readable manner. The evaluation under two different memory in the experimental section is comprehensive.\n\nCons:\n\n1. The explanations for Fig. 1 and 2 in the paper are insufficient, which leads to some misunderstandings, especially in the upper right part of Fig. 2.\n\n2. Several parts of method section is difficult to understand. For example, between lines 166 and 188, there are too many inline equations and the explanations are not clear enough. What is the difference between two feature fusion approaches?\n\n3. The authors emphasize that their approach addresses the lack of holistic perspective issue present in current interactive methods, but there is a lack of discussion and comparison with related methods.\n\n4. Need to provide a more in-depth discussion of the two memory settings. Does the fine-grained memory setting risk label leakage?\n\nQUESTIONS:\n1. More discussion of existing interactive methods?\n\n2. More detailed explanation of methods?\n\n3. Further discussion of experimental settings?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of retrieving 3D scenes from textual queries, emphasizing scenarios where the queries are incomplete or otherwise imperfect—a realistic concern in practical deployments. The proposed method, IDeal, introduces an interactive retrieval refinement (IRR) framework, in which a questioner, answerer (simulated via LLMs), and retriever iteratively interact to enrich scene understanding and align multi-turn text with scene features through both feature-level and semantic-level fusion. To reduce the domain gap between initial queries and dialog-augmented texts, the Interaction Adaptation Tuning (IAT) strategy is developed, leveraging contrastive domain adaptation principles. Experiments on several benchmarks show improved performance over both traditional and interactive baselines, with ablations evidencing the contributions of each proposed component.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1: The paper identifies a genuine gap in existing text-3D scene retrieval systems: the assumption of information-complete queries, which rarely holds in real-world settings. By focusing on interactive, user-in-the-loop refinement, it realistically addresses this shortcoming.\n\n2: The Interactive Retrieval Refinement (IRR) framework is conceptually well-motivated. It employs a router in the questionnaire that adaptively selects between in-depth, detail-probing questions and divergent, exploratory questions. This is supported by a formalization using a corrected affinity entropy that accounts for density bias in feature space, which is a thoughtful design.\n\n3: The paper evaluates IDeal extensively on relevant datasets (ScanRefer, Nr3D, Sr3D), using standard metrics (R@1, R@5, R@10, Rsum) and compares not only to standard retrieval baselines but also to state-of-the-art interactive approaches, as detailed in Table 1. Ablation studies and parameter analyses provide further depth.\n\nWeaknesses:\n\n1: **Dependency on Quality of LLM-generated Responses.** The framework’s core relies on the LLM’s ability to answer questions accurately. When the LLM fails or hallucinates (as hinted at in the discussion of Figure 4), retrieval performance can degrade or plateau. This dependency may limit the method's practical robustness, particularly in domains where LLMs have limited training data or understanding. It would be better if the authors could provide an analysis of the generated content from the LLM to convince the audience. \n\n2: **Evaluation with More Types of LLMs**. The authors have only benchmarked their method against academic baselines and failed to include any evaluation with industry-grade vision-language models (VLMs), such as GPT-4o, Gemini 2.5 Flash.\n\n3: Can the authors clarify the process for constructing and updating the questioner’s router, especially with regard to parameter sensitivity? While Figure 3c explores $k$ and $\\beta$, it’s not clear how sensitive the system is to these parameters in practice, or how robust the metric is to different gallery/scene densities.\n\nQUESTIONS:\n1: How does IDeal handle interaction rounds when the LLM answerer \"hallucinates\" or provides inconsistent information?\n\n2: Would performance differ substantially if the \"answerer\" were replaced with real end-users?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}