Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intention

Authors: Weitai Kang, Mengxue Qu, Jyoti Kini, Yunchao Wei, Mubarak Shah, Yan Yan
Primary area: datasets and benchmarks
Keywords: 3D Visual Grounding, 3D Multimodal Learning

Abstract:
In real-life scenarios, humans seek out objects in the 3D world to fulfill their daily needs or intentions. This inspires us to  introduce 3D intention grounding, a new task in 3D object detection  employing RGB-D, based on human intention, such as "I want something to support my back." Closely related, 3D visual grounding focuses on understanding human reference. To achieve detection based on human intention, it relies on humans to observe the scene, reason out the target that aligns with their intention ("pillow" in this case), and finally provide a reference to the AI system, such as "A pillow on the couch". Instead, 3D intention grounding challenges AI agents to automatically observe, reason and detect the desired target solely based on human intention. To tackle this challenge, we introduce the new Intent3D dataset, consisting of 44,990 intention texts associated with 209 fine-grained classes from 1,042 scenes of the ScanNet dataset. We also establish several baselines based on different language-based 3D object detection models on our benchmark. Finally, we propose IntentNet, our unique approach, designed to tackle this intention-based detection problem. It focuses on three key aspects: intention understanding, reasoning to identify object candidates, and cascaded adaptive learning that leverages the intrinsic priority logic of different losses for multiple objective optimization.

Decision: accept (poster)
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729455641323

thread_type:
official_review

<thread_object>
creator: reviewer_qhnc

<content>
summary:
This paper presents a novel framework for 3D object detection that integrates human intention into the detection process. The authors introduce the Intent3D dataset, aiming to enhance the model's understanding of human needs in real-world scenarios. The proposed method, named IntentNet, employs a multi-instance detection approach, where the model is tasked with identifying multiple instances of objects based on free-form textual descriptions of human intentions. The authors evaluate their approach against several baselines, including expert models designed for 3D visual grounding, foundation models for generic 3D understanding tasks, and Large Language Model (LLM)-based models. The evaluation demonstrates the effectiveness of IntentNet in achieving state-of-the-art performance on the Intent3D benchmark.

strengths:
1. The introduction of the 3D Intention Grounding (3D-IG) task could contribute to the 3D visual grounding community.
2. The proposed Intent3D dataset is extensive, comprising 44,990 intention texts linked to 209 fine-grained object classes from 1042 3D scenes. This dataset provides a valuable resource for training and evaluating models in the context of human intention.
3. The proposed modules in IntentNet are generally technically sound.

weaknesses:
1. Object detection based on human intention is indeed a new task. However,  why should we need to have a dataset and a model dedicated to this task? I think the detection based on human intention can be achieved in a modulized manner. For example, first, let the 3D detection module detect all types of objects in the scene. Then ask  LLM to decide the subsets of the detected objects that can fulfill the human intention. I think this could be more flexible compared with training a dedicated detection model based on human intention prompts. 
2. In L210, it is mentioned that around six intention texts are generated per object. How do you determine this number (six)? Can it guarantee that all possible intentions can be covered and trained well?
3. The contents around Eq (3) are hard to understand for me. What is t in Eq(3)? Although the authors claimed that the code would be released to facilitate the understanding, it would be better to add a figure to illustrate the connections in the network.  
4. Figure 4 is too abstract. Even though many connections and modules are included, it is not very informative. I suggest more detailed diagrams can be provided for the key modules further.   
5. Figure 5 shows that the verb alignment is very helpful for the prediction quality. Do you think this is caused by the limited training data? If more data is available for training, would this module still be essential?

questions:
Please refer to the weakness part/
</content>
</thread_object>

</thread>
<thread>
creation date:
1730180543561

thread_type:
official_review

<thread_object>
creator: reviewer_krab

<content>
summary:
The paper introduces a 3D Intention Grounding (3D-IG) task and constructs a novel dataset called Intent3D (sourced from ScanNet data and generated using GPT). Additionally, it proposes a baseline model named IntentNet. I am not an expert in 3D-related fields, so my confidence in this review is not very high.

strengths:
1. The overall quality of the paper is high, with clear writing and easy-to-understand presentation.
2. The contribution of the dataset is significant, as it is the first to construct a 3D detection task focused on intention understanding.

weaknesses:
1. More comparisons with recent works should be provided in Tables 1 and 2. Additionally, there is a minor mistake: the detector names “GroupFree” and “Group-Free” in the first two rows of Tables 1 and 2 do not match.
2. The article gives a subtractive ablation experiment. I would like to see an additive ablation experiment, such as how the effect of verb alone works.
3. The article does not give the performance of the proposed IntentNet in traditional 3D grounding.

questions:
1. Will the proposed dataset and code be open-sourced?
2. IntentNet seems to be a two-stage approach, where a pre-trained 3D detector first extracts proposals, which are then matched with text. Are there any one-stage methods that directly fuse 3D data and text to generate boxes? If so, the paper does not seem to provide comparisons with such methods.
3. Regarding the training process, on what hardware was the method trained, and how long did the training take?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730371490788

thread_type:
official_review

<thread_object>
creator: reviewer_tznr

<content>
summary:
This paper introduces the task of 3D Intention Grounding (3D-IG), which aims to automate the reasoning and detection of target objects in real-world 3D scenes using human intention cues. To this end, the authors constructed the Intent3D dataset, comprising 44,990 intention texts across 209 fine-grained object categories, and developed several baseline models to evaluate various 3D object detection techniques. Finally, the authors proposed a novel method, IntentNet, which optimizes intention understanding and detection tasks through techniques such as verb-object alignment and candidate box matching, achieving state-of-the-art performance on the Intent3D benchmark.

strengths:
1: A new task in 3D object detection employing RGB-D, based on human intention, facilitates smoother and more natural communication between humans and intelligent agents.

2:The author propose a high-quality vision-language dataset and focuses on the human’s intention for 3D object detection, which will facilitate the progress of 3D scene understanding.

weaknesses:
1: There has been a few methods to combine 3D scene understanding with LLM beyond Chat3D v2, such as LL3DA, Grounded 3D-LLM, ReGround3D and so on.  The paper does not highlight the advantages compared to them.

2: The object selection method is too crude, as it removes some commonly used objects by humans when filtering Non-trivial Objects. Figures 3 (d) and (e) indicate that the dataset lacks sufficient diversity in the types of objects included.

3: The limited variety of object category included in the dataset, fails to demonstrate the grounding effect for the missing objects in the dataset.

questions:
1：There has been a few methods to combine 3D scene understanding with LLM beyond Chat3D v2, such as LL3DA, Grounded 3D-LLM, ReGround3D and so on.  What are the advantages of this paper compared to theirs? More experimental evidence is needed to demonstrate the advantages of the this paper over other methods.

2: Current large language models can also directly infer human intentions; what are the advantages of this paper compared to them?

3：When filtering Non-trivial Objects, objects with more than six instances in fine-grained categories are directly removed, which may lead to the exclusion of commonly used objects. Could we consider adding more fine-grained descriptions for these objects instead of outright deletion?

4:  Figures 3 (d) and (e) indicate that the dataset lacks sufficient diversity in the types of objects included. Experimental validation is necessary to determine whether the variety of object types included in the dataset is sufficient for the model to learn effectively.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698506782

thread_type:
official_review

<thread_object>
creator: reviewer_ggzj

<content>
summary:
This paper introduces a new task named 3D-intention grounding, which is 3D object-detection from direct human-intentions. The paper collects Intent3D dataset which includes 1042 scenes from ScanNet and corresponding paired human-intentions questions and 3D object detections answers. IntentNet is proposed to tackle 3D-intention grounding task by candidate box-matching, verb-object alignment and cascaded adaptively learning.

strengths:
1. The paper presents a clear motivation for 3D-intention grounding, and it includes clear illustrations and presentations of dataset collection procedure.

2. Soundness of each component design of the IntentNet, and thoroughly ablations on each component of the proposed pipeline design.

3. Extensive experiments and discussions demonstrate the effectiveness of the proposed framework compared to different types of baselines.

weaknesses:
Major concern:

I am concerned about possible baseline unfair comparison in the experiment section. Most baselines are designed to tackle nouns-types of questions instead of human-intention types of questions. What if we pass the question to a finetuned LLM and let it infers what types of nouns/objects the question is targeting at from possible objects in a scene detected by existed 3D object detectors? The possible performance of these baselines might be much higher after it is given the object it is expected to detect in a scene. 

Minor concern:
It would be interesting if the author can provide some cases where IntentNet fails but other models succeed. Particularly if other models are fed with object/noun directly.

questions:
Please see the weakness section. 

I am willing to increase my score if the author resolves my major concern with some additional baseline experimentations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732482230728

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Our Contributions and Rebuttal Responses

comment:
Dear PCs, ACs, and Reviewers,

We have submitted detailed responses during the rebuttal stage. 
**We appreciate the recognition from Reviewer qHNc & kRaB, *currently the only two responsive reviewers*, for acknowledging that our rebuttal addressed their concerns and for their decision on maintain/raise to the acceptance rate.**
Unfortunately, we have not yet received the other two reviewers' feedback. It is worth noting that **[** Reviewer gGZj **]** **indicated a willingness to raise their score based on the rebuttal**. We also invested significant resources, including renting numerous servers at considerable expense, to conduct the experiments requested by the reviewers. It is disappointing that we have not received the rest reviewers' responses. Your attention to this matter is greatly appreciated.

To facilitate further discussion among PCs, ACs, and reviewers, and to assist in finalizing the decision regarding the acceptance of our paper, we would like to briefly recapitulate our contributions and highlight key responses to the questions raised by **[** Reviewer gGZj (R1), Reviewer tzNR (R2), Reviewer kRaB (R3), and Reviewer qHNc (R4) **]**:

### **Summary:**

**(Contribution)**

- In this paper, we propose a new task, 3D Intention Grounding (3D-IG), for detecting objects in 3D real-world environments based on human intention instructions. We greatly appreciate **[** R1 (Strengths 1), R2 (Strengths 1), and R4 (Strengths 1) **]** recognize our **clear motivation** and its **contribution to our community**.
- To systematically study this new problem, we develop the Intent3D dataset to support both the training and benchmarking of 3D-IG. We are thankful that **[** R1 (Strengths 1) **]** acknowledge our dataset collection procedure, **[** R2 (Strengths 2) **]** describe our **dataset as high-quality**, **[** R3 (Strengths 2) **]** emphasize the **dataset’s significant contribution**, and **[** R4 (Strengths 2) **]** note that the **dataset is extensive** and **provides a valuable resource** for this task.
- After building the benchmark, we conduct a comprehensive evaluation of the existing methods. We appreciate **[** R1 (Strengths 2) **]** for recognizing our discussions of **extensive baselines**.

**(Soundness)**

- We propose IntentNet, a strong baseline that achieves SOTA on Intent3D. IntentNet introduces three novel modules: Candidate Box Matching, Verb-Object Alignment, and Cascaded Adaptive Learning. We are grateful for the recognition of IntentNet’s **soundness and effectiveness** from **[** R1 (Strengths 2), R4 (Strengths 3) **]**.
- Finally, to validate the effectiveness of each module in IntentNet, we conducted extensive ablation studies. We thank **[** R1 (Strengths 2 & 3) **]** for acknowledging our **thorough and extensive ablation study** as well as the **effectiveness** of each module.

**(Presentation)**

- We appreciate **[** R1 (Strengths 1) **]** for recognizing our **clear presentation** and **[** R3 (Strengths 1) **]** for acknowledging our **paper's overall high quality** with **clear writing** and **ease of understanding**.

### **Rebuttal:**

**(New experiments or results)**

- Added experiments comparing with LLM + detector [Response 1.1, 3.1, 4.1].
- Added experiments comparing with baseline + detector [Response 1.2, 3.1].
- Added experiments comparing with LLM-based methods [Response 2.1, 3.1].
- Conducted additional ablation studies [Response 3.2].
- Included analysis of failed cases [Response 1.3].
- Provided detailed demonstrations [Response 4.3].

**(Clarify misunderstandings)**

- Clarified our data selection [Response 2.3].
- Clarified our data diversity [Response 2.3].
- Clarified the comparison with one-stage methods [Response 3.5].

**(Explain questions)**

- Explained our advantages over other methods [Responses 2.1, 2.2].
- Explained the fairness of comparisons [Responses 1.1, 2.1].
- Explained the effectiveness of our dataset [Responses 2.3, 4.2].
- Explained potential improvements [Response 2.4].
- Explained validation on visual grounding tasks [Response 3.3].
- Explained open-source plans and experimental details [Responses 3.4, 3.6].
- Explained the design of dataset construction [Response 4.2].
- Explained the scaling problem [Response 4.4].
</content>
</thread_object>

</thread>
<thread>
creation date:
1732821299393

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
[Urgent] Please encourage the reviewers to participate in the rebuttal discussion.

comment:
Dear Area Chair,

As the extended Discussion Period is nearing its end, I kindly request your assistance in prompting **[** Reviewer gGZj **]** and **[** Reviewer tzNR **]** to respond to my comments. 

Notably, 
- **The only two responseive reviewers**, **[** Reviewer kRaB **]** and **[** Reviewer qHNc **]**, **decide to accept** my paper after reviewing my rebuttal. **[** Reviewer qHNc **]** has **the highest confidence score** toward our paper.
- **[** Reviewer gGZj **]** had previously indicated **a willingness to raise their score** based on my rebuttal and has **the highest confidence rate** among the reviewers who gave a score of 5. But they remain unresponsive.
- **[** Reviewer tzNR **]** remains unresponsive and has the lowest confidence rate.

Despite my repeated attempts to engage **[** Reviewer gGZj **]** and **[** Reviewer tzNR **]**, I have not yet received any response. Your help in expediting their feedback would be greatly appreciated.

Thank you for your time and support.

Best regards,
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces the 3D Intention Grounding (3D-IG) task, which aims to detect objects in 3D scenes based on human intention instructions. To support this task, the authors develop the Intent3D dataset and propose a method called IntentNet, which consists of three modules: candidate box matching, verb-object alignment, and cascaded adaptive learning.

Initial reviewer concerns focused on several aspects of the paper, including:

-	Unfair baseline comparisons (gGZj)
-	The motivation behind the 3D-IG task and the Intent3D dataset (gGZj)
-	The contribution of the verb-object alignment module (qHNc)
-	Clarifications regarding the advantages over large language models (LLMs) (tzNR)
-	Object selection and dataset diversity (tzNR)
-	The training process (kRaB)
-	Issues with equations and figures (qHNc)
-	Request for failure case analysis (gGZj)
-	The need for additional baseline comparisons (tzNR, kRaB)
-	The request for further ablation studies (kRaB)
-	Evaluation on traditional visual grounding tasks (tzNR)

The authors addressed these concerns comprehensively in their rebuttal, providing clarifications and additional experimental results where requested. Reviewer qHNc acknowledged that their concerns had been addressed, while the other three reviewers (gGZj, tzNR, and kRaB) did not respond to the rebuttal. After carefully reviewing the authors' responses and the remaining concerns, the AC believes that most issues have been adequately addressed.

Given that Intent3D represents a new and promising task for embodied AI, along with its contribution of a novel dataset and a strong baseline method, the AC recommends accepting this paper.