Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intention

## Abstract

Abstract In real-life scenarios, humans seek out objects in the 3D world to fulfill their daily needs or intentions. This inspires us to introduce 3D intention grounding, a new task in 3D object detection employing RGB-D, based on human intention, such as " I want something to support my back ". Closely related, 3D visual grounding focuses on understanding human reference. To achieve detection based on human intention, it relies on humans to observe the scene, reason out the target that aligns with their intention (" pillow " in this case), and finally provide a reference to the AI system, such as " A pillow on the couch ". Instead, 3D intention grounding challenges AI agents to automatically observe, reason and detect the desired target solely based on human intention. To tackle this challenge, we introduce the new Intent3D dataset, consisting of 44,990 intention texts associated with 209 fine-grained classes from 1,042 scenes of the ScanNet [ 10 ] dataset. We also establish several baselines based on different language-based 3D object detection models on our benchmark. Finally, we propose IntentNet , our unique approach, designed to tackle this intention-based detection problem. It focuses on three key aspects: intention understanding, reasoning to identify object candidates, and cascaded adaptive learning that leverages the intrinsic priority logic of different losses for multiple objective optimization.

## 1 Introduction

Performing object detection by following natural language instruction is crucial, as it can enhance the cooperation between humans and AI agents in real-world. Previous studies in Visual Grounding [, , , , , , ] have focused on localizing objects described by referential language in images. In recent years,
research interests have expanded from Visual Grounding in 2D images to 3D indoor scenes, now recognized as 3D Visual Grounding (3D-VG) [, , , , , , ].

In alignment with the sentiment expressed by Wayne Dyer, "Our intention creates our reality", humans identify and locate objects of interest for specific purposes or intentions. As depicted in Fig.1 (left), in real-world applications, to detect targets based on human intention, 3D-VG requires users to first observe the 3D world, reason about their desired target and provide a reference for the target’s category, attribution, or spatial relationship. However, various factors in daily use cases, such as engagement in intensive activities or experiencing visual impairments, may hinder humans from providing referential instructions. Therefore, the ability to automatically infer and detect the desired target based on user’s intentions, as shown in Fig.1 (right), becomes crucial and more intelligent.

Recently, problem akin to reasoning on human instruction have been explored in 2D images, such as intention-oriented object detection [], implicit instructions understanding [], and affordance understanding [, , , , ]. However, their focus restricts to a partial view of our reality. Instead, 3D data mirrors the actual world more closely, featuring depth information, complete geometric and appearance characteristics of objects, and a comprehensive spatial context, elements that are lacking in 2D images, as highlighted in 3D object detection (3D-OD) [, , , , , ]. Furthermore, understanding 3D spaces plays a crucial role in the advancement of embodied AI [, , ], autonomous driving [, , ], and AR/VR applications [, , ].
Therefore, intention-based detection in 3D settings not only better simulates and addresses real-world complexities but also holds greater promise for future advancements.

In this paper, we introduce the 3D Intention Grounding (3D-IG) task, which focuses on using human intention cues to automate reasoning and detection of desired objects in real-world 3D scenes.
Firstly, we devise a data generation strategy to collect our 3D-IG dataset named Intent3D. Human intentions, being diverse, may face limitations in generalization with predefined formats. Hence, we used ChatGPT, enriched with human daily life knowledge, to generate intentions for each object. Our Intent3D dataset comprises 44,990 intention texts representing 209 fine-grained classes of objects from 1,042 ScanNet [] point cloud scenes.
Note that 3D-IG is a multi-instance detection problem, for example, given "I want to present slides to my coworkers", we can infer multiple instances of "Monitor" in Fig. 1, making 3D-IG even more challenging. Secondly, to fully gauge our current research’s capability in solving this problem, we construct several baselines using three main language-based 3D-OD techniques for our benchmark. This involves evaluating our dataset using the following models: state-of-the-art expert model specifically designed for 3D visual grounding, existing foundation model formulated for generic 3D understanding tasks, and Large Language Model (LLM)-based model. We evaluate these baselines with multiple settings, namely training from scratch, fine-tuning, and zero-shot. Thirdly, we propose IntentNet, our novel method, to tackle this intention-based 3D object detection problem. To achieve a comprehensive understanding of intention texts, we introduce the Verb-Object Alignment technique, focusing on initially recognizing and aligning with the verb and subsequently aligning with the corresponding object features. For explicitly identifying relevant object candidates in the sparse 3D point cloud, we incorporate the Candidate Box Matching component.
Furthermore, to concurrently optimize the intention understanding, candidate box matching, and detection objectives, we propose a novel Cascaded Adaptive Learning mechanism to
adaptively scale up different losses based on their intrinsic priority logic.
We summarize our contributions in this paper as follows:

- •
We introduce a new task, 3D Intention Grounding (3D-IG), which takes a 3D scene and free-form text describing the human intention as input to directly detect the target of interest. To construct a benchmark for the 3D-IG task, we compile a new dataset called Intent3D, comprising 44,990 intention texts covering 209 fine-grained classes of objects from 1,042 3D scenes.
- •
We construct comprehensive baselines on our Intent3D, encompassing three main classes of language-based 3D-OD methods: expert model, foundation model, and LLM-based model.
- •
We propose a new method, IntentNet, for intention-based detection.
- •
We plan to release the dataset, source code and model checkpoints upon acceptance of the paper for future research development.

## 2 Related work

Referential Language Grounding
Visual Grounding (VG), also known as Referring Expression Comprehension [, , , , , , ], aims to detect the object by a bounding box, given a 2D image and a human referential instruction pertaining to the object of interest. Utilizing point cloud scans from ScanNet [], recently, ScanRefer [] extended this task to 3D Visual Grounding (3D-VG), aiming to predict a 3D bounding box for the target object given a 3D point cloud of the indoor environment and a human referential language.
Similarly, ReferIt3D [] proposes the 3D-VG task also based on ScanNet [] but with a simpler configuration, where it supplies ground truth boxes for all candidate objects in the scene and only needs approaches to choose the correct bounding box. Nevertheless, existing 3D-VG benchmarks predominantly evaluate the capability of methods in interpreting human referential instructions, typically encompassing the category, attribution, or spatial relationship of the object. The exploration of directly inferring and detecting the desired target based on human intention instructions in the 3D domain remains an uncharted area.

Affordance Understanding Derived from the concept in [], affordance detection aims to localize an object or a part of object in 2D [, , , ] or 3D [, ] focusing on feasible/affordable actions that can be executed while engaging with the object of interest. However, the affordable action still cannot convey human intention; for example, the action "grasp a bottle" does not necessarily equate to the underlying intention "I want something to drink". Furthermore, the dependency on phrase expressions [], closed-set classifications [, , , ], and object-centered approaches [] restricts its ability to interpret free-form intention and make inferences about 3D scenes.

Complex Instruction Understanding
To explore more complex instruction understanding,
RIO [] introduces a dataset and solutions for intention-oriented object detection based on MSCOCO [] 2D images. Moreover, previous studies also utilize the advancement of large multimodal models to address instruction reasoning in 2D open environments. LISA [] adopts large language model for understanding complex and implicit query texts to generate segmentation masks. Ferret [] reasons about multiple targets involved in instructions and introduces a dataset enriched with spatial knowledge to enhance model robustness. However, reasoning in 3D environments based on complex instructions, especially human intentions, remains underexplored.

Language-based 3D Object Detection Methods
Currently, the 3D-VG community has developed various expert (specialist) models [, , , , , , ], trained from scratch to comprehend the free-form language and sparse point clouds for detection task. State-of-the-art expert models like BUTD-DETR [] and EDA [] inherit the merit from MDETR [], focusing on understanding the target’s category or attribution expressed in the language.
3D-VisTA [], on the other hand, introduces a transformer-based foundation model to jointly train across multiple 3D tasks, including 3D-VG, 3D Dense captioning, 3D Question Answering, and 3D Situated Reasoning. Furthermore, inspired by advancements made in large multimodal models [, , , , , ], researchers are actively exploring comprehension of point cloud information through the utilization of large language models. Models such as PointLLM [] and Point-Bind [] primarily focus on object point cloud scans, while 3D-LLM [], Chat-3D [], and Chat-3D v2[] are designed to comprehend complex 3D environment point cloud scans and perform detection.

## 3 3D Intention Grounding

Here, we begin by defining the task of 3D Intention Grounding (3D-IG) in Section 3.1. Next, in Section 3.2, we delve into the construction of the Intent3D dataset, which serves as the benchmark for 3D-IG. Lastly, we provide comprehensive statistics for our dataset in Section 3.3.

### 3.1 Task Definition

3D Intention Grounding aims to detect 3D objects within a 3D scene to match a given human intention. It takes as input a 3D scene (e.g. "living room"), represented by a point cloud, and a free-form text (e.g. "I need a place to sit down and relax") describing a human intention. The goal is to predict all the 3D bounding boxes for all instances of the target object (i.e. detect all the "chair").

Figure: Figure 2: (Upper row) Flowchart for dataset construction. Using ScanNet’s data, we first select objects by three principal criteria: Common Object, Non-trivial Object, Unambiguous object. Then, we employ ChatGPT to generate intention texts given the prompt we designed. Finally, we manually clean the data. (Lower row) Examples from our dataset for different number of targets and length of texts.
Refer to caption: x2.png

### 3.2 Intent3D Dataset

Our 3D scenes and bounding box annotations are sourced from ScanNet [], which includes 1,513 indoor environment scenes of point clouds.
We use ChatGPT (GPT-4) to generate human intention text for selected objects in the 3D scenes.
The pipeline for our dataset construction is shown in Fig. 2 (upper row).

#### Step 1: Object Selection

We select the objects in each scene based on following three principal criteria. 1) Common Objects: We focus on common indoor objects with diverse human intentions, excluding structural elements like "wall" and "ceiling", and opting for objects occurring in at least four distinct scenes. 2) Non-trivial Objects: In each scene,
we select objects with fewer than six instances in their fine-grained categories,
thus excluding overly apparent objects and enhancing the challenge of our grounding task. 3) Unambiguous objects: Addressing ambiguity is crucial in generating human intention text, as objects with different fine-grained classes can fulfill the same human intention due to shared higher-level categories (e.g., "floor lamp" and "desk lamp" both classified under "lamp"), synonyms (e.g., "trash bin" vs. "garbage bin"), and variations between singular and plural terms (e.g., "bookshelf" vs. "bookshelves").
Furthermore, ambiguity also arises when objects belong to different higher-level categories but can fulfill the same human intention (e.g., both "TV" and "Monitor" can satisfy the intention "I want to display my chart to my co-workers").
To mitigate this, we manually identify ambiguous situations for each scene and filter out objects that serve similar human intentions.

#### Step 2: Text Generation

Unlike referential language in 3D-VG, which is associated with the deterministic characteristic of objects, intention language in 3D-IG stems from the diverse and spontaneous nature of human daily needs.
Hence, instead of prescribing specific formats or contents [, ] for language generation, we leverage ChatGPT (GPT-4), endowed with extensive real-world knowledge, to craft anthropomorphic intention text using a question-and-answer method. We construct the question in two parts: one for constraints (Prompt-1) and the other for scene context (Prompt-2). Specifically, the first part is designed to emulate human daily expressions of their intentions and preventing the disclosure of any referential details, such as the object’s category or location, in the response. This is done to challenge AI agents to engage in reasoning independently:

Prompt-1:
"You are a helpful assistant in providing human intention towards each object. The intention is from a first-person perspective, in the format of ‘I want / need / intend / … to …’. The intention must avoid mentioning synonyms, categories, locations, or attributes of the object."

In the second part, we supply ChatGPT with relevant information about the scene using the following template:

Prompt-2:
“In a $<$scene$>$ with $<$objects$>$, what can you do with each object?"

where <$scene$> refers to the type of scene and <$objects$> implies a list of all object classes present in the scene.
On average, we generate six intention texts per object from ChatGPT.
Fig. 2 (lower row) displays some samples we generated with varying lengths and different numbers of targets.

#### Step 3: Data Cleaning

Due to occasional nonsensical responses from ChatGPT, we manually filter out gibberish from the generated text. Furthermore, we discard responses that fail to generate. In cases of ambiguity, such as with the term "keyboard" in our dataset, where it should refer to a computer keyboard rather than a musical instrument, we manually recreate the correct intention sentences. Additionally, in samples with repeated intention sentences, we manually regenerate them to ensure diversity.

### 3.3 Dataset Statistics

Figure: Figure 3: Dataset statistics: (a) The distribution of the text lengths of all intentions; (b) Word cloud of verbs used in all intention texts; (c) Word cloud of nouns used in all intention texts; (d) Number of different verbs used for each fine-grained class; (e) Number of different nouns used for each fine-grained class.
Refer to caption: x3.png

We generate a total of 44,990 intention texts for 209 fine-grained object classes from 1,042 ScanNet scenes. The dataset includes 63,451 instances and 7,524 distinct scene-object pairs. On average, each scene contains 61 instances and 43 texts, with each scene-object pair receiving 6 intention texts.

Given that verbs and nouns play pivotal roles in expressing intentions, we conduct an in-depth analysis of these parts of speech. Within these texts, we identify a total of 1,568 different verbs and 2,894 different nouns. The greater variety of nouns compared to the number of object categories suggests that the language of intention is diverse and incorporates a wide-ranging vocabulary from human daily life. On average, the intention texts associated with each object category utilize 58 different types of verbs and 77 different types of nouns.

For a more granular understanding, we offer detailed statistics in Fig. 3. Specifically, (a) shows the distribution of text lengths; (b) presents a word cloud of the verbs found in all intention texts, excluding the template-derived verbs "want", "need", "aim", and "intend"; (c) shows a word cloud of the nouns, omitting the template-derived pronoun "I"; (d) and (e) respectively presents the number of distinct verbs and nouns used for each fine-grained class, which are organized in descending order and displayed every tenth data point for clarity.

The dataset is split into train, val, and test sets, including 35,850, 2,285, and 6,855 samples, respectively, each with disjoint scenes. Our train set comes from ScanNet’s train split, while the val and test sets are derived from its val split.

Figure: Figure 4: IntentNet: (Backbones) PointNet++ is used to compute point features, MLP is used to encode the boxes predicted by a 3D object detector, and RoBERTa is used to encode the text. (Encoder) Attention-based blocks are use for multimodal fusion. The box features are enhanced by fusing with text features. (Decoder) Point features with top-k confidence are selected as proposed queries and then updated by attention-based blocks. Several MLPs are used to linearly project the queries for following losses. (Losses) We learn to match the candidate boxes with target by $L_{bce}$. Queries are learned to notice verbs (by $L_{vPos}$), align with verbs (by $L_{vSem}$), and align with objects (by $L_{voSem}$) after being modulated by verbs. Cascaded adaptive factors are used for optimization. The colored boxes in the loss blocks represent different feature tokens.
Refer to caption: x4.png

## 4 Our Method

We introduce the structure of IntentNet in 4.1.
Given that 3D-IG poses a challenge in multimodal reasoning, demanding concurrent engagement in 3D perception, intention comprehension, and joint supervision of multiple objectives, we incorporate Candidate Box Matching for 3D understanding (Section 4.2), Verb-Object Alignment for text insight (Section 4.3), and Cascaded Adaptive Learning for joint supervision (Section 4.4). We elaborate on our pipeline below.

### 4.1 Multimodal Feature Extraction

Our proposed IntentNet adopts backbones and an encoder-decoder structure for multimodal feature extraction 4.1, similar to [, ].

Backbones
As shown in Fig. 4 (Backbones),
we use the pre-trained PointNet++ [] and RoBERTa [] as point cloud and language backbones to extract point features $P\in\mathbb{R}^{p\times c}$ and text features $T\in\mathbb{R}^{l\times c}$ respectively.
We also use a pretrained detector, GroupFree [], to detect 3D candidate boxes as additional visual reference. These candidate bounding boxes are subsequently encoded as box features $B\in\mathbb{R}^{b\times c}$ by MLP module.

Encoder
As shown in Fig.4 (Encoder), the point, box, and text features undergo N = 3 layers of
attention-based blocks for multimodal fusion. Specifically, in the Point Attention Block, point features perform self-attention and then cross-attention on text and box features. In the Box Attention Block, box features are enhanced by performing cross-attention on text features for the following matching objective. In the Text Attention Block, text features first perform self-attention and then cross-attention on point features.

Decoder
As shown in Fig. 4 (Decoder),
similar to [, ], point features are projected to predict whether they are within target boxes.
We select the top 50 point features, linearly project them as queries, and update them across M = 6 Decoder Attention Blocks, where
queries are involved in self-attention, cross-attention on text features, cross-attention on box features and cross-attention on point features.
Finally, the queries are fed into four MLPs to compute the alignment with verb and object text tokens, and predict the boxes. The box regression loss $L_{box}$ is the combination of L1 loss and 3D IoU loss, same as [, ].

### 4.2 Candidate Box Matching

Simple use of candidate box features serves as an additional visual reference for better scene understanding in previous 3D-VG methods [, ].
However, unlike 3D-VG which explicitly specifies the target object in its referential language, 3D-IG requires models to automatically infer the desired object
in the scene. Therefore, to tackle this reasoning challenge, it is necessary to explicitly identify candidate boxes in close proximity to the target for improved 3D perception associated with the intention-related task.
In Fig. 4 (Losses) Candidate Box Matching, we identify matched candidate boxes using binary cross entropy:

$$ $\displaystyle\begin{split}Confidence=\text{Sigmoid}(\text{MLP}(B)),~{}L_{bce}= \text{BCE}(Confidence,GT)\end{split}$ (1) $$

where MLP and Sigmoid project box tokens into confidence scores, $Confidence\in\mathbb{R}^{b\times 1}$. $L_{bce}$ is the binary cross entropy. $GT$ is the label, which is set to 1 for boxes whose IoU with target boxes exceeds 0.25, and 0 otherwise.

### 4.3 Verb-Object Alignment

Unlike referential language, where a single noun primarily denotes the target object, intention language demands that the model concurrently comprehends all verb-object pairs to fully grasp the intended behavior (see Fig. 4 (lower left)), such as "want"–"something", "support"–"back", and "relieve"–"pressure". Evidence of this can be seen in the statistics 3.3, where each object is described by a variety of verbs and nouns.
Therefore, as seen in Fig. 4 (Losses), to enhance intention understanding, we propose that the queries first develop the awareness of the verb tokens within the intention text by Verb Position
Prediction,
and then align with the semantics of the verb tokens by Query-Verb Contrastive Learning,
and finally, when modulated by the verb tokens, align with the semantics of the corresponding object tokens by Modulated Query-Object Contrastive Learning.
Specifically, we use the part-of-speech tagging and dependency parsing from spaCy [] to extract
the
position of the verbs and their corresponding objects (nouns).
For example, "010100" indicates a 6-token sentence with the verbs appearing at the second and the fourth position.
To recognize the verbs within the intention text, the queries are fed into a MLP to predict the distribution of verbs, while the label is attained by applying softmax to the verbs’ position:

$$ $\displaystyle\begin{split}\textit{V}_{dist}=\text{Softmax}(V_{pos}),~{}{L}_{ vPos}=\text{CE}(\text{MLP}(Q),\textit{V}_{dist})\end{split}$ (2) $$

where $V_{pos}\in\mathbb{R}^{l}$ is verbs’ position, $Q$ is query, ${L}_{vPos}$ is the cross-entropy loss.
Secondly, to comprehend the sematic of the verbs, we align the queries with the verb tokens by contrastive learning:

$$ (3) $$

where $\mathcal{T}$ and $\mathcal{Q}$ are the set of all the text tokens and queries, which are linearly projected to the same dimension. $\mathcal{T}_{l}$ and $\mathcal{Q}_{l}$ are their subsets for the linked verb tokens and queries.
Specifically, the matched queries of bipartite matching [] is linked with the verb tokens.
We append "not mentioned" at sentence ends, and link unmatched queries with the "not mentioned" token.
$k$ and $lv$ are the number of queries and verb tokens.
Thirdly, to further comprehend the verb-object combination, we compel the queries to infer the object token from the provided verb token. Specifically, both the queries and verb tokens undergo linear projection. For each verb-object combination in the sentence, the queries are modulated by element-wise multiplication with the verb token. Subsequently, we employ the same contrastive learning in Eq. 3 to align the modulated queries with the corresponding object token, denotes as ${L}_{voSem}$.
In inference, we calculate the confidence score based on $Softmax(Q^{T}T_{verb})$, where $Q$ is query and $T_{verb}$ is verb tokens.
We omit temperature and normalization in the similarity calculation in the above statement for ease of reading. We will open-source the code after acceptance to help everyone grasp the details.

### 4.4 Cascaded Adaptive Learning

Since we deal with multiple losses, optimizing all the objectives simultaneously becomes challenging. However, there is an inherent chain of logic among these objectives. Intuitively, the loss $L_{vPos}$ should be addressed before $L_{vSem}$, as the queries should first identify the verbs before they can understand their semantics. Similarly, $L_{vSem}$ should precede $L_{voSem}$, as understanding the semantics of the verbs is necessary before inferring the object token of that verb token. Finally, $L_{voSem}$ is a prerequisite for $L_{box}$, as understanding the verb-object combination is essential for intention understanding before intention-related detection.
Inspired by Focal Loss [], adaptively adjusting the loss based on difficulty can effectively optimize the learning process and make the best of training data.
Therefore, we leverage the aforementioned logic chain to propose the Cascaded Adaptive Learning scheme that transforms a loss of higher priority into a factor to adaptively adjust the loss of the subsequent priority.
Specifically, given the loss of A, say $Loss_{A}$, we compute the adaptive factor by $f(x)=x.sigmoid()+0.5$ (0.5 makes sure the factor is larger than 1). Then we module the $Loss_{B}$ which has the subsequent priority by $Loss_{B}=Loss_{B}*f(Loss_{A})$.
As shown in Fig.4 (Losses),
we cascade this process,
i.e. $L_{vPos}\rightarrow L_{vSem}\rightarrow L_{voSem}\rightarrow L_{box}$.

## 5 Experiments

We conduct comprehensive evaluations, including quantitative comparisons on our val and test set 5.4, ablation studies 5.5, and provide qualitative results 5.6.

### 5.1 Baselines

We construct several baselines by adopting three main types of language-related 3D object detection methods for the 3D-IG challenge: two expert models, one foundation model, and one LLM-based model.

Expert model We adopt two existing 3D-VG expert methods, BUTD-DETR [] and EDA [] (current SOTA).
These models use PointNet++ [] and RoBERTa [] as point cloud and language backbones. Additionally, they incorporate box candidates
predicted from a pretrained detector [] as additional visual reference and use transformer layers for multimodal fusion. Limited point features are then chosen for the decoding and prediction stages. We train these models from scratch using all ground truth boxes for all target instances.

Foundation model We fine-tune the SOTA foundation model, 3D-VisTA [], on our benchmark.
They use a pretrained 3D segmenter [] to segment candidate objects and PointNet++ to extract each object’s point clouds.
The text branch uses four layers of BERT []. A stack of transformer layers is used for multimodal fusion.
The model undergoes pretraining across various 3D tasks with unsupervised losses. For detection, it is fine-tuned to identify the best match instances and uses boxes from segmentation as detection results.

LLM-based model
Chat-3D v2 [] is the leading LLM-based method in 3D-VG.
They use a 3D segmenter [] to identify object candidates and encode them with a pretrained SOTA 3D encoder [].
They then project object features into the LLM space and fine-tune Vicuna-7B [], incorporating attribute, relation, and object ID learning.
Object ID strings are extracted from LLM responses and the corresponding boxes, derived from segmentation, serve as detection outputs.
We follow their code’s instruction, formatting all instance annotations into question-answering scripts for training.
In evaluation, we calculate the confidence score for each ID strings by applying Softmax to their logits of each token prediction and choosing the highest value as the tokens’ score. The final confidence score for each object ID is then the product of these scores.

### 5.2 Implementation Details

For baselines, we follow their configuration on ScanRefer [], using their official implementations. However, for BUTD-DETR [], we adopt the implementation in EDA [] since we do not provide text span labeling, which has been further discussed in []. We train these two expert models from scratch.
For the foundation model, 3D-VisTA [], we adopt their pretrained checkpoint and finetune it.
For the LLM-based model, Chat-3D-v2 [], we also use their pretrained checkpoint and conduct both zero-shot evaluation and finetuning based on official settings. Specifically, for finetuning, we follow their setting to train the last two layers of Vicuna-7B [] along with the rest linear layers.
Our IntentNet is trained from scratch for 90 epochs with a 24 batch size.
The learning rate is 0.001 for PointNet++ and 0.0001 for the rest of the network, which is decayed by 0.1 at the 65th epoch.
The RoBERTa is frozen.
The number of point tokens is 1024, and the maximum length for text tokens is set to 256. The hidden dimension used is 288.
All the methods are evaluated on the val and test set of Intent3D using their best checkpoint based on the AP@0.5 from the val set.

**Table 1: 3D intention grounding results on Intent3D’s val set. ^0 indicates the zero-shot results. The best results are in bold, and the second best results are underlined.**
| Method | Detector | Top1-Acc@0.25 | Top1-Acc@0.5 | AP@0.25 | AP@0.5 |
| --- | --- | --- | --- | --- | --- |
| BUTD-DETR [] | GroupFree [] | 47.12 | 24.56 | 31.05 | 13.05 |
| EDA [] | Group-Free [] | 43.11 | 18.91 | 14.02 | 5.00 |
| 3D-VisTA [] | Mask3D [] | 42.76 | 30.37 | 36.1 | 19.93 |
| Chat-3D-v2^0 [] | PointGroup [] | 5.86 | 5.24 | 0.15 | 0.13 |
| Chat-3D-v2 [] | PointGroup [] | 36.71 | 32.78 | 3.23 | 2.58 |
| IntentNet (ours) | GroupFree [] | 58.34 | 40.83 | 41.90 | 25.36 |

**Table 2: 3D intention grounding results on Intent3D’s test set. ^0 indicates the zero-shot results. The best results are in bold, and the second best results are underlined.**
| Method | Detector | Top1-Acc@0.25 | Top1-Acc@0.5 | AP@0.25 | AP@0.5 |
| --- | --- | --- | --- | --- | --- |
| BUTD-DETR [] | GroupFree [] | 47.86 | 25.74 | 31.41 | 13.46 |
| EDA [] | Group-Free [] | 44.00 | 19.62 | 14.56 | 5.18 |
| 3D-VisTA [] | Mask3D [] | 43.88 | 31.44 | 37.29 | 22.00 |
| Chat-3D-v2^0 [] | PointGroup [] | 5.63 | 4.93 | 0.14 | 0.11 |
| Chat-3D-v2 [] | PointGroup [] | 33.46 | 29.32 | 2.67 | 2.1 |
| IntentNet (ours) | GroupFree [] | 58.92 | 42.28 | 44.01 | 27.60 |

### 5.3 Evaluation Metrics

We use Top1-Accuracy and Average Precision for evaluation. Top1-Accuracy gauges the accuracy of the model’s highest confidence prediction, while Average Precision evaluates precision across varying confidence thresholds. We calculate the Intersection over Union (IoU) between predicted and ground truth boxes with thresholds of 0.25 and 0.5 to indicate correctness.

### 5.4 Quantitative Comparisons

Tab. 1 and Tab. 2 show quantitative comparisons on val and test set of Intent3D.

Results of our IntentNet Due to explicit modeling the comprehension of intention language and reasoning on box candidates with cascaded optimization, our IntentNet outperforms all the previous methods by a large margin, achieving 11.22%, 8.05% improvement on Top1-Acc@0.25 and Top1-Acc@0.5 and 9.12%, 5.43% improvement on AP@0.25 and AP@0.5 on the val set, compared with the second best method.
Similarly, on the test set, our IntentNet outperforms all the previous methods by a large margin, achieving 11.06%, 10.84% improvement on Top1-Acc@0.25 and Top1-Acc@0.5 and 6.72%, 5.6% improvement on AP@0.25 and AP@0.5 compared with the second best method.

Results of expert models The expert models mostly focus on aligning with nouns on the sentence, since they are originally built for targeting referential language. However, noun is not the target but just one of the object for a certain verb in our intention language. Therefore, they perform inferiorly compared with our IntentNet, where we explicitly comprehend all the verb-object parts. It worth notice that a better 3D-VG model, EDA, performs worse than its baseline, BUTD-DETR, since EDA further digs into the alignment with the attributes or auxiliary object toward the noun in the sentence, which is not the target in our case and thus misleads the model.

Results of foundation model The foundation model, 3D-VisTA, performs better than the expert models on most metrics, which is attributed to its widely multimodal alignment from pretraining. However, it still falls short of our IntentNet, since their full reliance on the detection outputs from the detector [], which are not perfect on evaluation. Instead, we only take the predictions from the pretrained detector as visual reference and provide reasoning on those box candidates. Therefore, even if we are using a less powerful detector [] than theirs, we still outperform them.

Results of LLM-based model The LLM-based model, Chat-3D-v2, performs the worst in both zero-shot and finetuning on most metrics, which is expected given that current LLM-based models perform poorly on 3D-VG [] and 3D-IG is even more challenging.
The main bottleneck might still be the lack of training data for aligning 3D visual features into LLM space [, ]. Also, due to hallucination problems in the LLM, Chat-3D-v2 performs significantly worse on AP metrics, which are more sensitive to the false positive predictions. Its decent performance on Top1-Acc is mainly due to the use of a pre-trained detector [] and the powerful LLM to reason out the correct category of the target.

**Table 3: Ablation studies on Intent3D’s val set. "Verb" indicates the alignment with verb tokens in the sentence, which is learned by $L_{vPos}$ and $L_{vSem}$. "Verb2Obj" indicates the alignment with object token in the sentence when the queries are modulated by the verb. "MatchBox" indicates the Candidate Box Matching, "Adapt" indicates the Cascaded Adaptive Learning.**
| ID | Verb | Verb2Obj | MatchBox | Adapt | Top1-Acc@0.25 | Top1-Acc@0.5 |
| --- | --- | --- | --- | --- | --- | --- |
| (a) |  | ✓ | ✓ | ✓ | 53.09 | 34.62 |
| (b) | ✓ |  | ✓ | ✓ | 57.87 | 39.42 |
| (c) | ✓ | ✓ |  | ✓ | 56.25 | 38.37 |
| (d) | ✓ | ✓ | ✓ |  | 57.39 | 36.93 |
| (e) | ✓ | ✓ | ✓ | ✓ | 58.34 | 40.83 |

### 5.5 Ablation Studies

Ablation studies are conducted on the val set to evaluate the effectiveness of each component in our IntentNet.
As shown in Tab. 3, we apply controlling variables, removing one component in each experiment based on the comparison with the complete verison (e).
We observe that every component is essential for optimal performance, as the exclusion of any component results in a notable decline in performance.
Specifically, the comparison between (a) and (e) shows that the alignment with verb tokens in the sentence is foundamental for the model to understand the intention, which is learned by $L_{vPos}$ and $L_{vSem}$. Omitting this causes a significant reduction in performance. The contrast between (b) and (e) illustrates that additionally learning about reasoning about the object given verb can further improve the comprehension of the intention. The evaluation between (c) and (e) demonstrates that the Candidate Box Matching effectively extracts distinctive features near the target boxes to boost overall performance. Lastly, the comparison between (d) and (e) confirms that Cascaded Adaptive Learning facilitates the model to concurrently optimize each of the objectives.

Figure: Figure 5: Qualitative results of ablation studies. "Verb" indicates the alignment with verb tokens. "Verb2Obj" indicates the alignment with object token in the sentence when the queries are modulated by the verb. "MatchBox" indicates the Candidate Box Matching, "Adapt" indicates the Cascaded Adaptive Learning. Red boxes indicate the ground truth, Blue boxes indicate the counterpart model’s prediction, and Green boxes indicate our model’s prediction.
Refer to caption: x5.png

### 5.6 Qualitative Results

We provide qualitative results of ablation studies in Fig. 5 to further demonstrate the effectiveness of each component in our IntentNet.
The first column: The model without alignment with the verb fails to grasp the intention to identify the target. This highlights the fundamental importance of aligning with verb tokens.
The second column: The model lacking alignment with object tokens when given verb tokens does not fully capture the intention. In this context, "add" is too vague to specifically identify "plant". However, with a better understanding of the object "greenery", our final model can accurately detect the intended object.
The third column: The model without the Candidate Box Matching cannot effectively extract distinctive box features near the target, leading it to identify irrelevant items.
The fourth column: The model lacking Cascaded Adaptive Learning does not efficiently optimize each objective. Hence, despite understanding the intention and reasoning about the target "chair" through contrastive learning, it fails to localize all instances of the target due to insufficient optimization of the regression loss.

## 6 Conclusion

In this paper, we propose a new challenging task, 3D Intention Grounding, designed to detect target objects based on human intention within 3D scenes. We introduce a new dataset, Intent3D, with detailed statistics to support this task. To tackle this problem, we first construct comprehensive baselines involving current expert model, foundational model, and LLM-based model. Finally, we propose a novel method, IntentNet, which includes three key aspects: intention comprehension, reasoning to identify object candidates, and cascaded optimization of objective functions, to effectively perform 3D Intention Grounding.

## 7 Additional Discussion of Intent3D

Failed Cases
Here we aim to provide more details of failed cases in the responses from the GPT. As shown in Table 4, (1) the GPT sometimes fails to generate a complete sentence. (2) The GPT may generate a sentence that includes Unicode characters. (3) The GPT may generate a sentence that is not directly related to the object, since the name of the object’s category is polysemous. For instance, "keyboard" can refer to a musical instrument, but in our dataset, it actually denotes a computer peripheral. (4) We exclude objects whose categories are too general, such as "machine", in which cases the GPT fails to generate a specific intentional text.

**Table 4: Examples of failed cases in the responses from the GPT.**
| (1) "bathroom cabinet" : | "I" |
| --- | --- |
| (2) "chair" : | "I need to sit for the duratio\u043d of the conference" |
| (3) "keyboard" : | "I need to compose music for my upcoming online performance" |
| (4) "machine" : | "I want to utilize the object (depending on the type) to assist me in tasks" |

API Usage
In employing ChatGPT (GPT-4), adjustment of the temperature parameter enables control over output randomness. Higher temperature values foster greater diversity and creativity, while lower values yield more deterministic but potentially repetitive responses. We set temperature=1.2 to optimize the diversity of model outputs.
We have explored alternative GPT versions such as gpt-3.5-turbo-1106, and gpt-4-1106-preview. However, empirical evaluation revealed that GPT-4 consistently outperforms others, hence our preference for its utilization.

Limitations
The quality and variety of intention sentences generated for each object class are contingent upon the performance of the GPT-4. Despite GPT-4 demonstrating significant proficiency in real-world knowledge and our efforts to refine outputs manually, primarily addressing garbled cases, inherent subjectivity might be potentially introduced by GPT-4’s training data.

## 8 Additional Implementation Details

Here, we provide additional details on the implementation of each method.
For BUTD-DETR [], we adhere to its official configuration in ScanRefer []. The batch size is set to 24, and the learning rate (which is the same as ours) decreases to one-tenth at the 65th epoch. We follow the suggestion in its source code to continue training the model while monitoring the validation accuracy until the model converges. It finally takes 100 epochs to converge.
For EDA [], we also follow its official configuration, where the batch size is set to 48. The learning rate for backbones is 0.002, and for the rest, it is 0.0002. The learning rate decreases to one-tenth at the 50th and 75th epochs. It takes 104 epochs to converge.
For 3D-VisTA [], we use its official configuration in ScanRefer [], where the batch size is set to 64, and the learning rate is 0.0001. The warm-up step is set to 5000. Although the official training epoch is 100, we find that the model converges at the 47th epoch. We adopt its pre-trained checkpoint provided in the source code for fine-tuning.
In the case of Chat-3D v2, we follow its official instructions to prepare the annotations. We adopt its checkpoint provided in "Step 3: Scene-level Alignment" for fine-tuning. It takes 3 epochs to fine-tune the model.

Figure: Figure 6: Visualization of correct cases and false cases of IntentNet. Red boxes are ground truth boxes, green boxes are prediction outputs

## 9 Additional Qualitative Results of IntentNet

In this section, we present additional qualitative results of our IntentNet, showcasing both correct and false cases.

Correct Cases
In the upper row (Fig. 7(a)), IntentNet demonstrates impressive accuracy in various scenarios where the number of target instances exceeds one and the instances are not closely situated, as observed in cases of "cabinet" or "refrigerator". Furthermore, for the less common intention "enjoy aesthetical pleasure", IntentNet also performs well.

False Cases
In the lower row (Fig. 7(b)), we highlight cases where IntentNet fails. Particularly, in the case of "washing machines", IntentNet struggles to detect all instances in large environments with multiple targets (four instances). However, we also find that some false cases attribute to the ambiguity or subjectivity in the bounding box annotations provided by ScanNet []. For instance, in the "desk" case, the annotation includes a double-person desk, while IntentNet detects only one of the single-person desks, which is also reasonable.

## 10 Additional Discussion of Ambiguity

In our dataset, intentions and objects aren’t one-to-one association. For example, “I want to sleep" relates to both “couch" in living room and “bed" in bedroom. However, the reason we generate intention towards a “bed” in a “bedroom" that does not have a “couch" in it, is to provide a clear and unambiguous mapping of intention to object for better training.
As shown in Fig. 7, we provide a demo where we input “I intend to rest my legs” with the above ambiguous scenario where multiple objects can fulfill this intention. When we lower the confidence score filter to allow multiple outputs, our model can ground many reasonable objects (“footrest”, “couch”, “table”, etc.), since these objects have similar intentions in our training set but just appear in different scenes for unambiguous training.

Figure: Figure 7: Example for multiple objects prediction.

## 11 Additional Comparison with GPT-4

We provide an additional comparison with a two-stage method based on GPT-4.
For the two-stage method, we fairly use the same detector as ours to predict bounding boxes and category. Given the intention and the detector’s predictions, GPT-4 selects the most relevant objects. Results are shown on validation set in Table 5. Due to the detector’s prediction errors and hallucinations from GPT-4, it performs poorly in AP metric.

**Table 5: Compare with two-stage method based on GPT-4.**
| Method | Detector | top1-acc@0.25 | top1-acc@0.5 | ap@0.25 | ap@0.5 |
| --- | --- | --- | --- | --- | --- |
| GPT-4 | GroupFree | 41.40 | 28.40 | 15.10 | 7.76 |
| IntentNet | GroupFree | 58.34 | 40.83 | 41.90 | 25.36 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see the weakness section. 

I am willing to increase my score if the author resolves my major concern with some additional baseline experimentations.
soundness: 3
strengths: 1. The paper presents a clear motivation for 3D-intention grounding, and it includes clear illustrations and presentations of dataset collection procedure.

2. Soundness of each component design of the IntentNet, and thoroughly ablations on each component of the proposed pipeline design.

3. Extensive experiments and discussions demonstrate the effectiveness of the proposed framework compared to different types of baselines.
summary: This paper introduces a new task named 3D-intention grounding, which is 3D object-detection from direct human-intentions. The paper collects Intent3D dataset which includes 1042 scenes from ScanNet and corresponding paired human-intentions questions and 3D object detections answers. IntentNet is proposed to tackle 3D-intention grounding task by candidate box-matching, verb-object alignment and cascaded adaptively learning.
weaknesses: Major concern:

I am concerned about possible baseline unfair comparison in the experiment section. Most baselines are designed to tackle nouns-types of questions instead of human-intention types of questions. What if we pass the question to a finetuned LLM and let it infers what types of nouns/objects the question is targeting at from possible objects in a scene detected by existed 3D object detectors? The possible performance of these baselines might be much higher after it is given the object it is expected to detect in a scene. 

Minor concern:
It would be interesting if the author can provide some cases where IntentNet fails but other models succeed. Particularly if other models are fed with object/noun directly.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: 1：There has been a few methods to combine 3D scene understanding with LLM beyond Chat3D v2, such as LL3DA, Grounded 3D-LLM, ReGround3D and so on.  What are the advantages of this paper compared to theirs? More experimental evidence is needed to demonstrate the advantages of the this paper over other methods.

2: Current large language models can also directly infer human intentions; what are the advantages of this paper compared to them?

3：When filtering Non-trivial Objects, objects with more than six instances in fine-grained categories are directly removed, which may lead to the exclusion of commonly used objects. Could we consider adding more fine-grained descriptions for these objects instead of outright deletion?

4:  Figures 3 (d) and (e) indicate that the dataset lacks sufficient diversity in the types of objects included. Experimental validation is necessary to determine whether the variety of object types included in the dataset is sufficient for the model to learn effectively.
soundness: 2
strengths: 1: A new task in 3D object detection employing RGB-D, based on human intention, facilitates smoother and more natural communication between humans and intelligent agents.

2:The author propose a high-quality vision-language dataset and focuses on the human’s intention for 3D object detection, which will facilitate the progress of 3D scene understanding.
summary: This paper introduces the task of 3D Intention Grounding (3D-IG), which aims to automate the reasoning and detection of target objects in real-world 3D scenes using human intention cues. To this end, the authors constructed the Intent3D dataset, comprising 44,990 intention texts across 209 fine-grained object categories, and developed several baseline models to evaluate various 3D object detection techniques. Finally, the authors proposed a novel method, IntentNet, which optimizes intention understanding and detection tasks through techniques such as verb-object alignment and candidate box matching, achieving state-of-the-art performance on the Intent3D benchmark.
weaknesses: 1: There has been a few methods to combine 3D scene understanding with LLM beyond Chat3D v2, such as LL3DA, Grounded 3D-LLM, ReGround3D and so on.  The paper does not highlight the advantages compared to them.

2: The object selection method is too crude, as it removes some commonly used objects by humans when filtering Non-trivial Objects. Figures 3 (d) and (e) indicate that the dataset lacks sufficient diversity in the types of objects included.

3: The limited variety of object category included in the dataset, fails to demonstrate the grounding effect for the missing objects in the dataset.

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Will the proposed dataset and code be open-sourced?
2. IntentNet seems to be a two-stage approach, where a pre-trained 3D detector first extracts proposals, which are then matched with text. Are there any one-stage methods that directly fuse 3D data and text to generate boxes? If so, the paper does not seem to provide comparisons with such methods.
3. Regarding the training process, on what hardware was the method trained, and how long did the training take?
soundness: 3
strengths: 1. The overall quality of the paper is high, with clear writing and easy-to-understand presentation.
2. The contribution of the dataset is significant, as it is the first to construct a 3D detection task focused on intention understanding.
summary: The paper introduces a 3D Intention Grounding (3D-IG) task and constructs a novel dataset called Intent3D (sourced from ScanNet data and generated using GPT). Additionally, it proposes a baseline model named IntentNet. I am not an expert in 3D-related fields, so my confidence in this review is not very high.
weaknesses: 1. More comparisons with recent works should be provided in Tables 1 and 2. Additionally, there is a minor mistake: the detector names “GroupFree” and “Group-Free” in the first two rows of Tables 1 and 2 do not match.
2. The article gives a subtractive ablation experiment. I would like to see an additive ablation experiment, such as how the effect of verb alone works.
3. The article does not give the performance of the proposed IntentNet in traditional 3D grounding.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Please refer to the weakness part/
soundness: 3
strengths: 1. The introduction of the 3D Intention Grounding (3D-IG) task could contribute to the 3D visual grounding community.
2. The proposed Intent3D dataset is extensive, comprising 44,990 intention texts linked to 209 fine-grained object classes from 1042 3D scenes. This dataset provides a valuable resource for training and evaluating models in the context of human intention.
3. The proposed modules in IntentNet are generally technically sound.
summary: This paper presents a novel framework for 3D object detection that integrates human intention into the detection process. The authors introduce the Intent3D dataset, aiming to enhance the model's understanding of human needs in real-world scenarios. The proposed method, named IntentNet, employs a multi-instance detection approach, where the model is tasked with identifying multiple instances of objects based on free-form textual descriptions of human intentions. The authors evaluate their approach against several baselines, including expert models designed for 3D visual grounding, foundation models for generic 3D understanding tasks, and Large Language Model (LLM)-based models. The evaluation demonstrates the effectiveness of IntentNet in achieving state-of-the-art performance on the Intent3D benchmark.
weaknesses: 1. Object detection based on human intention is indeed a new task. However,  why should we need to have a dataset and a model dedicated to this task? I think the detection based on human intention can be achieved in a modulized manner. For example, first, let the 3D detection module detect all types of objects in the scene. Then ask  LLM to decide the subsets of the detected objects that can fulfill the human intention. I think this could be more flexible compared with training a dedicated detection model based on human intention prompts. 
2. In L210, it is mentioned that around six intention texts are generated per object. How do you determine this number (six)? Can it guarantee that all possible intentions can be covered and trained well?
3. The contents around Eq (3) are hard to understand for me. What is t in Eq(3)? Although the authors claimed that the code would be released to facilitate the understanding, it would be better to add a figure to illustrate the connections in the network.  
4. Figure 4 is too abstract. Even though many connections and modules are included, it is not very informative. I suggest more detailed diagrams can be provided for the key modules further.   
5. Figure 5 shows that the verb alignment is very helpful for the prediction quality. Do you think this is caused by the limited training data? If more data is available for training, would this module still be essential?

Meta-review: This paper introduces the 3D Intention Grounding (3D-IG) task, which aims to detect objects in 3D scenes based on human intention instructions. To support this task, the authors develop the Intent3D dataset and propose a method called IntentNet, which consists of three modules: candidate box matching, verb-object alignment, and cascaded adaptive learning.

Initial reviewer concerns focused on several aspects of the paper, including:

-	Unfair baseline comparisons (gGZj)
-	The motivation behind the 3D-IG task and the Intent3D dataset (gGZj)
-	The contribution of the verb-object alignment module (qHNc)
-	Clarifications regarding the advantages over large language models (LLMs) (tzNR)
-	Object selection and dataset diversity (tzNR)
-	The training process (kRaB)
-	Issues with equations and figures (qHNc)
-	Request for failure case analysis (gGZj)
-	The need for additional baseline comparisons (tzNR, kRaB)
-	The request for further ablation studies (kRaB)
-	Evaluation on traditional visual grounding tasks (tzNR)

The authors addressed these concerns comprehensively in their rebuttal, providing clarifications and additional experimental results where requested. Reviewer qHNc acknowledged that their concerns had been addressed, while the other three reviewers (gGZj, tzNR, and kRaB) did not respond to the rebuttal. After carefully reviewing the authors' responses and the remaining concerns, the AC believes that most issues have been adequately addressed.

Given that Intent3D represents a new and promising task for embodied AI, along with its contribution of a novel dataset and a strong baseline method, the AC recommends accepting this paper.

Decision: Accept (Poster)