Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning

## Abstract

Abstract Recent advancements in 3D Large Language Models (3DLLMs) have highlighted their potential in building general-purpose agents in the 3D real world,
yet challenges remain due to the lack of high-quality robust instruction-following data, leading to limited discriminative power and generalization of 3DLLMs. In this paper, we introduce Robin3D, a powerful 3DLLM trained on large-scale instruction-following data generated by our novel data engine, Robust Instruction Generation (RIG) engine. RIG generates two key instruction data: 1) the Adversarial Instruction-following data, which features mixed negative and positive samples to enhance the model’s discriminative understanding. 2) the Diverse Instruction-following data, which contains various instruction styles to enhance model’s generalization.
As a result, we construct 1 million instruction-following data, consisting of 344K Adversarial samples, 508K Diverse samples, and 165K benchmark training set samples.
To better handle these complex instructions, Robin3D first incorporates Relation-Augmented Projector to enhance spatial understanding,
and then strengthens the object referring and grounding ability through ID-Feature Bonding. Robin3D consistently outperforms previous methods across five widely-used 3D multimodal learning benchmarks, without the need for task-specific fine-tuning.
Notably, we achieve a 7.8% improvement in the grounding task (Multi3DRefer) and a 6.9% improvement in the captioning task (Scan2Cap).
Code is available at https://github.com/WeitaiKang/Robin3D .

## 1 Introduction

Spatial Intelligence [Li, ]
refers to the ability of AI to understand the 3D world and reason within 3D space. Related ideas, such as Embodied AI [Duan et al., ] and Robotic Agent [Bousmalis et al., ], express a similar aim to build general-purpose assistants in the 3D real world.
To achieve this goal, researchers have drawn inspiration from the success of 2D Multimodal Large Language Models (MLLMs) [Liu et al., ; You et al., ] and have started exploring the potential of 3D Large Language Models (3DLLMs) to create general agents [Hong et al., ; Chen et al., ; Wang et al., ; Huang et al., ]
in the 3D domain or to attain Spatial Intelligence.

Instruction-following tuning [Liu et al., ; ; ] in MLLMs refers to training the LLM to execute natural language commands by integrating both textual and visual information.
In contrast to the versatile image-text pairs employed for training 2D MLLMs, collecting 3D instruction-following data for 3DLLM remains a significant challenge. Although existing works have made progress [Hong et al., ; Huang et al., ; Chen et al., ] in generating more instruction data, they still lack robustness in two aspects: 1) Most of the existing instruction data consist of positive pairs, lacking adversarial or negative samples. Therefore, models trained on such data tend to be less discriminative because they might overfit to the positive pairs and are more likely to hallucinate positive responses to any input. 2) Some instruction data also lack diversity in language styles, as human annotators or generative models [; Wang et al., ] are typically asked to follow fixed instructions when describing objects [Chen et al., ; ], or the data is generated using predefined templates [Achlioptas et al., ], which may limit models’ generalizability.

To address these challenges, we introduce Robin3D, a robust and powerful 3D Large Language Model tuned on large scale instruction-following data generated by our novel data engine, Robust Instruction Generation (RIG) engine.
Specifically, RIG is designed to generate two types of data:
❶
Adversarial Instruction-following data, which is characterized by mixing adversarial or negative samples with positive ones. This process decouples the potential memorized positive pairs in the training set, leading to a more discriminative understanding of individual objects and instructions.
To present a comprehensive adversarial dataset, we cover both object-level and scene-level instructions, from category-based identification problems to expression-based reasoning challenges, resulting in four new tasks.
❷
Diverse Instruction-following data, which first comprehensively collects various types of instructions from existing studies or transforms current tasks into instruction-following format.
To harness the powerful in-context learning capability of LLMs, we use ChatGPT [] to diversify the language styles of the instructions by crafting specific prompt engineering templates tailored to each task.
Combining these with the original training sets of current benchmarks, we construct 1 million instruction-following samples, with approximately 344K adversarial data (34%), 508K diverse data (50%), and 165K benchmark data (16%), as shown in Fig. 1 (right).

Figure: Figure 1: Robin3D surpasses previous SOTA across all the benchmarks (left), by training on our RIG-generated 1 million data (right).
Refer to caption: x1.png

Akin to current SOTA approaches [Huang et al., ; ], our Robin3D model uses pre-trained 3D [Zhou et al., ] and 2D [Oquab et al., ] models to effectively represent 3D scenes at the object level, leveraging object IDs [Huang et al., ] to refer to the objects in the input or ground them in the output. However, the original methods inevitably weaken the spatial relationships between object features due to the object-centric normalization in their 3D backbone [Zhou et al., ], which hinders the learning of our diverse visual grounding data. Moreover, in the current approaches [Huang et al., ; ], object IDs are simply appended to object features in an interleaved manner, which may lack sufficient connection between the IDs and the features, making it struggle with complex referring and grounding requirements in our adversarial instruction data.
To tackle these problems, Robin3D first integrates the Relation-Augmented Projector (RAP) to enhance spatial relationship understanding by enriching object-centric features with scene-level context and positional information. We then introduce ID-Feature Bonding (IFB), which bonds each ID to its corresponding features by wrapping the features with identical ID tokens. IFB further reinforces this connection by a post-vision order, resulting in a more informative ID-feature sequence to improve the referring and grounding ability.

We conduct comprehensive experiments across five representative 3D multimodal learning benchmarks, including ScanRefer [Chen et al., ], Multi3DRefer [Zhang et al., ], Scan2Cap [Chen et al., ], ScanQA [Azuma et al., ], and SQA3D [Ma et al., ]. As shown in Fig. 1 (left), our Robin3D achieves state-of-the-art performance across all the benchmarks without fine-tuning on specific tasks, making a significant step towards Spatial Intelligence.

In summary, our contributions are threefold: (i) We introduce Robin3D, a powerful 3DLLM trained on robust instruction-following data generated by our novel data engine, RIG. (ii) We incorporate two modules into Robin3D, RAP and IFB, to enhance its spatial understanding as well as its referring and grounding capabilities when handling our complex instructions. (iii) Our Robin3D achieves state-of-the-art performance across five widely-used benchmarks without task-specific fine-tuning.

## 2 Related Work

##### 3D Vision-Language Learning

Recent advancements in 3D vision-language (3D-VL) learning [Chen et al., ; Achlioptas et al., ; Azuma et al., ; Chen et al., ; Kang et al., ] have focused on bridging the gap between 3D scene understanding and natural language, which is essential for developing embodied intelligence.
Similar to 2D vision-language learning [Kazemzadeh et al., ; Kang et al., ; Johnson et al., ; You et al., ; Kang et al., ; Antol et al., ; Zhang et al., ; Kang et al., ], a variety of tasks such as 3D Visual Grounding [Chen et al., ; Zhang et al., ; Achlioptas et al., ], 3D Dense Captioning [Chen et al., ], and 3D Question Answering [Azuma et al., ; Ma et al., ] have been proposed to evaluate the ability of human instruction following in relation to 3D object properties and spatial relationships.
Initial efforts focus on building a task-specific model for a single tasks, such as EDA [Wu et al., ] for grounding and Vote2Cap-DETR++ [Chen et al., ] for captioning. Recent research has shifted towards developing unified models capable of handling multiple 3D scene-language tasks. Approaches like 3DJCG [Cai et al., ] and D3Net [Takahashi & Mitsufuji, ] leverage task synergies, while 3D-VisTA [Zhu et al., ], 3D-VLP [Jin et al., ] and PQ3D [Zhu et al., ] introduce pre-training techniques and unified representations to align 3D visual data with language. However, their dependence on task-specific heads restricts their flexibility for more generalized user-assistant interactions.

##### 3D Large Language Model

Following the success of 2D Multimodal Large Language Models [You et al., ; Zhang et al., ; Liu et al., ; ; ], researchers begin to explore the reasoning abilities and promising generalizability of LLMs in human instruction following within the 3D domain, leading to the emergence of 3D Large Language Models (3DLLMs). Models like PointLLM [Xu et al., ] and Imagebind-LLM [Han et al., ] show strong performance in object-level tasks by mapping 3D data into LLMs. However, they face difficulties in handling scene-level reasoning.
3D-LLM [Hong et al., ] incorporates positional embeddings and location tokens,
and Oryx [Liu et al., ] offers a solution to support multi-view arbitrary resolution image.
However, their reliance on 2D encoders limits its ability to fully capture 3D spatial structures. Models such as LL3DA [Chen et al., ], Chat-3D [Wang et al., ], LEO [Huang et al., ], and Scene-LLM [Fu et al., ] have made progress in improving scene-level dialogue capabilities, showing promising results in question-answering and captioning tasks.
However, their insufficient visual grounding capability limits their application in Embodied AI or Robotic Agents, which require precise object localization and manipulation following human instruction.
To further enhance grounding abilities, Grounded 3D-LLM [Chen et al., ] introduces referent tokens and contrastive learning to unify grounding and textual responses. Similarly, Chat-3D v2 [Huang et al., ] proposes the use of object identifiers (object IDs) for referring and grounding. Building on Chat-3D v2[Huang et al., ], Chat-Scene [] further incorporates DINO v2 [Oquab et al., ] to provide strong multi-view, object-centric 2D representations.
Despite these advancements, current 3D LLMs, which are trained solely on positive 3D visual-language pairs and template-based instructions, suffer from suboptimal generalization and a potential for overfitting.

## 3 Methodology

### 3.1 Preliminary

To train a 3D LLM using instruction fine-tuning, we first represent the 3D scene as a sequence of vision tokens, then append it with system and question prompts, expressed as sequences of language tokens, to indicate the task. Taking the above tokens as input, a LLM is supervised to output the answer tokens via next token prediction.
Specifically,
as shown in Fig. 2, given the point cloud of a 3D scene, we use the pre-trained 3D segmenter Mask3D [Schult et al., ] to extract object features along with their corresponding 3D masks. Following , we further sample each object’s point cloud based on the 3D masks, normalize it, and employ the pre-trained Uni3D [Zhou et al., ] to extract unified object-centric 3D features. Additionally, 2D masks projected from the 3D masks are used to sample and average 2D features, which are extracted by DINO v2 from multi-view images of each object.
Our Relation-Augmented Projector fuses the 3D features and position embeddings from Mask3D and Uni3D into our final 3D features.
In line with , we incorporate special tokens $\{<\texttt{OBJ}_{i}>\}_{i=1...n}$ as object IDs into the vocabulary. These ID tokens are paired with 2D & 3D object features to indicate each object, for referring to the object in the input instruction or grounding the object in model’s output.
Our ID-Feature Bonding combines each object feature with its corresponding object ID, and appends the system and question prompts at the beginning of the sequence, which are then fed into the LLM. For more details on the object IDs and the extraction of different types of features, please refer to .

Figure: Figure 2: Overview of the proposed Robin3D model structure. Bottom: Our Relation-Augmented Projecter fuses the features and position embedding from Mask3D and Uni3D to generate final 3D features. 2D features from DINO v2 are projected into the LLM space. We freeze the Mask3D, Uni3D, and DINO v2. Middle: ID-Feature Bonding enhances the connection between object IDs and object features by wrapping the features with identical IDs and the Post-Vision order. Top: We use LoRA to fine-tune the LLM on our constructed 1 million instruction data.
Refer to caption: x2.png

### 3.2 Relation-Augmented Projecter (RAP)

As shown in Fig. 2 (bottom), to obtain relation-aware 3D features while preserving the unified object-centric characteristics, our RAP considers three types of 3D features: a) the object features from Mask3D, $X_{mask3d}$, which are scene-level respresentations, containing spatial relationship, as they come across multiple cross-attention layers to exchange information. b) the position embeddings of the Mask3D, $X_{pos}$, which are directly projected from the coordinates of the corresponding object features. c) the unified object features, $X_{uni3d}$, from Uni3D. Our RAP is formulated as:

$$ $X=\text{Concat}(\text{Norm}_{L2}(X_{uni3d}),\text{Norm}_{L2}(X_{mask3d})), \quad X_{rap}=\text{MLP}(X)+\text{MLP}(X_{pos})$ (1) $$

where $\text{Norm}_{L2}$ is the L2 normalization, Concat is the concatenation alongside the channel dimension, and MLP is a multi-layer perceptron with GELU activation [Hendrycks & Gimpel, ]. The $X_{rap}$ represents our final relation-aware unified 3D features, which is augmented by the spatial relationship information from Mask3D and the position embeddings.

### 3.3 ID-Feature Bonding (IFB)

We propose IFB for better referring and grounding in our instruction-following data by improving the connection between object IDs and object features.
As shown in Fig. 2 (middle), we first use two identical ID tokens to wrap the object features. Adhering to the causal attention nature of the LLM, this approach links ID information to the object features via the first ID token, and links object information to its ID via the second ID token.
Secondly, we propose a post-vision order, which places the vision tokens at the end of the input sequence, closer to the answer tokens generated by the model. This approach mitigates attention deviation from the answer tokens to the ID-Feature pairs, a problem caused by their relative token distance and rotary position embeddings [Su et al., ; Ma et al., ], while reinforcing the visual information for improved answer generation.
The post-vision order is structured as:
[$<$System tokens$>$, $<$Instruction tokens$>$, $<$Vision tokens$>$, $<$Answer tokens$>$], where $<$Vision tokens$>$ comprises the ID tokens and object feature tokens.

## 4 Robust Instruction Generation (RIG)

### 4.1 Adversarial Data Generation

The Adversarial data is designed to challenge the model’s discriminative capabilities by introducing adversarial or negative samples, ranging from the object-level to the scene-level. It features both category-based identification tasks and expression-based reasoning challenges. As shown in Fig. 3, we ultimately formulate four novel tasks: Hybrid Object Probing Evaluation (Sec. 4.1.1), Hybrid Referring Object Classification (Sec. 4.1.2), Partial Factual 3D Visual Grounding (Sec. 4.1.3), and Faithful 3D Question Answering (Sec. 4.1.4).
Details of each type are as follows:

Figure: Figure 3: The visualization of examples of adversarial / negative data. For better visualization, we associate each object ID with the same color as its bounding box. The black solid circles with numbers are solely for visualization purposes and are not included in the actual data.
Refer to caption: x3.png

#### 4.1.1 Hybrid Object Probing Evaluation (HOPE) – Fig. 3 (upper left)

To construct a scene-level category-based task, we introduce HOPE, which is inspired by the POPE benchmark [Li et al., ] in 2D domain. POPE evaluates the tendency of 2D MLLMs to hallucinate by asking yes/no questions about the presence of one specific object at a time.
Building on this, HOPE further incorporates such hallucination challenges into the training stage in the 3D domain, aiming to train our model to be more discriminative. Additionally, HOPE presents a hybrid scenario, introducing greater complexity to further advance the decoupling of memorized positive vision and language pairs.
Specifically, given a 3D scene, we ask the model to determine the presence of various randomly specified objects. The objects may or may not be present in the scene, and each existing object might have one or more instances. The model is required to answer “No” when the object is not present in the scene, and answer “Yes” with the object ID of each instance of the object when it exists.
As shown in Fig. 3 (upper left), the question combines multiple objects, separated by semicolons (“;”), and the answer combines responses for each object, also separated by semicolons. This structure creates a challenging setting that involves hybrid recognition of both positive and negative object presence, combined with multi-instance object localization.

#### 4.1.2 Hybrid Referring Object Classification (HROC) – Fig. 3 (upper right)

Referring Object Classification [You et al., ] evaluates a model’s ability to understand a referred region in 2D domain,
focusing on a classification problem by “Region-in Text-out” format.
Our HROC dataset extends this task into the training data for 3D domain to create an object-level category-based task,
by incorporating adversarial and hybrid challenges.
Specifically, in a 3D scene, we randomly create hybrid positive and negative ID-Category pairs to form our questions, as illustrated in Fig. 3 (upper right). A positive pair consists of a valid object ID and the ground truth category. The bounding box of the corresponding object ID must overlap with one ground truth bounding box, and the category of the overlapping object is defined as the ground truth category. A negative pair includes a valid object ID and a randomly selected category that is present in the scene but not the ground truth category to serve as an adversarial challenge. The model is required to answer “Yes” for positive pairs and “No” with the correct category for negative pairs. The pairs and corresponding answers are separated by semicolons (“;”).

#### 4.1.3 Partial Factual 3D Visual Grounding (PF-3DVG) – Fig. 3 (lower left)

Our PF-3DVG introduces a scene-level expression-based task, featuring three types of data in 3DVG: unfactual data, partially factual data, and factual data. For unfactual data, given a 3D scene, we randomly select a reference from Sr3D+ [Achlioptas et al., ] where the indicated object does not exist in the scene. The model is required to answer “No” when prompted with the question, as shown in the first example of Fig. 3 (lower left).
For partial factual data, given a reference from Sr3D+ and its corresponding 3D scene, we randomly switch the described spatial relationship with a different one based on the predefined template of Sr3D+. For example, as shown in the second example of Fig. 3 (lower left), we change the original reference “the pillow lying on the couch” to “the pillow under the couch”. In this case, it is still possible for human annotators to ground the target based on this partial factual information, but this introduces an adversarial challenge since the information is not completely accurate.
Therefore, we require the model to retify the information and answer “It is ‘lying on’” while providing the grounding result (object ID). Notably, we only use references whose target object has no distractors sharing the same category, ensuring that the partial factual information is still informative enough for grounding the target and does not lead to ambiguity.
For factual data, we randomly augment the spatial relationship with its synonym to improve diversity. For example, the synonym of “below” can be “under”, “beneath”, or “underneath”.

#### 4.1.4 Faithful 3D Question Answering (3DFQA) – Fig. 3 (lower right)

The original 3D Question Answering (QA) task [Azuma et al., ] includes only positive samples, which can potentially lead to the model memorizing fixed combinations of 3D scenes and QA pairs.
To address this, we propose Faithful 3D Question Answering, a scene-level expression-based task which incorporates both negative and positive examples with an additional grounding requirement.
To construct negative samples, we first sample a QA pair and collect the related objects that are mentioned in the question or the target objects of the answer from Azuma et al. []. Then, we randomly select a 3D scene that lacks those related objects. A new instruction is added to the question: “If you can, answer the question… and provide all the IDs…” as illustrated in Fig. 3 (lower right). In this case, the model must faithfully answer “No” based on the absence of related objects in the 3D scene and must not provide any object IDs, demonstrating its reliance on the scene for making decisions.
For positive samples, directly taken from Azuma et al. [], the model must answer the question while faithfully grounding its “evidence” for the answer, i.e., providing the IDs of the related objects.
Therefore, the model trained on our 3DFQA dataset
is forced to generalize beyond memorization,
learning to respond faithfully to both positive and negative samples.

### 4.2 Diverse Data Generation

The Diverse data aim to enhances the model’s generalization by first incorporating multiple different types of instruction-following data and then increasing the linguistic diversity of the instructions.

We first collect large scale data from different tasks outside the benchmark dataset.
Specifically, given a 3D scene, we collect question-answering pairs from the following tasks: 1) Category Question-Answering task from Huang et al. [], where the model is asked to answer the category of a specified object. 2) Nr3D Captioning task from Huang et al. [], where the model is asked to caption the spatial relationship of a specified object to its neighboor. The ground truth is constructed from Nr3D [Achlioptas et al., ] dataset. 3) Appearance Captioning task from Chen et al. [], where the model is asked to captioning the physical attributes or visual characteristics of a specified object. The ground truth is generated by CogVLM [Wang et al., ]. 4) Region Captioning task from Chen et al. [], where the model is asked to caption the region encircling a specified object. The ground truth is organized by ChatGPT []. 5) End to end 3D Visual Grounding from Nr3D dataset [Achlioptas et al., ], where the model is not provided ground truth candidates, different from the original setting in Nr3D. 6) End to end 3D Visual Grounding from Sr3D+ dataset [Achlioptas et al., ], where the model is also not provided ground truth candidates, different from the original setting in Sr3D+.

Figure: Figure 4: (Upper) Pipeline to generate our Diverse Instruction data by the in-context learning of ChatGPT. (Lower) The one-shot examples for ChatGPT to rephrase the instruction-following data.
Refer to caption: x4.png

To diversify the wording style,
we develop a scalable pipeline by harnessing ChatGPT’s [] in-context learning ability to rephrase the above data. This is achieved through a combination of one-shot examples and structured prompt engineering, as shown in Fig. 4(upper).
Formally speaking, given a collected instruction-following dataset $\mathcal{D}_{\mathit{task}}$, where $\mathit{task}\in\{\text{ScanRefer},\text{Multi3DRefer},\text{Nr3D},\text{Sr3D+
},\text{Nr3D\ Captioning},\text{ScanQA},\text{SQA3D},\text{PF-3DVG},\text{3
DFQA}\}$, we construct a system prompt, $\mathcal{P}_{\mathit{system}}$, to indicate the rephrase requirement and structured output format to ChatGPT, a one-shot example prompt, $\mathcal{P}_{\mathit{eg}}$, to show a rephrased example and output format for ChatGPT to better understand the requirement, and randomly assign a temperature $\mathcal{T}$ from $[1.1,1.2,1.3]$ for ChatGPT to increase the randomness of the output diversity. Our rephrased output, $\mathcal{D}_{\mathit{rephrase}}$, is generated by $\mathcal{D}_{\mathit{rephrase}}=\mathcal{M}(\mathcal{P}_{\mathit{system}},
\mathcal{P}_{\mathit{eg}},\mathcal{D}_{\mathit{task}},\mathcal{T})$, where $\mathcal{M}$ is the GPT-4o version of ChatGPT. We provide the details of $\mathcal{P}_{\mathit{system}}$ and $\mathcal{P}_{\mathit{eg}}$ in the Fig. 4(upper) for data of ScanRefer as an example. With our “sentence=” and “rephrase=” structured prompt, GPT-4o can easily follow the requirement and we can conveniently collect the output by detecing the “rephrase=” keywords.
In Fig. 4(lower), we provide details regarding the one-shot example for each task. Since Nr3D Captioning is constructed from Nr3D, PF-3DVG is from Sr3D+, and 3DFQA is from ScanQA, we do not provide additional examples for them.

### 4.3 Data Summary

In summary, our Robust Instruction Generation engine produces two types of data:
1) Adversarial Instruction data: a total of 344K samples, consisting of mixed positive and negative pairs, formulated into four new tasks.
2) Diverse Instruction data: a total of 508K samples, covering multiple tasks and various language styles, diversified through in-context learning from ChatGPT.

## 5 Experiments

### 5.1 Benchmarks and Metrics

We provide quantitative results on five widely-used 3D multimodal learning benchmarks: ScanRefer [Chen et al., ] for 3D Visual Grounding, Multi3DRefer [Zhang et al., ] for General 3D Visual Grounding including zero, single and multiple target objects, Scan2Cap [Chen et al., ] for 3D Dense Captioning, ScanQA [Azuma et al., ] for 3D Question Answering, and SQA3D [Ma et al., ] for 3D Situated Question Answering. The vision data are all based on the ScanNet dataset [Dai et al., ], which contains real world 3D point clouds across 1,513 indoor scenes with detailed object annotations. All these benchmarks follow the same data split as ScanNet.

We follow the standard evaluation metrics widely adopted in the respective benchmarks. For ScanRefer, we calculate accuracy at Intersection over Union (IoU) thresholds of 0.25 and 0.5 (Acc@0.25, Acc@0.5). For Multi3DRefer, we use the F1 score with IoU thresholds of 0.25 and 0.5 to measure performance. In Scan2Cap, we apply the CIDEr@0.5 and BLEU-4@0.5 (C@0.5, B-4@0.5) metrics, combining standard captioning metrics with the IoU metric. For ScanQA, the METEOR and ROUGE metrics, denoted as M and R, are employed. Lastly, SQA3D is assessed with exact match accuracy (EM) and its extended form, EM-R, as suggested by LEO [Huang et al., ].

### 5.2 Implementation Details

We extract 150 object features from each 3D scene, along with the corresponding position embeddings and 3D masks generated by Mask3D. The 2D Projector, as shown in Fig. 2, is a two-layer MLP. We use the Vicuna-7B-v1.5 model [Chiang et al., ] as our LLM and fine-tune it using LoRA [Hu et al., ] (with a rank of 16) by Cross Entropy loss.
The global learning rate is formulated as [batch size $\times$ base learning rate $\times$ number of GPUs] and is set to 0.00064, with a cosine annealing schedule.
For our results in Tab. 1, we first train for 2 epochs on the RIG-generated data, and then train for 2 epochs on the benchmark training sets in the second stage.
We train for 1 epoch for each stage to efficiently conduct ablation studies of RIG-generated data.
For ablation on RAP and IFB, we train for 1 epoch on the benchmark training sets to avoid potential compound effects.

**Table 1: Quantitative comparison. “Task-Specific Training” denotes models trained on a specific task, while “Joint Training” denotes models trained jointly on multiple tasks. Entries in gray denote using ground truth question-relative objects annotations. The best and second best results in a fair comparison are highlighted in bold and underline, respectively.**
| Model | ScanRefer | Multi3DRefer | Scan2Cap | ScanQA(val) | SQA3D(val) | SQA3D(test) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc@0.25 | Acc@0.5 | F1@0.25 | F1@0.5 | B-4@0.5 | C@0.5 | M | R | EM | EM-R | EM | EM-R |  |
| Task-Specific Training |  |  |  |  |  |  |  |  |  |  |  |  |
| ScanRefer | 37.3 | 24.3 | - | - | - | - | - | - | - | - | - | - |
| EDA | 53.8 | 41.7 | - | - | - | - | - | - | - | - | - | - |
| Concretenet | 50.6 | 46.5 | - | - | - | - | - | - | - | - | - | - |
| M3DRef-CLIP | 51.9 | 44.7 | 42.8 | 38.4 | - | - | - | - | - | - | - | - |
| Scan2Cap | - | - | - | - | 23.3 | 39.1 | - | - | - | - | - | - |
| Vote2Cap-DETR++ | - | - | - | - | 37.1 | 67.6 | - | - | - | - | - | - |
| ScanQA | - | - | - | - | - | - | 13.1 | 33.3 | - | - | - | - |
| SQA3D | - | - | - | - | - | - | - | - | - | - | 46.6 | - |
| Joint Training |  |  |  |  |  |  |  |  |  |  |  |  |
| D3Net | - | 37.9 | - | 32.2 | 35.7 | 62.6 | - | - | - | - | - | - |
| 3DJCG | 49.6 | 37.3 | - | 26.6 | 31.0 | 49.5 | - | - | - | - | - | - |
| 3D-VLP | 51.4 | 39.5 | - | - | 32.3 | 54.9 | - | - | - | - | - | - |
| 3D-VisTA | 50.6 | 45.8 | - | - | 34.0 | 66.9 | 13.9 | 35.7 | - | - | 48.5 | - |
| PQ3D | - | 51.2 | - | 50.1 | 36.0 | 80.3 | - | - | - | - | 47.1 | - |
| 3DLLM | 30.3 | - | - | - | - | - | 14.5 | 35.7 | - | - | - | - |
| Oryx | - | - | - | - | - | - | 15.0 | 37.3 | - | - | - | - |
| LL3DA | - | - | - | - | 36.8 | 65.2 | 15.9 | 37.3 | - | - | - | - |
| LEO | - | - | - | - | 38.2 | 72.4 | 20.0 | 49.2 | - | - | 50.0 | 52.4 |
| Scene-LLM | - | - | - | - | - | - | 16.6 | 40.0 | - | - | 54.2 | - |
| Chat-3D v2 | 35.9 | 30.4 | - | - | 15.5 | 28.1 | 16.1 | 40.1 | - | - | - | - |
| Grounded-3DLLM | 47.9 | 44.1 | 45.2 | 40.6 | 35.5 | 70.6 | 15.2 | 37.1 | - | - | - | - |
| Chat-Scene | 55.5 | 50.2 | 57.1 | 52.4 | 36.3 | 77.1 | 18.0 | 41.6 | 53.2 | 56.1 | 54.6 | 57.5 |
| Robin3D (Ours) | 60.8 | 55.1 | 64.9 | 59.7 | 38.4 | 87.2 | 19.2 | 44.0 | 56.0 | 58.6 | 56.9 | 59.8 |

### 5.3 Quantitative Results

We classify current methods into two categories: Task-Specific Training and Joint Training. Task-Specific Training refers to models only trained for a specific task, while Joint Training means training on multiple tasks jointly. Our Robin3D does not conduct task-specific fine-tuning.

- •
Task-Specific Training: As shown in Table 1, models like EDA and M3DRef-CLIP perform well on their respective tasks due to customized model design for the task. However, they lack the ability to generalize to other tasks. Models like Vote2Cap-DETR++ and SQA3D encounter the similar issue. Therefore, they are not suitable to serve as general-purpose 3D AI agents.
- •
Joint Training: Benefiting from sharing the knowledge across multiple tasks, models like 3D-VisTA and PQ3D show decent performance across multiple tasks, but their dependence on task-specific heads restricts their generalizability. Models like LEO and Chat-Scene show promising results by leveraging LLMs, but their sole training on positive pairs and template-based instructions leads to suboptimal generalization.
- •
Our Robin3D: Due to the robust instruction data generated by RIG, Robin3D significantly outperforms previous models across all the benchmarks. Specifically, Robin3D brings a 6.9% improvement on Scan2Cap CIDEr@0.5 and a 5.3% improvement on ScanRefer Acc@0.25. Notably, on the evaluation of Multi3DRefer, which contains zero-target cases that are challenging for models to be discriminative and learn to say “No”, our Robin3D achieves a 7.8% improvement in F1@0.25 and a 7.3% improvement in F1@0.5.

### 5.4 Ablation Study

**Table 2: Ablation study on Robust Instruction Generation. Benchmark denotes training on the original training set of the benchmarks. Adversarial denotes adding the Adversarial Instruction data to the training set. Diverse denotes adding the Diverse Instruction data to the training set.**
| Data | ScanRefer | Multi3DRefer | Scan2Cap | ScanQA(val) | SQA3D(val) |
| --- | --- | --- | --- | --- | --- |
| Acc@0.5 | F1@0.5 | C@0.5 | M | EM |  |
| Benchmark | 45.3 | 50.2 | 73.6 | 17.7 | 48.9 |
| + Adversarial | 49.0 | 55.1 | 82.5 | 18.2 | 50.7 |
| + Diverse | 50.6 | 53.1 | 80.5 | 18.0 | 50.9 |
| + Adversarial & Diverse | 51.8 | 56.9 | 84.1 | 18.4 | 52.6 |

**Table 3: Ablation study on our proposed modules: Relation-Augmented Projecter (RAP) and ID-Feature Bonding (IFB).**
| Model | ScanRefer | Multi3DRefer | Scan2Cap | ScanQA(val) | SQA3D(val) |
| --- | --- | --- | --- | --- | --- |
| Acc@0.5 | F1@0.5 | C@0.5 | M | EM |  |
| Baseline | 41.0 | 45.3 | 69.2 | 17.4 | 48.2 |
| + RAP | 42.5 | 46.3 | 72.0 | 17.6 | 48.6 |
| + RAP & IFB | 45.3 | 50.2 | 73.6 | 17.7 | 48.9 |

We perform ablation studies from the perspective of training data and model structure, respectively. We first evaluate the effectiveness of RIG-generated data by progressively adding the Adversarial Instruction data and the Diverse Instruction data to the training set. We then investigate the contribution of RAP and IFB by comparing models with and without these components.

Robust Instruction Generation (RIG): As shown in Tab. 2, by adding the Adversarial Instruction data, we observe a consistent improvement across all benchmarks. Specifically, performance on ScanRefer and Multi3DRefer increases by 3.7% and 4.9%, respectively. It is worth noting that the performance on Scan2Cap improves by 8.9%, even though there is not any object captioning data in the Adversarial Instruction data, which highlights its contribution on enhancing the understanding towards each object by challenging the model with mixed positive and negative samples.
Additionally, by adding the Diverse Instruction data, we also provide comprehensive improvements. Specifically, descriptions from the original ScanRefer are annotated by human following a fixed instruction template or expression style, which limits the language diversity. In contrast, the Diverse Instruction data contains various language styles and task formats, which helps the model generalize better, resulting in a 5.3% improvement on ScanRefer. Finally, by combining both two types of data, we achieve a further improvement, demonstrating the effectiveness of RIG-generated data.

Relation-Augmented Projecter (RAP) and ID-Feature Bonding (IFB): As shown in Tab. 3, the integration of RAP leads to steady improvements across all benchmarks. Notably, the performance on Visual Grounding tasks, including ScanRefer and Multi3DRefer, shows significant gains due to RAP’s enhanced spatial comprehension. When IFB is incorporated, further improvements are observed, emphasizing the importance of refining the model’s object referring and grounding capabilities by reinforcing the connection between IDs and features.

### 5.5 Qualitative Results

Figure: Figure 5: Visualization of Robin3D’s responses on all the five benchmarks.
Refer to caption: x5.png

We provide visualization of Robin3D’s responses on all the benchmarks in Fig. 5 with the prompts of each task. These results demonstrates the generalization ability of Robin3D on various tasks.

## 6 Conclusion

To build a general-purpose AI agent in the 3D real world, we identify the problem of a lack of robust instruction training data in current 3DLLMs. To tackle this challenge, we introduce Robin3D, a powerful 3DLLM trained on large-scale instruction-following data generated by our novel data engine, Robust Instruction Generation (RIG) engine. We generate and collect 1 million instruction data, including benchmark data, adversarial data, and diverse data. To better handle these complex instructions, Robin3D incorporates a Relation-Augmented Projector to enhance the understanding of spatial relationships among objects, and ID-Feature Bonding for better object referring and grounding. Finally, Robin3D achieves state-of-the-art performance across five widely-used 3D multimodal learning benchmarks, making significant progress towards Spatial Intelligence.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: - Table 1 does not clarify what kinds of LLM each method uses but it is worth doing that since multimodal LLM performance usually depends on LLM performance. \
- What is the baseline in Table 3? I couldn't find out the network architecture of the baseline.
soundness: 3
strengths: • The paper introduces a large-scale 3D scene-instruction dataset that includes diverse instruction types, integrating varied instruction styles, existing benchmark instructions, and challenging adversarial instructions, enhancing the model’s robustness and generalization.\
• It proposes novel architectures that effectively leverage both 2D and 3D object-centric features, enabling richer spatial understanding and stronger object-grounding capabilities in complex 3D environments.
summary: This paper introduces Robin3D, a 3D large language model trained to follow instructions in 3D environments using the Robust Instruction Generation (RIG) engine, which creates a one-million-sample dataset. RIG generates Adversarial and Diverse instruction data to improve Robin3D’s discriminative power and generalization. Robin3D employs a Relation-Augmented Projector for spatial understanding and IDFeature Bonding for object grounding, achieving notable improvements over previous models, including a 7.8% gain in grounding and 6.9% in captioning without task-specific fine-tuning.
While Robin3D performs impressively across multiple ScanNet benchmarks, as noted in the weaknesses, some concerns remain regarding its network architecture and experiments.
weaknesses: • Relies on off-the-shelf 3D instance segmentation models trained on ScanNet with closed-set categories. I recommend the authors to consider Segment3D [1] for open-vocab, class-agnostic segmentation.\
• Cropping instance-level point clouds and applying object-level 3D point cloud CLIP (Uni3D) can limit the receptive fields and be computationally heavy. I recommend the authors to try scene-level CLIP (OpenScene [2], RegionPLC[3]) and then cropping the output features.\
• Table 1 reports only traditional NLP metrics (e.g., BLEU, CIDEr, METEOR, Rouge). I recommend the authors include LLM-based evaluation (e.g., GPT or Mistral) for better alignment with human assessment.\
• The experiments are limited to the ScanNet dataset. I recommend that the authors expand to other datasets (e.g., SceneVerse [4]) for broader evaluation.

[1] Huang et al., "Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels", ECCV, 2024.\
[2] Peng et al., "OpenScene: 3D Scene Understanding with Open Vocabularies", CVPR, 2023.\
[3] Yang et al., "RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding", CVPR, 2024.\
[4] Jia et al., "SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding", ECCV, 2024.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The motivation in the paper is somewhat mixed. Although it emphasizes pre-training with adversarial samples, it also highlights improvements through the Relation-Augmented Projector (RAP) and ID-Feature Bonding (IFB), which may seem like an attempt to pad contributions.

2. The ablation study shows that RAP and IFB contribute less to 3D QA (with low improvements in Table 3's ScanQA and SQA3D) but significantly help 3D grounding. Can the authors explain why?

3. The paper lacks details on the prompts used for Adversarial Data Generation and the data creation process. Is the input for adversarial samples only the ground truth prompt?

4. The ablation for Adversarial Data is insufficient, making it unclear whether the performance improvement is due to the increase in data volume or specifically from the adversarial samples.

5. The authors should compare methods like VLM-Grounder[1] and Coarse Correspondences[2] using video as a modality.

6. Should the authors consider extending their approach to non-ScanNet scenes?

7. The pre-training work should provide training configurations and training time.

8. Can the proposed RIG be extended to the point level to enhance point-level LLM performance, such as with PointLLM [3] or GPT-4Point [4]? Additionally, could it be generalized to outdoor 3D LLMs like DriveLM [5] or LiDAR-LLM [6]? It would be beneficial for the authors to discuss this in the paper.

[1] A VLM Agent for Zero-Shot 3D Visual Grounding

[2] Coarse Correspondence Elicit 3D Spacetime Understanding in Multimodal Language Model

[3] PointLLM: Empowering Large Language Models to Understand Point Clouds

[4] GPT4Point: A Unified Framework for Point-Language Understanding and Generation

[5] DriveLM: Driving with Graph Visual Question Answering

[6] LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding
soundness: 2
strengths: 1. Reasonable motivation
    - Expands existing ScanNet 3D text annotations through the data engine.
2. Strong experimental results
   - Demonstrates excellent performance.
3. Clear and complete paper writing.
summary: The paper introduces Robin3D, a powerful 3D Large Language Model (3DLLM) trained using large-scale instruction-following data generated by an innovative Robust Instruction Generation (RIG) engine to address the lack of robustness and diversity in current 3DLLMs' training data.

Besides, Robin3D incorporates two important modules: Relation-Augmented Projector (RAP) and ID-Feature Bonding (IFB). RAP enhances the model's understanding of spatial relationships between objects, while IFB strengthens the connection between object IDs and features, improving the model's referring and grounding capabilities, enabling it to better handle complex instructions.
weaknesses: I have some questions about this paper that need further discussion. Please see them below.

If the authors can address my concerns, I am willing to raise my score.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please address the questions raised in the weakness section.
soundness: 3
strengths: 1. The paper targets the challening problem of 3D LLM for ground task as well as caption task.
2. To address the problem, the paper presents a robust instruction generation engine and 1M instruction-following data has been presented.
3. The paper obtains promising experimental results on five 3D multimoal learning benchmarks.
summary: The paper targets the interesting problem of instruction tuning on 3D LLMs. As there lacks sufficient dataset, the paper introduces a new 1M instruction-tuning datset, which contains 344K adversarial samples, 508K diverse samples as well as 165K benchmark training set samples. Based on the dataset, the proposed algorithm, called Robin3D, obtains promising results in the ground task as well as caption task.
weaknesses: 1. Will the 1M 3D instruction dataset be release to the public? The main contribution of the paper lies on the datasets, thus whether the dataset will be released to public is important to evaluate the contribution of the paper.
2. The dataset seems to be designed specifially for the 3D indoor environment. How about the generation ability of the dataset and the model used for the outdoor environment, like the 3D street?
3. Is it possible to provide an ablation study on different of training examples? It would be better to know the model performance with different number of training data.
4. The model is based on Vicuna-7B-v1.5 backbone. How about the performance if other LLM models are utilized? Besides, if larger LLM model is utilized, is a larger training dataset can further boost the performance?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. This paper employs the same detectors and 2D&3D encoders as chat-scene (Mask3D, Uni3D, DINO-v2). What are the significant innovations of this model compared to chat-scene?

2. In Table 1, what do the grey sections referring to "ground truth question-relative objects annotations" specifically indicate? Is the explicit positional information P introduced by the dedicated detector Mask3D on the ScanQA test set considered as "ground truth question-relative objects annotations"?

3. Results of Baseline(+ RAP & IFB) in Table3 are the same as the benchmark results in Table 2. In the "ABLATION STUDY" section, it seems there might be a confusion regarding the order of incorporating modules and datasets. Benchmark(+ Adversarial & Diverse) should include RAP&IFB and encompass all datasets. Why are the results of the ablation study (+ Adversarial & Diverse) inconsistent with the results in Table 1?
soundness: 3
strengths: 1. The RIG engine's capability to generate adversarial and diverse instruction data significantly enhances the robustness and generalizability of 3DLLMs. The innovative proposal of adversarial data may help mitigate the hallucination tendencies of large models. The collection of diverse instructions, expanded by GPT to enrich the diversity of expressions, may alleviate the issue of rigid model outputs.
2. The integration of RAP and IFB modules improves the model's spatial understanding and object grounding capabilities.
3. Robin3D achieves superior performance across multiple benchmarks, showcasing its effectiveness and versatility.
4. The models are trained on the whole task data, rather than on individual tasks.
summary: This paper introduce a 3DLLM Robin3D trained on their proposed dataset generated by RIG, a pipeline to acquire diverse and discriminative data. Robin3D incorporates two key modules, the Relation-Augmented Projector (RAP) and ID-Feature Bonding (IFB), to enhance spatial understanding and object grounding. The model demonstrates state-of-the-art performance across five 3D multimodal learning benchmarks without task-specific fine-tuning.
weaknesses: 1. The module's innovativeness is found to be lacking: RAP utilizes linear layers to separately connect the 3D features from the scene, individual object 3D features, and positional information features, followed by concatenation. A possible baseline (chat-scene) employs the exact same encoders, using linear layers to connect 3D features and positional features, and then concatenating individual object 3D features. The only modification made is the interchange of inputs to the linear layers. Similarly, IFB introduces an ending ID token to signal the end of an object presentation, followed by a rearrangement of vision tokens and prompt tokens. This method of simply altering the prompt tokens is not particularly innovative.

2. Are the results state-of-the-art (SOTA): The experimental results compared against the Chat-Scene model have been open-sourced and are being continuously updated. Prior to the submission deadline for this conference, the accuracy on the ScanRefer dataset had already surpassed 61% and 55%, outperforming the method proposed in this paper. This paper should have utilized the most recent projects and results as benchmarks; otherwise, the effectiveness of the proposed method cannot be ascertained.

3. Model generality: Mainstream approaches about 3DLLMs typically employ a single, well-trained model to evaluate performance across multiple tasks. The joint training without task-specific fine-tuning method described in the paper does not represent a contribution unique to this work.

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: Please refer to the weakness section.
soundness: 3
strengths: 1. This paper constructs a large instruction-following fine-tuning dataset containing adversarial and diverse samples. 
2. The zero-shot performance improvement of the trained Robin3D appears evident across various benchmarks and the ablation experiments clearly demonstrate the gains of different designs in the paper.
3. The writing of the article is fluent and easy to understand.
summary: This paper first constructs 1 million instruction-following data, including adversarial samples and diversified samples to bridge the drawbacks of existing 3D MLLM instruction following fine-tuning datasets. To better handle the proposed complex instructions, this paper first incorporates Mask3D and Relation-Augmented Projector to enhance spatial understanding, and then improve the object referring and grounding ability through ID-Feature Bonding. The trained model Robin3D shows superior performance across five widely used 3D multimodal learning benchmarks.
weaknesses: 1. The related work section lacks clarity on the novelty and advantages of the RAP and IFB modules in comparison to existing studies. 
(1) Explain how object IDs are linked to object features in previous research and discuss the benefits of wrapping these features with identical ID tokens before and after them.
(2) Describe how earlier studies extract and utilize 3D and 2D features, and highlight the advantages of introducing Mask3D information using RAP.

2. How will the relative proportions of diverse and adversarial samples generated with RIG affect the performance of Robin3D? 
Please conduct ablation studies to examine and analyze how datasets with varying proportions of adversarial and diverse samples influence Robin3D's performance across different tasks.


3. If the dataset constructed in this paper is used to fine-tune existing task-specific or joint trained models, will it provide consistent performance gains? 
The authors could consider selecting 1 or 2 task-specific and jointly trained models, respectively, and tuning them on the proposed instruction-following tuning dataset to further demonstrate the contribution of this dataset to the community.