3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination

## Abstract

Abstract The integration of language and 3D perception is crucial for developing embodied agents and robots that comprehend and interact with the physical world. While large language models (LLMs) have demonstrated impressive language understanding and generation capabilities, their adaptation to 3D environments (3D-LLMs) remains in its early stages. A primary challenge is the absence of large-scale datasets that provide dense grounding between language and 3D scenes. In this paper, we introduce 3D-GRAND, a pioneering large-scale dataset comprising 40,087 household scenes paired with 6.2 million densely-grounded scene-language instructions. Our results show that instruction tuning with 3D-GRAND significantly enhances grounding capabilities and reduces hallucinations in 3D-LLMs. As part of our contributions, we propose a comprehensive benchmark 3D-POPE to systematically evaluate hallucination in 3D-LLMs, enabling fair comparisons among future models. Our experiments highlight a scaling effect between dataset size and 3D-LLM performance, emphasizing the critical role of large-scale 3D-text datasets in advancing embodied AI research. Notably, our results demonstrate early signals for effective sim-to-real transfer, indicating that models trained on large synthetic data can perform well on real-world 3D scans. Through 3D-GRAND and 3D-POPE, we aim to equip the embodied AI community with essential resources and insights, setting the stage for more reliable and better-grounded 3D-LLMs.

## 1 Introduction

Embodied Artificial Intelligence (EAI) represents a frontier in robotics and machine learning. In EAI, the integration of perception, language, and action within physical spaces is crucial for developing intelligent systems capable of meaningfully navigating and interacting with their environments. Central to this vision is the concept of grounding language in the physical world [, ]. Grounding connects abstract linguistic constructs to concrete objects in three-dimensional space, thereby enabling robots and intelligent agents to effectively understand and manipulate their surroundings.

Recent advancements in Large Language Models (LLMs) have greatly benefited Embodied Artificial Intelligence (EAI). LLMs demonstrate exceptional capabilities in understanding language instructions [, ], perceiving the environment [, , , , ], and planning detailed actions [, ].
The primary inputs to LLMs, other than pure language, have been the combination of language and 2D images, categorizing these models as 2D-LLMs. The significant advancements in 2D-LLMs can be largely attributed to their training on extensive vision-language datasets. These datasets [, ], comprising billions of image and text pairs, have been instrumental in enhancing the models’ understanding of visual content and its contextual relevance to textual information. These large datasets have provided the foundational data necessary for training models that excel at integrating visual perception with language processing. Despite some progress in equipping LLMs to understand 3D scenes (3D-LLMs) [, , , , , , ], these models remain in their early stages due to the scarcity of 3D scene and text pairs. In this work, we introduce 3D-GRAND, a pioneering million-scale dataset designed for densely-grounded 3D Instruction Tuning.

Recently, SceneVerse [] concurrently introduced a large-scale 3D vision-language dataset. However, a significant limitation of this dataset is the absence of object grounding in language, which is crucial for enhancing model usability in robotics tasks and reducing hallucination. Research on 2D-LLMs indicates that grounding language to 2D contexts notably mitigates hallucination in language models [, , , , , ], thereby enhancing the reliability and interpretability of generated responses. While 2D grounding has been extensively explored, extending these principles to 3D environments is still underdeveloped. This situation raises two critical questions: (1) Is there any hallucination in 3D-LLMs and if so, how severe is it? (2) Can densely-grounded data mitigate hallucination for 3D-LLMs? These questions underscore a critical need within the research community for the development of an evaluation benchmark specifically designed for 3D-LLMs and the construction of a large-scale, 3D-grounded dataset.

To quantify hallucination in 3D LLMs, this work introduces 3D-POPE (3D Polling-based Object Probing Evaluation).
3D-POPE provides a comprehensive and standardized protocol for evaluating hallucination that enables systematic assessment and facilitates fair comparisons across 3D-LLMs, enhancing our understanding of model capabilities in object hallucination. Specifically, we pose existence questions to 3D-LLMs and evaluate their responses, as illustrated in Fig 1.

To evaluate the role of densely-grounded dataset, we introduce a pioneering million-scale dataset, 3D-GRAND, for densely grounded 3D instruction tuning.
3D-GRAND includes 40,087 household scenes paired with 6.2 million scene-language instructions, featuring dense phrase-to-object grounding. We conduct rigorous human evaluations to ensure the dataset’s quality. Our results trained with 3D-GRAND highlight the dataset’s effectiveness in enhancing grounding and reducing hallucination for 3D-LLMs. We highlight the effectiveness of incorporating 3D-GRAND in Fig 1 and introduce each category of 3D-GRAND and provide examples in Fig 2.

To sum up, our contributions include:

- •
3D-GRAND, the first million-scale, densely-grounded 3D-text dataset for grounded 3D Instruction Tuning. 3D-GRAND includes 40K household scenes paired with 6.2M densely-grounded scene-language instructions.
- •
3D-POPE, a suite of benchmarks and metrics that systematically evaluate hallucination, enabling fair comparisons of future 3D-LLM models in terms of object hallucination.
- •
Quantitative research findings regarding hallucination, grounding, and scaling that provide guidance to future research: (1). training 3D-LLMs with 3D-GRAND significantly reduces hallucinations, particularly when the data is densely grounded; (2). densely grounded instruction tuning significantly enhances the grounding capabilities of 3D-LLMs; (3). scaling densely grounded data consistently improves grounding accuracy and reduces hallucination; and (4). models can successfully transfer from sim-to-real, providing an early signal for a low-cost and sustainable future of scaling synthetic 3D data to help on real tasks.

**Table 1: Comparison of 3D-GRAND with existing 3D scene datasets with language annotations. 3D-GRAND is the largest language-grounded dataset.**
| Dataset | Which part is grounded? | Densely Grounded? | Language source | # 3D Scenes | # Language pairs |
| --- | --- | --- | --- | --- | --- |
| ReferIt3D [] | obj-refer | ✗ | Human,Template | 0.7K | 125K |
| ScanRefer [] | obj-refer | ✗ | Human | 0.7K | 51K |
| Scan2Cap [] | obj-refer | ✗ | Human | 0.7K | 51K |
| ScanEnts3D [] | obj-refer | ✓ | Human | 0.7K | 84K |
| PhraseRefer [] | obj-refer | ✓ | Human | 0.7K | 170K |
| ScanQA [] | answer | ✗ | Human | 0.7K | 41K |
| SQA3D [] | question | ✗ | Human | 0.65K | 33.4K |
| 3DVQA [] | ✗ | ✗ | Template | 0.7K | 500K |
| CLEVR3D [] | ✗ | ✗ | Template | 8.7K | 171K |
| 3DMV-VQA [] | ✗ | ✗ | Template | 4.1K | 55K |
| EmbodiedScan [] | ✗ | ✗ | Template | 3.4K | 970K |
| 3DMIT [] | ✗ | ✗ | LLM | 0.7K | 75K |
| M3DBench [] | obj-refer, question | ✗ | LLM | 0.7K | 327K |
| 3D-DenseOG [] | scene | ✓ | Human | 0.7K | 51K |
| 3D-LLM [] | obj-refer | ✗ | LLM | 0.9K | 200K |
| LL3DA [] | question, answer | question | Template,LLM | 0.9K | 200K |
| Chat3D-v2 [] | scene | ✓ | Human,LLM | 0.7K | 0.7K |
| 3D-VisTA [] | question | ✗ | Template,LLM | 3K | 278K |
| LEO [] | question | ✗ | LLM | 3K | 579K |
| SceneVerse [] | obj-refer | ✗ | Template,LLM | 62K | 2.5M |
| 3D-GRAND | scene, obj-refer, question, answer | ✓ | Template,LLM | 40K | 6.2M |

## 2 Related Work

Injecting 3D into LLMs.
Recent advancements in large language models (LLMs) have inspired research into extending their capabilities to 3D environments, leading to the development of 3D-LLMs [, , , ]. Notable works in this field include 3D-LLM [], which integrates 3D point clouds and features into LLMs to enable tasks such as captioning, question answering, and navigation. LEO [] excels as an embodied multi-modal generalist agent in perception, grounding, reasoning, planning, and action in 3D environments, highlighting the potential of 3D-LLMs in understanding and interacting with the physical world. The most relevant work to our model is Chat-3Dv2 [, ], which grounds generated scene captions to objects in 3D scenes. However, Chat-3Dv2’s dataset is limited to one type of 3D-text task (scene captioning) and only includes 705 captions from a subset of ScanNet scenes. In 3D-GRAND, we expand this concept by diversifying 3D-text tasks and increasing the dataset size to a million-scale. Our results demonstrate promising data scaling effects and sim-to-real transfer, paving the way for future large-scale training of 3D-LLMs.

Object Hallucination of VLMs.
While 2D VLMs have achieved impressive performance, they are prone to hallucinating objects that do not exist in the provided images, a problem known as object hallucination [, ]. Several methods have been suggested to mitigate the object hallucination issue, such as integrating an external object detector [], applying visually grounded visual instruction tuning [, ] or reinforcement learning [, ], performing iterative refinement [], and adapting the decoding strategies []. To quantify and mitigate this issue, several benchmarks have been proposed. CHAIR (Caption Hallucination Assessment with Image Relevance) [] measures the frequency of hallucinated objects in image captions by comparing the objects mentioned to the ground truth annotations. POPE (Probing Object Hallucination Evaluation) [] assesses a VLM’s ability to identify the presence or absence of objects through yes/no probing questions. However, these studies primarily focus on 2D image-text datasets like COCO []. In contrast, object hallucination in 3D-LLMs remains largely unexplored. Our work addresses this gap by introducing 3D-POPE, a comprehensive benchmark for evaluating object hallucination in 3D-LLMs. To the best of our knowledge, this is the first object hallucination benchmark for 3D-LLMs.

Grounding datasets.
In the 2D domain, large-scale datasets with grounding information have been instrumental in advancing vision-language research. Notable examples include RefCOCO [], which provides referring expressions for objects in COCO images []. Additionally, 2D LLMs [, , , , ] have been trained with densely-grounded web-crawled image-text pairs.
In the 3D domain, there is a growing interest in creating datasets that pair 3D scenes with textual annotations [, , , ]. ScanRefer [] pioneered this effort by contributing a dataset of ScanNet [] scenes with referring expressions. Table 1 introduces the efforts made by the community. However, these datasets have limited grounding annotations and often focus on a single task, such as referring expression comprehension or visual question answering. In contrast, our proposed dataset, 3D-GRAND, stands out by providing 6.2 million densely-grounded scene-language instructions across a diverse set of 3D-text tasks and 40,087 household scenes. This enables a wide range of grounding tasks and facilitates the development of more reliable and better-grounded 3D-LLMs.

Figure: Figure 2: 3D-GRAND dataset and statistics. (Left): 3D-GRAND is a large-scale, densely-grounded 3D-text dataset with 8 different tasks. (Right): From 40K 3D scenes, 3D-GRAND annotates 6.2M 3D-text pairs.
Refer to caption: x2.png

## 3 3D-GRAND: The 3D Gr ound An ything D ataset

In this section, we introduce 3D-GRAND, a large-scale, densely-grounded 3D-text dataset designed for grounded 3D instruction tuning. We describe the data collection process, dataset statistics, and the unique features that make 3D-GRAND a valuable resource for advancing research in 3D-LLMs.

3D scene collection. The majority of 3D-text research is currently based on ScanNet scenes collected from real camera scans, which are limited in scale. However, recent advancements have led to the development of numerous synthetic data generation pipelines [, , , , , , , , , , , , ]. Given the scalability of these synthetic data generation pipelines, we explore the potential of using synthetic 3D scenes to enhance 3D-text understanding.

Synthetic data offers significant advantages, such as lower costs and greater accessibility, making it an attractive alternative. If models trained on simulated 3D-text data can effectively transfer to real-world 3D scenes, the research community stands to benefit immensely.

To this end, we curate a diverse collection of 40,087 high-quality 3D indoor scenes from the 3D-FRONT [] and Structured3D [] datasets. These datasets are chosen for their large quantities of synthetic indoor scenes with professionally designed layouts. The collection includes a variety of room types, such as living rooms, bedrooms, kitchens, office spaces, and conference rooms. We further process these 3D scenes to generate per-room 3D point clouds. Details on point cloud rendering and cleaning are provided in the Appendix.

Densely-grounded text annotation. The definition of *densely-grounded* text is that every noun phrase of object mentioned in the text should be associated with an 3D object in the 3D scene. This is illustrated in Figure 2. This is a difficult type of data to get annotations on.
Early work such as ScanEnts3D [] relies on hiring *professional* human annotators to obtain such annotations. The authors report that crowd-sourcing annotators (Amazon Mechanical Turk (AMT) []) were not able to reliably complete this task and they had to hire professional annotators (error rate AMT: 16%, professional: $<$5%). Yet our human quality check shows that LLMs (GPT-4 []) can achieve $<$8.2-5.6% densely-grounding error rate (see Appendix for detail). This finding is in accordance with recent studies [, ] reporting LLMs can be human-level annotators. The accuracy of LLM-annotation provides one motivation for considering LLMs as densely grounding annotation tool.

The second, and perhaps more critical, motivation is the scalability of annotation. While we can potentially scale up 3D scenes using synthetic data generation pipelines, annotating these scenes with human effort is both costly and time-consuming, especially for complex tasks like densely grounding annotation. To put the cost of money and time in perspective, for the data we annotated in this paper, we estimate that obtaining the same annotations with human annotator would cost at least $539,000 and require 5.76 years (no eat, no sleep) worth of work from a professional annotator (earning minimum wage of $10.67 per hour). In contrast, using LLMs (GPT4 []), we achieve the same results for $3,030 within 2 days, representing a 178x reduction in cost and a 1051x reduction in time. At the time of writing, the cost and time further decreases by 50% to $1,500 and 1 day, with the introduction of GPT-4o [].

As previously discussed, using humans to annotate 3D scenes can be an exhaustive process. Meanwhile, 2D-LLMs demonstrate remarkable capabilities in understanding visual inputs and generating language, making them well-suited for creating high-quality, grounded language annotations. However, due to the hallucination issues and data issues in 2D-LLMs, aggregating information across images, even those originating from the same scene, is not feasible yet.

In contrast, Large Language Models (LLMs) excel at understanding structural data and generating diverse and fluent language []. They have demonstrated capabilities in spatial reasoning [], solving both elementary and sophisticated math problems [, ]. To address the limitations of 2D-LLMs when annotate 3D scenes, we leverage the strengths of LLMs. By integrating detailed, accurate information into a reliable scene graph, we provide LLMs with the necessary data to reason effectively and generate precise annotations.

Figure: Figure 3: 3D-GRAND Data Curation Pipeline.
Refer to caption: x3.png

Here are the key steps of applying our pipeline to obtain densely-grounded annotation for any synthetic 3D scene:

$\bullet$ 3D Model to 2D Image. In the 3D-Front dataset, each object is sourced from 3D-Future [], which provides a ground truth 2D image for each object. For the Structured3D dataset, individual images for each object are not available. Therefore, we utilize the set-of-mark prompting technique [], where each object to be annotated is circled in red in the images.
$\bullet$ 2D Image to Attributes. We use GPT-4V to generate detailed language annotations for each 2D object image, including attributes like name, color, finish, and texture. The naming is now open-vocabulary, contrary to being class-agnostic.
$\bullet$ List of Attributes to Scene Graph. We structure each individual objects’ annotations into a JSON-based scene graph that captures the relationships and attributes of objects within the scene. Note that we obtain this scene graph from synthetic data which we can guarantee the correctness.
$\bullet$ Scene Graph to Generated Annotations. Based on the given scene graph, we will be able to produce 3D-Grounded Object Reference, 3D-Grounded Scene Description and 3D-Grounded QA using GPT-4 [] and GPT4-Turbo with various prompts, which we will show in the appendix.
$\bullet$ Generated Annotations to Processed Annotations.
After we acquire raw annotations, we will apply hallucination filters and template augmentation for the phrase tag to remove low quality annotations and augment generated annotations.

We want to reinforce the set of grounded language tasks that cover a diverse range of language understanding and generation scenarios in 3D environments which we show in Figure 2:

$\bullet$ 3D-Grounded Object Reference: Given a 3D scene and an object of interest, 3D-LLM is required to generate a description that uniquely identifies the target object. The description includes text and grounding information, not only for the target object but also for any landmark objects mentioned in the description. This task is conceptually similar to Visual Grounding, Scene-aware Object Captioning, and Dense Captioning in 2D vision-language research.
$\bullet$ 3D-Grounded Scene Description: Given a 3D scene, the 3D-LLM generates a description that captures the salient aspects of the environment. The description includes both text and grounding information, linking the language to specific objects or regions in the scene.
$\bullet$ 3D-Grounded QA: Given a 3D scene and a question about the environment, the 3D-LLM generates an answer that is grounded in the scene. Both the question and answer include text and grounding information, ensuring that the 3D-LLM’s responses are contextually relevant and accurate.

Dataset highlights.
3D-GRAND possesses several unique features that distinguish it from existing 3D-language datasets: (1). *Large-scale*: With 40,087 scenes and 6.2 million annotations, 3D-GRAND is the largest 3D-language dataset to date, providing ample data for training and evaluating 3D-LLMs. (2). *Dense grounding*: Unlike recent million-scale datasets like SceneVerse, which lack grounded language annotations, each language annotation in 3D-GRAND is densely grounded to specific objects or regions within the 3D scenes, facilitating fine-grained language understanding and generation. (3). *Diverse language tasks*: 3D-GRAND supports a broad array of grounded language tasks, including object reference , spatial reasoning, and scene understanding, making it a comprehensive benchmark for evaluating 3D-LLMs.
(4). *High-quality annotations*: The language annotations in 3D-GRAND are meticulously collected, filtered, and evaluated to ensure they are of high quality, diverse, and natural.

These unique features establish 3D-GRAND as a valuable resource for advancing research in 3D-LLMs and embodied AI. By providing a large-scale, densely-grounded 3D-text dataset, 3D-GRAND enables the development and evaluation of more capable and reliable 3D-LLMs that can effectively understand and interact with the physical world.

## 4 3D-POPE: A Benchmark for Evaluating Hallucination in 3D-LLMs

To systematically evaluate the hallucination behavior of 3D-LLMs, we introduce the 3D Polling-based Object Probing Evaluation (3D-POPE) benchmark. 3D-POPE is designed to assess a model’s ability to accurately identify the presence or absence of objects in a given 3D scene.

Dataset.
To facilitate the 3D-POPE benchmark, we curate a dedicated dataset from the ScanNet [] dataset, utilizing the semantic classes from ScanNet200 []. Specifically, we use the ScanNet validation set as the foundation for evaluating 3D-LLMs on the 3D-POPE benchmark.

Benchmark design.
3D-POPE consists of a set of triples, each comprising a 3D scene, a posed question, and a binary answer (“Yes” or “No”) indicating the presence or absence of an object (Fig. 1 middle). To ensure a balanced dataset, we maintain a 1:1 ratio of existent to nonexistent objects when constructing these triples.
For the selection of negative samples (i.e., nonexistent objects), we employ three distinct sampling strategies:

- •
*Random Sampling*: Nonexistent objects are randomly selected from the set of objects not present in the 3D scene.
- •
*Popular Sampling*: We select the top-$k$ most frequent objects not present in the 3D scene, where $k$ equals the number of objects currently in the scene.
- •
*Adversarial Sampling*: For each positively identified object in the scene, we rank objects that are not present and have not been used as negative samples based on their frequency of co-occurrence with the positive object in the training dataset. The highest-ranking co-occurring object is then selected as the adversarial sample. This approach differs from the original POPE [] to avoid adversarial samples mirroring popular samples, as indoor scenes often contain similar objects.

These sampling strategies are designed to challenge the model’s robustness and assess its susceptibility to different levels of object hallucination.

Metrics.
To evaluate the model’s performance on the 3D-POPE benchmark, we use key metrics including Precision, Recall, F1 Score, Accuracy, and Yes (%).
Precision and Recall assess the model’s ability to correctly affirm the presence of objects and identify the absence of objects, respectively. Precision is particularly important as it indicates the proportion of non-existing objects generated by the 3D-LLMs. The F1 Score, combining Precision and Recall, offers a balanced view of performance and serves as the primary evaluation metric. Accuracy measures the proportion of correctly answered questions, encompassing both “Yes” and “No” responses. Additionally, the Yes (%) metric reports the ratio of incorrect “Yes” responses to understand the model’s tendencies regarding object hallucination.

Leaderboard.
We establish a public leaderboard for the 3D-POPE benchmark, allowing researchers to submit their 3D-LLM results and compare their performance against other state-of-the-art models. The leaderboard reports the evaluation metrics for each model under the three sampling strategies, providing a transparent and standardized way to assess the hallucination performance of 3D-LLMs.

**Table 2: 3D-POPE benchmark results for evaluating hallucination in 3D language models. Random Baseline refers to a model randomly predicting “yes” or “no” with 50% chance, given the 1:1 positive/negative sample ratio in the dataset.**
| Dataset | 3D-POPE | Model | Precision | Recall | F1 Score | Accuracy | Yes (%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ScanNet200 Val | Random | Random Baseline | 50.00 | 50.00 | 50.00 | 50.00 | 50.00 |
| 3D-LLM [] | 50.03 | 99.88 | 66.67 | 50.07 | 99.81 |  |  |
| 3D-VisTA [] | 50.12 | 53.58 | 51.79 | 49.66 | 53.95 |  |  |
| LEO [] | 51.95 | 77.65 | 62.25 | 52.91 | 74.73 |  |  |
| Ours zero-shot (Grounding) | 93.34 | 84.25 | 88.56 | 89.12 | 45.13 |  |  |
| Popular | Random Baseline | 50.00 | 50.00 | 50.00 | 50.00 | 50.00 |  |
| 3D-LLM [] | 49.97 | 99.88 | 66.61 | 49.94 | 99.94 |  |  |
| 3D-VisTA [] | 47.40 | 51.88 | 49.54 | 49.49 | 52.30 |  |  |
| LEO [] | 48.30 | 77.65 | 59.55 | 47.27 | 80.38 |  |  |
| Ours zero-shot (Grounding) | 73.05 | 84.28 | 78.26 | 76.59 | 57.69 |  |  |
| Adversarial | Random Baseline | 50.00 | 50.00 | 50.00 | 50.00 | 50.00 |  |
| 3D-LLM [] | 49.97 | 99.88 | 66.61 | 49.94 | 99.94 |  |  |
| 3D-VisTA [] | 48.28 | 54.39 | 51.15 | 51.14 | 52.99 |  |  |
| LEO [] | 48.47 | 77.98 | 59.78 | 47.52 | 80.45 |  |  |
| Ours zero-shot (Grounding) | 69.86 | 84.21 | 76.37 | 73.95 | 60.26 |  |  |

|  | Model | Acc@0.25 | Acc@0.5 |
| --- | --- | --- | --- |
| Non-zero-shot | 3D-LLM (flamingo) [] | 21.2 | - |
| 3D-LLM (BLIP2-opt) [] | 29.6 | - |  |
| 3D-LLM (BLIP2-flant5) [] | 30.3 | - |  |
| Zero-shot | LLM-Grounder [] | 17.1 | 5.3 |
| Sim-to-real zero-shot | 3D-GRAND | 38.0 | 27.4 |

Table: Table 3: ScanRefer Results for evaluating visual grounding capability of 3D-LLMs. 3D-GRAND achieves the best zero-shot performance among 3D-LLMs, providing signals for sim-to-real transfer.

| 3D-POPE | Model | Precision |
| --- | --- | --- |
| Random | 3D-GRAND | 93.34 |
| w/o grounding tokens | 91.96 |  |
| Popular | 3D-GRAND | 73.05 |
| w/o grounding tokens | 70.37 |  |
| Adversarial | 3D-GRAND | 69.86 |
| w/o grounding tokens | 67.48 |  |

Table: Table 4: Ablation on 3D-POPE. Without the grounding tokens, 3D-GRAND hallucinates more.

## 5 Experiments

In this section, we present our experimental setup, including the baselines, datasets, and implementation details. We then report the results of our approach, denoted as 3D-GRAND on the ScanRefer [] and the 3D-POPE benchmark, demonstrating the effectiveness in improving grounding and reducing hallucination. Finally, we conduct an ablation study to analyze the impact of different components of our model and training strategy.

### 5.1 Experimental Setup

Baselines.
We compare our 3D-GRAND against the following baselines: 3D-LLM [], LEO [], and 3D-Vista []. Each model, along with the specific checkpoint used to obtain the results, is documented in the appendix. Our proposed model is based on Llama-2 []. The input is object-centric context, including a scene graph with each object’s category, centroid (x, y, z), and extent (width, height, depth), along with the text instruction and user query. During training, we utilized ground-truth centroids and extents. For inference, we used bounding boxes predicted by Mask3D []. Examples of input/output and details of the model can be found in the supplementary material.

Datasets.
We evaluate our model 3D-GRAND on two datasets: 3D-POPE and ScanRefer.
3D-POPE is our newly introduced benchmark dataset for evaluating object hallucination in 3D-LLMs, as described in Section 4. For ScanRefer, We utilized the validation split which contains 9,508 natural language descriptions of 2,068 objects in 141 ScanNet [] scenes.

Metrics.
For the ScanRefer benchmark, we use the official evaluation metrics, including Accuracy@0.25IoU and Accuracy@0.5IoU. For the 3D-POPE benchmark, we report accuracy, precision, recall, F1 score, and “Yes” rate under the three sampling strategies described in Section 4.

Implementation Details.
The 3D-GRAND model is LoRA-finetuned [] based off Llama-2. We use DeepSpeed ZeRO-2 [] and FlashAttention [] to save GPU memory and speed up training. The model is trained in BF16 precision on 12 NVIDIA A40 GPUs with a combined batch size of 96 and a learning rate of 2e-4. We use the AdamW [] optimizer with a weight decay of 0.01 and a cosine learning rate scheduler. We train the mode for 10k steps, which takes approximately 48 hours.

### 5.2 Results on 3D-POPE

We first evaluate these approaches on 3D-POPE and report results on Table 2.
Results show that 3D-LLM [] almost always produces yes responses to any question. 3D-VisTA [] performs similarly to the random baseline. LEO [] tends to answer yes frequently, but its precision indicates a similar object hallucination rate to the random baseline.
In our evaluation, 3D-GRAND achieved exceptional performance, with 93.34% precision and 89.12% accuracy when tested with random sampling. However, our model struggles with the more difficult splits, Popular and Adversarial, which demonstrates the effectiveness and rigorousness of 3D-POPE as a benchmark. Moreover, we emphasize that our model has never encountered ScanNet during training. More analysis on 3D hallucination can be found in the supplementary material.

### 5.3 Results on ScanRefer

We report results on ScanRefer in Table 4. Our model significantly outperforms all the baselines, achieving state-of-the-art zero-shot results on both Acc@0.25 and Acc@0.5. Notably, our model surpasses the previous best-performing model, 3D-LLM, by 7.7% on accuracy@0.25IoU. We emphasize that our model, unlike 3D-LLM, has never seen ScanNet scenes during its training (zero-shot) and is only trained on synthetics 3D scene instead of real scans. Therefore, this results provide a promising early signal that sim-to-real transfer can be achieved via our dense-grounded large-scale dataset.

**Table 5: Ablation Study on Grounding Accuracy (%) on ScanRefer: Training with densely-grounded data significantly improves grounding accuracy, particularly when multiple distractor objects of the same category are present in the room.**
| Method | Det. | Unique | Multiple | Overall |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Acc@0.25 | Acc@0.5 | Acc@0.25 | Acc@0.5 | Acc@0.25 | Acc@0.5 |  |  |
| Best IoU (upper bound) | Mask3D (Top100) | 93.7 | 66.8 | 91.6 | 70.7 | 92.4 | 69.2 |
| Best IoU (upper bound) | Mask3D (Top40) | 81.2 | 58.7 | 80.7 | 62.4 | 80.9 | 61.0 |
| Non-grounded Model | Mask3D (Top40) | 51.8 | 33.1 | 21.3 | 17.9 | 34.2 | 24.3 |
| Grounded Model (ground later) | Mask3D (Top40) | 50.4 | 32.4 | 26.0 | 20.5 | 36.3 | 25.5 |
| Grounded Model (ground first) | Mask3D (Top40) | 54.4 | 36.4 | 26.0 | 20.8 | 38.0 | 27.4 |
| Best IoU (upper bound) | GT | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 |
| Non-grounded Model | GT | 90.8 | 90.8 | 26.0 | 26.0 | 53.4 | 53.4 |
| Grounded Model | GT | 91.0 | 91.0 | 32.1 | 32.1 | 57.0 | 57.0 |

Figure: (a) Grounding capability. Higher is better.
Refer to caption: x4.png

### 5.4 Ablation Study

To better understand the impact of different components of our 3D-LLM, we conduct an ablation study on the ScanRefer and 3D-POPE benchmarks.

Grounding tokens.
We show the results of our model with different types of grounding methods in Table 5.
We also show results on 3D-POPE in Table 4.
In general, the model has a worse grounding performance and more hallucinations without grounding tokens. “Ground First” and “ground later” refer to whether the dense grounding (grounding every single object mentioned) of the object reference query happens before or after the model outputs the final answer for the refer expression. The former effectively constitutes a chain-of-thought reasoning process [], which is likely why the performance increases compared to the latter. See Appendix for details.

Mask3D proposals.
Finally, we show the upper bound of our approach in Table 5.
Our results are based on Mask3D proposals.
Due to the context length of LLM, we only use top-40 proposals.

### 5.5 Data Scaling and Sim-to-Real Transfer

The results are presented in Figure 4. Our model is trained on synthetic 3D scenes from 3D-FRONT and Structured3D [, ], and evaluated on real-world 3D scans from ScanNet []. The grounding performance consistently improves, and the hallucination rate drops as the densely-grounded data scales up. Notably, our model trained on densely grounded data scales better than the same model trained without such data. These findings pave the way for a future where we can scale 3D-text understanding using synthetic scenes obtained from simulation, which is much cheaper and more accessible to obtain.

## 6 Conclusion

In this paper, we introduced 3D-GRAND, a large-scale, densely-grounded 3D-text dataset designed for grounded 3D instruction tuning, and 3D-POPE, a comprehensive benchmark for evaluating object hallucination in 3D-LLMs. Through extensive experiments, we demonstrated the effectiveness of our dataset on 3D-LLMs in improving grounding and reducing hallucination, achieving state-of-the-art performance on the ScanRefer and 3D-POPE benchmarks. Our ablation study and qualitative analysis highlighted the importance of densely-grounded instruction tuning, the data scaling law, and effective sim-to-real transfer in developing high-performing 3D-LLMs. We hope our contributions and findings can spark further research and innovation in this field, ultimately leading to the development of more advanced and capable 3D-LLMs for a wide range of applications.

## 7 Acknowledgement

This work is generously supported by NSF IIS-1949634, NSF SES-2128623, and has benefited from the Microsoft Accelerate Foundation Models Research (AFMR) grant program.

## Appendix A Model Details

### A.1 Model input and output demonstration

In Figure 5, we show an example of 3D-GRAND model’s input and output on the Grounded Object Reference task. Note how in the “Response”, we train the model to generate a $\langle$detailed_grounding$\rangle$ pair of tags to densely ground every single object mentioned in the refer expression after generating the grounding answer in $\langle$refer_expression_grounding$\rangle$. The “ground first” “ground later” in Table 5 means whether the $\langle$detailed_grounding$\rangle$ tags happen before or after the $\langle$refer_expression_grounding$\rangle$ tags. Figure 5 is an example of “ground later”, and Figure 6 shows an example of “ground first”.

Figure: Figure 5: 3D-GRAND model input and output on Grounded Object Reference task.
Refer to caption: extracted/5661014/figures/3d_grand_model_example.png

Figure: Figure 6: Demo of interactive chat interface with the 3D-GRAND model.
Refer to caption: extracted/5661014/figures/gradio_demo.png

### A.2 Training Data

There are two flavors of models that we fine-tuned: one grounded object reference model, and one grounded QA model. The grounded object reference model was trained using the grounded object reference data on 3D-FRONT train split, which consist of 234,791 3D-text pairs, each of which are densely grounded. This model was used to generate the ScanRefer results presented in Table 4, 5, and Figure 4 The grounded QA model was trained using a subset of 200k grounded QA: object existence data from the 3D-FRONT train split. The reason that we select a subset of 200k QAs is simply because the entire grounded QA dataset is too large and we do not have enough resource to train on all data. However, as shown in Table 2 and Figure 4, we find even such a subset is very effective in reducing hallucinations in 3D-LLMs.

We provide official data splits of train, val and test (90%, 5%, 5%) in our dataset release. The val and test proportion might seem small, but given our dataset’s million-scale, they should be sufficiently large for any development and evaluation purposes.

### A.3 Training Details

The two flavors of model mentioned above are LoRA-finetuned [] based off Llama-2. We use DeepSpeed ZeRO-2 [] and FlashAttention [] to save GPU memory and speed up training. The model is trained in BF16 precision on 12 NVIDIA A40 GPUs with a combined batch size of 96 and a learning rate of 2e-4. We use the AdamW [] optimizer with a weight decay of 0.01 and a cosine learning rate scheduler. We train the mode for 10k steps, which takes approximately 48 hours.

## Appendix B Additional 3D-GRAND Data Collection

### B.1 Point Cloud Generation Pipeline for 3D-Front

Here, we present an expanded version of Section 3, focusing on the methodologies employed in the collection and cleaning of 3D scenes, specifically detailing our process for deriving 3D point clouds from existing datasets.

Figure: Figure 7: Point Cloud Generation for 3D-Front.
Refer to caption: extracted/5661014/figures/pcdgeneration.png

In our workflow with 3D-FRONT, layouts and meshes are initially processed in Blender to produce multi-view images. These images are subsequently used to construct comprehensive point clouds for entire houses. Both point clouds and per-room meshes are utilized to generate scene-level point clouds. We avoid direct use of room meshes because they lack color information in ceilings, walls, and floors, necessitating the final output to be a point cloud.

For Structure3D, while per-scene multi-view images facilitate direct rendering of per-scene point clouds, we frequently encounter issues where parts of adjacent scenes are inadvertently reconstructed due to window transparency. To address this, we employ the layout of each scene to trim extraneous points, thus enhancing the precision of the resulting point clouds.

## Appendix C Addtional 3D POPE results

### C.1 3D Pope Results on NYU40

Table 6 presents evaluation results for 3D POPE using the NYU40 class set. NYU40 includes a subset of the classes from ScanNet200 featured in the main results table. The NYU40 class set consolidates many fine-grained classes into an “other” category, potentially reducing the challenge of negative sampling in the *Popular* and *Adversarial* settings compared to the ScanNet200 scenario.

**Table 6: Results of 3D-LLMs under three evaluation settings of 3D-POPE on the validation set of ScanNet using NYU40 class set. Yes denotes the proportion of answering “Yes” to the given question. The best results in each block are denoted in bold.**
| Dataset | 3D-POPE | Model | Accuracy | Precision | Recall | F1 Score | Yes (%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ScanNet Val (NYU40) | Random | 3D-LLM | 50.00 | 50.00 | 100.00 | 66.67 | 100.00 |
| 3D-VisTA | 50.12 | 50.08 | 77.13 | 60.73 | 77.01 |  |  |
| LEO | 54.03 | 52.70 | 78.52 | 63.07 | 74.50 |  |  |
| Ours zero-shot (No Grounding) | 86.45 | 87.26 | 85.36 | 86.30 | 48.91 |  |  |
|  | Ours zero-shot (Grounding) | 85.68 | 88.22 | 82.34 | 85.18 | 46.67 |  |
| Popular | 3D-LLM | 50.00 | 50.00 | 100.00 | 66.67 | 100.00 |  |
| 3D-VisTA | 50.27 | 50.23 | 77.13 | 60.84 | 76.91 |  |  |
| LEO | 48.86 | 49.28 | 77.44 | 60.23 | 78.58 |  |  |
| Ours zero-shot (No Grounding) | 80.85 | 78.30 | 85.35 | 81.68 | 54.50 |  |  |
|  | Ours zero-shot (Grounding) | 81.69 | 81.32 | 82.28 | 81.80 | 50.59 |  |
| Adversarial | 3D-LLM | 50.00 | 50.00 | 100.00 | 66.67 | 100.00 |  |
| 3D-VisTA | 50.44 | 50.48 | 77.14 | 61.03 | 76.86 |  |  |
| LEO | 49.77 | 49.85 | 77.67 | 60.73 | 77.91 |  |  |
| Ours zero-shot (No Grounding) | 81.47 | 78.98 | 85.78 | 82.24 | 54.31 |  |  |
|  | Ours zero-shot (Grounding) | 82.10 | 81.72 | 82.72 | 82.22 | 50.61 |  |

## Appendix D Human Validation

Because our dataset generation process involves GPT-4V, there is a potential for hallucinations. We identify three types of possible hallucinations that could impact our dataset: the text might inaccurately describe an object’s property, such as color or size (termed incorrect object attribute); it might incorrectly depict the spatial relationship between two objects (termed incorrect spatial relation); or it might describe an object that does not exist in the referenced scene at all (termed incorrect object existence). Additionally, inaccuracies in our dataset may also arise from incorrectly grounding the wrong object.

To validate our dataset against these potential failures, we plan to verify a subset of our data through crowdsourcing to ascertain the frequency of these failure cases.

### D.1 Crowd-sourcing

We crowd-source the validation of annotations using Hive, a platform commonly used for sourcing annotations for computer vision tasks. The platform can be accessed at [https://thehive.ai/](https://thehive.ai/).

We conceptualize our dataset validation as a data annotation problem, employing scene-text pairs as the data unit. Annotators are instructed to label these pairs as “True” or “False” to indicate the presence or absence of hallucinations or inaccuracies. Additionally, a “Cannot Decide” option is provided to accommodate cases where the scene view is unclear.

#### D.1.1 Task Generation

Hive only supports presenting static images to annotators, so we generate annotation tasks by composing snapshots of a scene with corresponding text annotations. For each task, we take snapshots from four different angles and pair them with a corresponding annotation. To maintain simplicity and conciseness, we require validation of just one sentence per task, providing some surrounding context and highlighting the target sentence. For grounding validation, the grounded object is outlined in the scene with a bounding box, and the referring phrase in the sentence is emphasized. An example of such a task, along with the annotation interface, is depicted in Figure 8. Figure 9 displays two text validation tasks and two grounding validation tasks that were presented to annotators.

Figure: Figure 8: Example of a dataset validation task presented to crowd-sourcing annotators: It displays a scene from four different angles alongside the sentence to be validated, which is highlighted. Annotators have the options to select "True," "Not True," or "Cannot Decide." On the left side of the screen, the instructions are repeated for annotators’ reference. In this example, "True" is selected.
Refer to caption: x6.png

Figure: (a) "True" example of a text validation task.
Refer to caption: x7.png

#### D.1.2 Crowd-sourcing Validity

Validating a dataset necessitates a high level of attention from annotators. We curate sets of instructions, qualifying tasks, and honeypot tasks to ensure that the annotations obtained from crowdsourcing are reliable. The crowdsourcing process is illustrated in Figure 10.

Figure: Figure 10: Illustration of the crowd-sourcing process. Annotators are first shown instruction sets that describe both how the task should be completed and the possible inaccuracies that could appear in the data. They are then presented with qualifier tasks, and annotators who do not get a high enough accuracy on these tasks are banned from annotating our dataset. Annotators who pass the qualifier are able to annotate real tasks, but are randomly presented with honeypots that are indistinguishable from real tasks. Annotators who do not get a high enough accuracy on honeypots are also banned from our dataset.
Refer to caption: x11.png

Before presenting any tasks to the workers, we present them with a set of instruction tasks that show an example annotation, the correct response (as determined by us), and the reason why that response is correct. They are paired with an incorrect example and an explanation of why it is incorrect in order to ensure unbiased annotations. Examples of qualifying instructions are shown in 11. These instructions are intentionally brief, as we found through trial-and-error that longer, paragraph-based instructions were largely ignored by annotators.

Figure: (a) "True" example instruction for the text validation task.
Refer to caption: x12.png

Qualifying tasks are presented to the annotators before they are shown any real tasks in order to train them to complete the real task with a high accuracy. Annotators are both shown the correct answer and a reasoning as to why it is correct for every qualifier. We set the minimum qualifier accuracy to 0.75 to ensure that annotators must achieve a minimum competency before annotating real tasks. Every dataset is given between 12 and 30 specially crafted qualifying tasks that demonstrate the possible inaccuracies that could appear in the data. These qualifiers are divided equally between true and false examples so as not to bias workers towards any one answer.

Honeypot tasks are randomly mixed in with real tasks in order to ensure that annotators are maintaining a high quality of annotations throughout the entire job. Because we annotate the honeypot tasks before showing them to annotators, we are able to evaluate any given worker’s accuracy on honeypot tasks. We set the minimum honeypot accuracy to 0.89 to ensure that annotators are maintaining correct annotations. Workers that do not maintain this accuracy are banned from annotating our tasks. This is higher than the required accuracy for qualifiers because we expect annotators to already be well trained in our annotation tasks from the instructions and qualifiers. Every data type is given between 18 and 35 honeypot tasks. The honeypots are also approximately divided equally between true and false examples so that workers who consistently select a single answer without paying attention to the task (e.g., someone who always selects “True") will be banned.

To further ensure high-quality annotations, we send each question to 3 different annotators and only accept an annotation if at least 2 out of the 3 annotators agree with each other on the truthfulness of an item. If agreement is not reached, the task is returned as inconclusive.

### D.2 Results

We perform validation on 10,200 room-annotation pairs. From each of the three data types, 1,700 pairs are sampled for validation of both text truthfulness and grounding accuracy. A subset of 800 rooms is uniformly chosen, with 400 designated for text truthfulness and another 400 for grounding accuracy. The text data is uniformly sampled from these rooms. We report accuracies for both text truthfulness and grounding accuracy in Table 7.

We report comprehensive statistics from the annotation process in Table 8. We observe a very low qualifier pass rate ranging from 11 - 20 % across the different tasks in our data, suggesting that our qualifiers were effective in allowing only the most attentive annotators qualify to annotate real tasks. In addition, none of these annotators were banned due to honeypots. This increases our confidence that our qualification process is effective in training annotators and filtering out those who were not attentive. We also observe that workers spend roughly the same time on real tasks and honeypot tasks, suggesting that the honeypots are indistinguishable from real tasks for the annotators. This further supports the validity of our annotations.

**Table 7: Text Truthfulness and Grounding Accuracy from crowdsourcing. Accuracy is computed by dividing the number of “True” responses by the total number of tasks (1700).**
| Method | Text Truthfulness | Grounding Accuracy |
| --- | --- | --- |
| Grounded Scene Description | 0.877 | 0.944 |
| Grounded QA | 0.852 | 0.956 |
| Grounded Object Reference | 0.863 | 0.918 |

**Table 8: Comprehensive annotation metrics. Includes qualifier pass rate, honeypot count, honeypot ban rate, percent of tasks marked inconclusive (where workers could not come to an agreement on the label), and the average time that workers spend on both real tasks and honeypot tasks. Each dataset was evaluated on 1700 annotations. At least 2 workers must agree on the label for an annotation to be considered valid.**
| Category | Type | % Qualifier Pass Rate | # Honeypots | % Honeypot Ban Rate | % of Inconclusive Tasks | Avg. Real Task Speed (s) | Avg. Honeypot Speed (s) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| (Pass) | (Total) | (Ban) | (Tasks) | (Real) | (Honeypot) |  |  |
| Text Accuracy | Scene Description | 17 | 35 | 0 | 2.03 | 17.91 | 17.08 |
| QA | 19 | 18 | 0 | 1.35 | 16.22 | 16.64 |  |
| Object Reference | 10 | 38 | 0 | 0.82 | 19.88 | 17.57 |  |
| Grounding Accuracy | Scene Description | 20 | 18 | 0 | 1.66 | 9.69 | 12.84 |
| QA | 16 | 18 | 0 | 1.11 | 6.65 | 11.14 |  |
| Object Reference | 20 | 18 | 0 | 1.88 | 9.69 | 12.84 |  |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: See Weaknesses
soundness: 3
strengths: - The paper introduces a large-scale, densely-grounded dataset, 3D-GRAND, comprising 40k 3D scenes and 6.2 million language annotations that support various grounded tasks such as object reference, scene description, and question-answering.
- 3D-POPE, a new benchmark, is introduced to systematically evaluate and quantify object hallucination in 3D-LLMs. By posing existence questions about objects within scenes, 3D-POPE enables measurement of hallucination rates and provides fair comparisons across models.
- The paper’s data collection pipeline leverages synthetic 3D scenes from the 3D-FRONT and Structured3D datasets, annotated using large language models like GPT-4 for dense grounding.
- Experimental results demonstrate that training with 3D-GRAND improves grounding performance and reduces hallucination, achieving superior results on benchmarks like ScanRefer and 3D-POPE. The authors conduct ablation studies and scaling analyses to highlight the impact of dense grounding data and explore sim-to-real transfer capabilities, establishing the dataset’s effectiveness in advancing 3D-LLMs.
summary: The paper introduces 3D-GRAND, a large-scale, densely grounded 3D-text dataset with 40,087 3D household scenes and 6.2 million text annotations for advancing 3D Large Language Models (LLMs) in understanding and grounding language in 3D environments. The dataset aims to improve the grounding capabilities of 3D-LLMs and reduce object hallucination, a common issue where models misidentify or invent objects in a scene.
weaknesses: - The benchmark primarily focuses on binary existence questions, lacking evaluation on more complex tasks such as spatial reasoning, attribute-based grounding, and relational understanding, which are essential for fully assessing a model’s hallucination in 3D comprehension.
- The proportion of data generated via templates versus LLM-generated annotations is unclear, and the paper does not provide specific examples of the templates used.
- How does the paper handle metric calculation in cases where objects in the model output are ungrounded? For example, if the model describes a “blue chair” that does not exist in the scene or fails to reference an existing object, how are grounding accuracy and hallucination rates calculated in such instances?
- In addition to grounding experiments, the paper could benefit from including language tokens prediction experiments, which would further demonstrate the dataset's effectiveness in training language understanding components of 3D-LLMs.
- The model design is relatively straightforward, relying on scene graphs to process 3D scenes and output JSON representations. This approach lacks advanced reasoning capabilities, which could limit the model’s ability to handle tasks requiring deeper understanding and inference in 3D environments.
- The figures in the paper need to be refined to improve their quality.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1) Given the reliance on synthetic scenes in 3D-GRAND, have you observed or tested any bias when generalizing to real-world 3D data, particularly regarding scene composition and object variability?
2) Could you elaborate on the criteria and process used by the hallucination filter to distinguish between correct and hallucinated object references?
3) Given the annotation error rate of 5.6–8.2%, could you discuss how these errors influence model predictions, especially in densely grounded tasks?
soundness: 3
strengths: 1) Innovative Large-Scale Dataset for 3D-Language Grounding:
3D-GRAND is a pioneering contribution in its scale and grounding precision, filling a critical gap in the resources available for 3D-LLMs. Its dense grounding annotations at the noun-phrase level improve object-level clarity and enable the model to better understand spatial relations and attributes within complex scenes. The dataset’s vast scale allows for diverse applications, from spatial QA to scene description, establishing a robust foundation for training 3D-LLMs.

2) Comprehensive Hallucination Benchmark with 3D-POPE:
The introduction of 3D-POPE as a dedicated benchmark for 3D object hallucination is a valuable addition. Its well-defined metrics, including precision, recall, and F1 scores, allow for standardized, transparent evaluation of hallucination tendencies across different 3D-LLMs. The benchmark’s structured evaluation through varied sampling (random, popular, and adversarial) makes it rigorous, challenging models to correctly identify the presence or absence of objects even under difficult conditions.

3) Effective Use of Synthetic Data for Cost and Scalability:
By leveraging synthetic 3D scenes and automated annotation, 3D-GRAND achieves substantial cost and time savings over traditional human-annotated datasets. The paper highlights the efficiency of using large language models (GPT-4) for annotation, which makes scaling feasible and reduces labor-intensive work while maintaining a low annotation error rate. This approach supports large-scale experimentation and promotes accessibility for researchers working with limited resources.

4) Promising Signals for Sim-to-Real Transfer: The paper provides encouraging evidence that models trained on synthetic 3D-GRAND data perform well on real-world 3D scans, indicating potential for sim-to-real transfer. This transfer capability is crucial for embodied AI applications, such as robotics, where real-world adaptability is often required. By validating the dataset’s utility for real-world tasks, the authors strengthen the practical relevance of their work.

5) Extensive Experimental Analysis and Ablations: The authors conduct comprehensive experiments across multiple datasets, including 3D-POPE and ScanRefer, to demonstrate the benefits of densely-grounded data. Ablation studies further highlight the impact of various grounding techniques, such as the inclusion of grounding tokens, on hallucination and grounding accuracy. This level of experimental rigor provides strong evidence for the dataset’s effectiveness and aids in understanding the benefits and limitations of the approach.
summary: The paper presents 3D-GRAND, a large-scale dataset designed to advance grounded language understanding in 3D-based large language models (3D-LLMs). With over 40,000 household scenes and 6.2 million densely-grounded scene-language pairs, 3D-GRAND aims to improve grounding accuracy and reduce object hallucination—a common issue in language models interpreting 3D environments. Alongside 3D-GRAND, the authors introduce 3D-POPE, a benchmark specifically designed to evaluate hallucination in 3D-LLMs. The paper demonstrates the effectiveness of training 3D-LLMs with 3D-GRAND by showing that the dataset significantly enhances grounding capabilities and reduces hallucinations. Results also indicate promising sim-to-real transfer capabilities, suggesting that synthetic data training could translate well into real-world performance. Together, 3D-GRAND and 3D-POPE contribute essential resources to the embodied AI community and provide a foundation for further research in 3D-LLM grounded reasoning.
weaknesses: 1) While 3D-GRAND demonstrates early signs of sim-to-real transfer, it relies heavily on synthetic data from the 3D-FRONT and Structured3D datasets. There’s limited analysis on potential biases that might arise from this reliance, especially since synthetic scenes can differ significantly from real-world 3D environments in object layout and scene composition. This limitation could affect the model’s performance on real-world tasks where variability is higher.
2) Although the paper introduces a hallucination filter, the methodology behind it is not fully explained. For instance, it’s unclear how specific filtering criteria are determined, or how consistently the filter distinguishes valid object references from hallucinated ones. This could lead to inconsistencies in the data and, consequently, model performance. A more transparent breakdown of the filter’s effectiveness, or examples of common hallucination cases, would enhance confidence in the filtering approach.
3) Dense grounding annotations at the noun-phrase level are presented as beneficial, but the paper does not discuss potential downsides, such as increased model complexity or the risk of overfitting to these annotations. It’s also unclear whether dense grounding provides a significant advantage over sparse grounding for all task types or only specific ones. A comparison with sparse grounding on certain benchmarks could clarify this point.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Relevant 3D vision-language datasets and benchmarks such as EmbodiedScan[1], SceneVerse[2], MMScan[3] and MSR3D[4] have trained baseline 3D VLMs on their large-scale dataset and provide a fair comparison. Why is no existing 3D VLM trained on 3D-GRAND? Are there any underlying limitations or challenges that have not been explicitly discussed in the paper? Additionally, does the performance improvement of the proposed 3D-GRAND model over other baselines arise primarily from the privileged inputs (e.g., object locations and attributes) or the pretraining on the 3D-GRAND dataset?

[1] EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI
[2] SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding
[3] MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations
[4] Multi-modal Situated Reasoning in 3D Scenes
soundness: 2
strengths: 1. The scale of the language instances is significantly larger than that of previous works, such as SceneVerse and EmbodiedScan.
2. The authors conducted manual and programmatic reviews of the dataset samples and provided statistical results related to data quality. The details of the data generation process are clearly presented, making it easy to understand.
3. The evaluation results emphasize the problem of object hallucination in existing 3D-VLMs for 3D applications.
summary: This paper introduces 3D-GRAND, a large-scale dataset comprising 6.2 million densely scene-grounded data instances. The dataset is built upon existing synthetic 3D assets, including 3D-Front and Structured3D. The authors have developed a data generation pipeline based on scene graphs, which have been proposed in previous works [1][2]. Additionally, the paper designs a benchmark called 3D-POPE to facilitate object hallucination. A benchmark 3D-POPE is designed based on experiences from 2D Vision-Language Models (VLMs) [3]. The experiment results reveal the limitation of existing 3D-VLMs in object hallucination. The paper proposes a large language model (LLM) named 3D-GRAND and evaluates its performance on the dataset compared to other 3D VLMs. However, no existing 3D VLMs have been trained on the large-scale 3D-GRAND dataset, leaving its impact on these models unknown.

In summary, while the dataset presents a scene graph-based data generation pipeline, its overall technical contribution to dataset collection is weak. The paper also proposes a new benchmark to evaluate the object hallucination of 3D VLMs. However, no existing baseline models have been trained on the large-scale 3D-GRAND dataset, which leads to a lack of analyses for the dataset and models. Consequently, the experiment results do not provide useful insights for the community to improve model design.

[1] An Embodied Generalist Agent in 3D World
[2] SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding
[3] Evaluating Object Hallucination in Large Vision-Language Models
weaknesses: 1. LEO [1] and SceneVerse [2] have proposed the scene graph-based data generation pipeline. The pipeline's technical contribution is very minor compared to previous work.
2. No existing 3D VLM is trained based on 3D-GRAND dataset. The author only mentions that they train 3D-GRAND models based on a specific subset of 3D-FRONT in the appendix. There should be 3D VLMs(existing models such as 3D-LLM, LEO, etc.) trained based on 3D-GRAND as evidence of the quality of the dataset. This results in poor integrity of the article and greatly reduces the contribution of the article.
3. Unfair comparison with the baseline models. The proposed baseline model 3D-GRAND takes the object locations and attributes as the input, which provides strong information on the whole 3D scene without any noise. While other models such as 3D-LLM, 3D-Vista, and LEO take the 3D point cloud or multi-view images as input. 
4. The training set of the baseline model 3D-GRAND should be presented in section 3 instead of in the appendix.

[1] An Embodied Generalist Agent in 3D World
[2] SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding