3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and
  Less Hallucination
Authors: name: Jianing Yang; name: Xuweiyi Chen; name: Nikhil Madaan; name: Madhavan Iyengar; name: Shengyi Qian; name: David F. Fouhey; name: Joyce Chai
Published: 2024-06-07 17:59:59

Abstract: The integration of language and 3D perception is crucial for developing
embodied agents and robots that comprehend and interact with the physical
world. While large language models (LLMs) have demonstrated impressive language
understanding and generation capabilities, their adaptation to 3D environments
(3D-LLMs) remains in its early stages. A primary challenge is the absence of
large-scale datasets that provide dense grounding between language and 3D
scenes. In this paper, we introduce 3D-GRAND, a pioneering large-scale dataset
comprising 40,087 household scenes paired with 6.2 million densely-grounded
scene-language instructions. Our results show that instruction tuning with
3D-GRAND significantly enhances grounding capabilities and reduces
hallucinations in 3D-LLMs. As part of our contributions, we propose a
comprehensive benchmark 3D-POPE to systematically evaluate hallucination in
3D-LLMs, enabling fair comparisons among future models. Our experiments
highlight a scaling effect between dataset size and 3D-LLM performance,
emphasizing the critical role of large-scale 3D-text datasets in advancing
embodied AI research. Notably, our results demonstrate early signals for
effective sim-to-real transfer, indicating that models trained on large
synthetic data can perform well on real-world 3D scans. Through 3D-GRAND and
3D-POPE, we aim to equip the embodied AI community with essential resources and
insights, setting the stage for more reliable and better-grounded 3D-LLMs.
Project website: https://3d-grand.github.io