Task-oriented Sequential Grounding in 3D Scenes
Authors: name: Zhuofan Zhang, name: Ziyu Zhu, name: Pengxiang Li, name: Tengyu Liu, name: Xiaojian Ma, name: Yixin Chen, name: Baoxiong Jia, name: Siyuan Huang, name: Qing Li

Grounding natural language in physical 3D environments is essential for the
advancement of embodied artificial intelligence. Current datasets and models
for 3D visual grounding predominantly focus on identifying and localizing
objects from static, object-centric descriptions. These approaches do not
adequately address the dynamic and sequential nature of task-oriented grounding
necessary for practical applications. In this work, we propose a new task:
Task-oriented Sequential Grounding in 3D scenes, wherein an agent must follow
detailed step-by-step instructions to complete daily activities by locating a
sequence of target objects in indoor scenes. To facilitate this task, we
introduce SG3D, a large-scale dataset containing 22,346 tasks with 112,236
steps across 4,895 real-world 3D scenes. The dataset is constructed using a
combination of RGB-D scans from various 3D scene datasets and an automated task
generation pipeline, followed by human verification for quality assurance. We
adapted three state-of-the-art 3D visual grounding models to the sequential
grounding task and evaluated their performance on SG3D. Our results reveal that
while these models perform well on traditional benchmarks, they face
significant challenges with task-oriented sequential grounding, underscoring
the need for further research in this area.