3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D
  Scene Understanding
Authors: name: Tatiana Zemskova; name: Dmitry Yudin
Published: 2024-12-24 14:21:58

Abstract: A 3D scene graph represents a compact scene model, storing information about
the objects and the semantic relationships between them, making its use
promising for robotic tasks. When interacting with a user, an embodied
intelligent agent should be capable of responding to various queries about the
scene formulated in natural language. Large Language Models (LLMs) are
beneficial solutions for user-robot interaction due to their natural language
understanding and reasoning abilities. Recent methods for creating learnable
representations of 3D scenes have demonstrated the potential to improve the
quality of LLMs responses by adapting to the 3D world. However, the existing
methods do not explicitly utilize information about the semantic relationships
between objects, limiting themselves to information about their coordinates. In
this work, we propose a method 3DGraphLLM for constructing a learnable
representation of a 3D scene graph. The learnable representation is used as
input for LLMs to perform 3D vision-language tasks. In our experiments on
popular ScanRefer, RIORefer, Multi3DRefer, ScanQA, Sqa3D, and Scan2cap
datasets, we demonstrate the advantage of this approach over baseline methods
that do not use information about the semantic relationships between objects.
The code is publicly available at
https://github.com/CognitiveAISystems/3DGraphLLM.