ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and
  Planning
Authors: name: Qiao Gu; name: Alihusein Kuwajerwala; name: Sacha Morin; name: Krishna Murthy Jatavallabhula; name: Bipasha Sen; name: Aditya Agarwal; name: Corban Rivera; name: William Paul; name: Kirsty Ellis; name: Rama Chellappa; name: Chuang Gan; name: Celso Miguel de Melo; name: Joshua B. Tenenbaum; name: Antonio Torralba; name: Florian Shkurti; name: Liam Paull
Published: 2023-09-28 17:53:38

Abstract: For robots to perform a wide variety of tasks, they require a 3D
representation of the world that is semantically rich, yet compact and
efficient for task-driven perception and planning. Recent approaches have
attempted to leverage features from large vision-language models to encode
semantics in 3D representations. However, these approaches tend to produce maps
with per-point feature vectors, which do not scale well in larger environments,
nor do they contain semantic spatial relationships between entities in the
environment, which are useful for downstream planning. In this work, we propose
ConceptGraphs, an open-vocabulary graph-structured representation for 3D
scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing
their output to 3D by multi-view association. The resulting representations
generalize to novel semantic classes, without the need to collect large 3D
datasets or finetune models. We demonstrate the utility of this representation
through a number of downstream planning tasks that are specified through
abstract (language) prompts and require complex reasoning over spatial and
semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer
video: https://youtu.be/mRhNkQwRYnc )