IGGT: Instance-Grounded Geometry Transformer for Semantic 3D
  Reconstruction
Authors: name: Hao Li, name: Zhengyu Zou, name: Fangfu Liu, name: Xuanyang Zhang, name: Fangzhou Hong, name: Yukang Cao, name: Yushi Lan, name: Manyuan Zhang, name: Gang Yu, name: Dingwen Zhang, name: Ziwei Liu

Humans naturally perceive the geometric structure and semantic content of a
3D world as intertwined dimensions, enabling coherent and accurate
understanding of complex scenes. However, most prior approaches prioritize
training large geometry models for low-level 3D reconstruction and treat
high-level spatial understanding in isolation, overlooking the crucial
interplay between these two fundamental aspects of 3D-scene analysis, thereby
limiting generalization and leading to poor performance in downstream 3D
understanding tasks. Recent attempts have mitigated this issue by simply
aligning 3D models with specific language models, thus restricting perception
to the aligned model's capacity and limiting adaptability to downstream tasks.
In this paper, we propose InstanceGrounded Geometry Transformer (IGGT), an
end-to-end large unified transformer to unify the knowledge for both spatial
reconstruction and instance-level contextual understanding. Specifically, we
design a 3D-Consistent Contrastive Learning strategy that guides IGGT to encode
a unified representation with geometric structures and instance-grounded
clustering through only 2D visual inputs. This representation supports
consistent lifting of 2D visual inputs into a coherent 3D scene with explicitly
distinct object instances. To facilitate this task, we further construct
InsScene-15K, a large-scale dataset with high-quality RGB images, poses, depth
maps, and 3D-consistent instance-level mask annotations with a novel data
curation pipeline.