Aligning Text, Images, and 3D Structure Token-by-Token
Authors: name: Aadarsh Sahoo, name: Vansh Tibrewal, name: Georgia Gkioxari

Creating machines capable of understanding the world in 3D is essential in
assisting designers that build and edit 3D environments and robots navigating
and interacting within a three-dimensional space. Inspired by advances in
language and image modeling, we investigate the potential of autoregressive
models for a new modality: structured 3D scenes. To this end, we propose a
unified LLM framework that aligns language, images, and 3D scenes and provide a
detailed ''cookbook'' outlining critical design choices for achieving optimal
training and performance addressing key questions related to data
representation, modality-specific objectives, and more. We evaluate performance
across four core 3D tasks -- rendering, recognition, instruction-following, and
question-answering -- and four 3D datasets, synthetic and real-world. We extend
our approach to reconstruct complex 3D object shapes by enriching our 3D
modality with quantized shape encodings, and show our model's effectiveness on
real-world 3D object recognition tasks. Project webpage:
https://glab-caltech.github.io/kyvo/