SceneScript: Reconstructing Scenes With An Autoregressive Structured
  Language Model
Authors: name: Armen Avetisyan; name: Christopher Xie; name: Henry Howard-Jenkins; name: Tsun-Yi Yang; name: Samir Aroudj; name: Suvam Patra; name: Fuyang Zhang; name: Duncan Frost; name: Luke Holland; name: Campbell Orme; name: Jakob Engel; name: Edward Miller; name: Richard Newcombe; name: Vasileios Balntas
Published: 2024-03-19 18:01:29

Abstract: We introduce SceneScript, a method that directly produces full scene models
as a sequence of structured language commands using an autoregressive,
token-based approach. Our proposed scene representation is inspired by recent
successes in transformers & LLMs, and departs from more traditional methods
which commonly describe scenes as meshes, voxel grids, point clouds or radiance
fields. Our method infers the set of structured language commands directly from
encoded visual data using a scene language encoder-decoder architecture. To
train SceneScript, we generate and release a large-scale synthetic dataset
called Aria Synthetic Environments consisting of 100k high-quality in-door
scenes, with photorealistic and ground-truth annotated renders of egocentric
scene walkthroughs. Our method gives state-of-the art results in architectural
layout estimation, and competitive results in 3D object detection. Lastly, we
explore an advantage for SceneScript, which is the ability to readily adapt to
new commands via simple additions to the structured language, which we
illustrate for tasks such as coarse 3D object part reconstruction.