ATISS: Autoregressive Transformers for Indoor Scene Synthesis

1 Introduction . Generating synthetic 3D content that is both realistic and diverse is a long-standing problem in computer vision and graphics . In the last decade , there has been increased demand for tools that automate the creation of 3D artificial environments for applications like video games and AR/VR , as well as general 3D content creation [ 61 , 16 , 36 , 4 , 62 ] . These tools can also synthesize data to train computer vision models , avoiding expensive and laborious annotations . Generative models [ 28 , 19 , 13 , 29 , 56 ] have demonstrated impressive results on synthesizing photorealistic images [ 7 , 1 , 24 , 8 , 25 ] and intelligible text [ 46 , 2 ] , and are beginning to be adopted for the generation of 3D environments . Recent works proposed to solve the scene synthesis task by incorporating procedural modeling techniques [ 45 , 43 , 23 , 9 ] or by generating scene graphs with generative models [ 34 , 57 , 65 , 35 , 44 , 64 , 63 , 27 , 12 ] . Procedural modeling requires specifying a set of rules for the scene formation process , ∗Work done during Despoina ’ s internship at NVIDIA . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . but acquiring these rules is a time-consuming task , requiring skills of experienced artists . Similarly , graph-based approaches require scene graph annotations , which may be laborious to obtain . Another line of research utilizes CNN-based [ 58 , 47 ] and transformer-based [ 59 ] architectures to generate rooms by autoregressively selecting and placing objects in a scene , i.e . one after the other . These approaches represent scenes as ordered sequences of objects . Typically , the ordering is defined using the spatial arrangement of objects in a room ( e.g . left-to-right ) [ 22 ] or the object class frequency ( e.g . most to least probable ) [ 47 , 59 ] . Such orderings impose unnatural constraints on the scene generation process , inhibiting practical applications . For example , in [ 47 , 59 ] , which order objects by class frequency , the probability of a bed ( more common ) appearing after an ottoman ( less common ) in the training set is zero . As a result , these methods can not generate more common objects after less common objects , which makes them impractical for interactive tasks like general room completion and partial room re-arrangement , where input is unconstrained ( e.g . Fig.1B ) . To address these limitations , we pose scene synthesis as an unordered set generation problem and introduce ATISS , a novel autoregressive transformer architecture to model this process . Given a room type ( e.g . bedroom , living room ) and its shape , our model generates meaningful furniture arrangements by sequentially placing objects in a permutation-invariant fashion . We train ATISS to maximize the log-likelihood of all possible permutations of object arrangements in a collection of training scenes , labeled only with object classes and 3D bounding boxes , which are easier to obtain , than costly support relationship [ 57 ] or scene graph annotations [ 34 ] . Unlike existing works [ 58 , 47 , 59 ] , we propose the first model to perform scene synthesis as an autoregressive set generation task . ATISS is significantly simpler to implement and train , requires fewer parameters and is up to 8× faster at run-time than the fastest available baseline [ 59 ] . Furthermore , we demonstrate that our model generates more plausible object arrangments without any post-processing on the predicted layout . Our formulation allows applying a single trained model to automatic layout synthesis and to a number of interactive scenarios with versatile user input ( Fig.1 ) , such as automatic placement of user-provided objects , object suggestion with user-provided constraints , and room completion . Code and data are publicaly available at https : //nv-tlabs.github.io/ATISS . 2 Related Work . In this section , we discuss the most relevant literature on interior scene synthesis , as well as transformer architectures [ 56 ] in the context of generative modeling . Procedural Modeling with Grammars : Procedural modeling describes methods that recursively apply a set of functions for content synthesis . Grammars are a formal instantiation of this idea and have been used for modeling 3D structures such as plants [ 52 ] , buildings and cities [ 37 , 40 ] , indoor [ 45 ] and outdoor [ 43 ] scenes . [ 52 ] employed reversible-jump MCMC to control the output of stochastic context-free grammars . Meta-Sim [ 23 ] learned a model that modifies attributes of scene graphs sampled from a known probabilistic context-free grammar to match visual statistics between generated and real data . [ 9 ] extended this model to also learn to sample from the grammar , allowing context dependent relationships to be learnt . Concurrently , [ 44 ] employed Grammar-VAE [ 31 ] to generate scenes using a scene grammar generated from annotated data . In contrast , our model implicitly encapsulates inter-object relationships , without having to impose hand-crafted constraints . Graph-based Scene Synthesis : Representing scenes as graphs has been extensively studied in literature [ 34 , 57 , 65 , 35 , 44 , 64 , 63 , 27 , 12 ] . Zhou et al . [ 65 ] introduced a neural message passing algorithm for scene graphs that predicts the category of the next object to be placed at a specific location . Similarly , [ 34 , 64 , 44 , 35 ] utilized a VAE [ 28 ] to synthesize 3D scenes as parse trees [ 44 ] , adjacency matrices [ 64 ] , scene graphs [ 35 ] and scene hierarchies [ 34 ] . Concurrently , [ 57 , 63 ] adopted a two-stage generation process that disentangles planning the scene layout from instantiating the scene based on this plan . Note that graph-based models require supervision either in the form of relation graphs [ 57 , 63 , 35 ] or scene hierarchies [ 34 ] . In contrast , ATISS infers functional and spatial relations between objects directly from data labeled only with object classes and 3D bounding boxes . Autoregressive Scene Synthesis : Closely related to our work are autoregressive indoor scene generation models [ 58 , 47 , 59 ] . Ritchie et al . [ 47 ] introduced a CNN-based architecture that operates on a top-down image-based representation of a scene and inserts objects in it sequentially by predicting their category , location , orientation and size with separate network modules . [ 47 ] requires supervision in the form of 2D bounding boxes as well as auxiliary supervision such as depth maps and object segmentation masks . In concurrent work , Wang et al . [ 59 ] introduced SceneFormer , a series of transformers that autoregressively add objects in a scene similar to [ 47 ] . Both [ 47 , 59 ] use separate models to generate object attributes ( e.g . category , location ) that are trained independently and represent scenes as ordered sequences of objects , ordered by the category frequency . In contrast , we propose a simpler architecture that consists of a single model trained end-to-end to predict all attributes . We provide experimental evidence that our model generates more realistic object arrangements while being significantly faster . While [ 47 , 59 ] assume a fixed ordering of the objects in each scene , our model does not impose any constraint on the ordering of objects . Instead , during training , we enforce that our model generates objects with all orderings , in a permutation invariant fashion . This allows us to represent scenes as unordered sets of objects and perform various interactive tasks such as rearranging any object in a room or suggesting new objects given any room . Transformers for Set Generation : Transformer models [ 56 ] demonstrated impressive results on various tasks such as machine translation [ 50 , 39 ] , language-modeling [ 2 , 10 ] , object detection [ 33 , 3 , 66 ] , image recognition [ 14 , 53 ] , semantic segmentation [ 60 ] as well as on image [ 42 , 26 , 5 , 15 , 54 ] and music [ 11 ] generation tasks . While there are works [ 32 , 30 ] that utilize the permutation equivariance property of transformers for unordered set processing and prediction , existing generative models with transformers assume ordered sequences [ 2 , 5 , 6 ] even when there exists no natural order such as for pointclouds [ 38 ] and objects in a scene [ 59 ] . Instead , we introduce an autoregressive transformer for unordered set generation that enforces that the probability of adding a new element in the set is invariant to the order of the elements already in the set . We show that for the scene synthesis task , our model outperforms transformers that consider ordered sets of elements in every metric . 3 Method . Given an empty or a partially complete room of a specific type ( e.g . bedroom ) together with its shape , as a top-down orthographic projection of its floor , we want to learn a generative model that populates the room with objects , whose functional composition and spatial arrangement is plausible . To this end , we propose an autoregressive model that represents scenes as unordered sets of objects ( Sec . 3.1 ) and describe our implementation using a transformer network ( Sec . 3.2 ) . Finally , we analyse the training and inference details of our method ( Sec . 3.3 ) . 3.1 Autoregressive Set Generation . Let X = { X1 , . . . , XN } denote a collection of scenes where each Xi = ( Oi , Fi ) comprises the unordered set of objects in the sceneOi = { oij } Mj=1 and its floor layout Fi . To compute the likelihood of generating Oi we need to accumulate the likelihood of generating { oij } Mj=1 autoregressively in any order . This is formally written as pθ ( Oi|Fi ) = ∑ Ô∈π ( Oi ) ∏ j∈Ô pθ ( o i j | oi < j , Fi ) , ( 1 ) where pθ ( oij | oi < j , Fi ) is the probability of the j-th object , conditioned on the previously generated objects and the floor layout , and π ( · ) is a permutation function that computes the set of permutations of all objects in the scene . As a result , the log-likelihood of the whole collection X is log pθ ( X ) = N∑ i=1 log ∑ Ô∈π ( Oi ) ∏ j∈Ô pθ ( o i j | oi < j , Fi ) . ( 2 ) However , training our generative model to maximize the log-likelihood of ( 2 ) poses two problems : ( a ) the summation over all permutations is intractable and ( b ) ( 2 ) does not ensure that all orderings will have high probability . The second problem is crucial because we want our generative model to be able to complete any partial set in a plausible way , namely we want any generation order to have high probability . To this end , instead of maximizing ( 2 ) , we maximize the likelihood of generating a scene in all possible orderings , p̂θ ( · ) , which is defined as log p̂θ ( X ) = N∑ i=1 log ∏ Ô∈π ( Oi ) ∏ j∈Ô pθ ( o i j | oi < j , Fi ) = N∑ i=1 ∑ Ô∈π ( Oi ) ∑ j∈Ô log pθ ( o i j | oi < j , Fi ) . ( 3 ) Note that training our generative model with ( 3 ) allows us to approximate the summation over all permutations using Monte Carlo sampling thus solving both problems of ( 2 ) . Modelling Object Attributes : We represent objects in a scene as labeled 3D bounding boxes and model them with four random variables that describe their category , size , orientation and location , oj = { cj , sj , tj , rj } . The category cj is modeled using a categorical variable over the total number of object categories C in the dataset . For the size sj ∈ R3 , the location tj ∈ R3 and the orientation rj ∈ R1 , we follow [ 48 , 55 ] and model them with mixture of logistics distributions sj ∼ K∑ k=1 πsklogistic ( µ s k , σ s k ) tj ∼ K∑ k=1 πtklogistic ( µ t k , σ t k ) rj ∼ K∑ k=1 πrklogistic ( µ r k , σ r k ) ( 4 ) where πsk , µ s k and σ s k denote the weight , mean and variance of the k-th logistic distribution used for modeling the size . Similarly , πtk , µ t k and σ t k and π r k , µ r k ans σ r k refer to the weight , mean and variance of the k-th logistic of the location and orientation , respectively . In our setup , the orientation is the angle of rotation around the up vector and the location is the 3D centroid of the bounding box . Similar to prior work [ 47 , 59 ] , we predict the object attributes in an autoregressive manner : object category first , followed by position , orientation and size as follows : pθ ( oj | o < j , F ) = pθ ( cj |o < j , F ) pθ ( tj |cj , o < j , F ) pθ ( rj |cj , tj , o < j , F ) pθ ( sj |cj , tj , rj , o < j , F ) . ( 5 ) This is a natural choice , since we want our model to consider the object class before reasoning about the size and the position of an object . To avoid notation clutter , we omit the scene index i from ( 5 ) .

Summary: This paper proposes an transformer based framework for autoregressive indoor scene synthesis. A scene is modelled by an orthographic projection of the floor geometry (encoded by a CNN), and an unordered set of objects, characterized by category (encoded with a learned embedding), location, orientation and size (encoded with positional encoding). These attributes are then fed into a transformer encoder, which predicts a feature vector that is then decoded by a cascade of MLPs into attributes of the object to be predicted, where a categorical distribution is used for category, and a mixture of logistics is used for the remaining three attributes.   The proposed model is thus able to generate indoor scenes by autoregressively add the objects in arbitrary order. The scene synthesis results outperform relevant prior works convincingly with respect to multiple evaluation metrics. The paper also showcases a myriad of applications such as scene completion, failure detection and object suggestion, which are enabled by the design of the method.