ATISS: Autoregressive Transformers for Indoor Scene Synthesis
Year: 2021

Abstract:
The ability to synthesize realistic and diverse indoor furniture layouts automatically or based on partial input, unlocks many applications, from better interactive 3D tools to data synthesis for training and simulation. In this paper, we present ATISS, a novel autoregressive transformer architecture for creating diverse and plausible synthetic indoor environments, given only the room type and its floor plan. In contrast to prior work, which poses scene synthesis as sequence generation, our model generates rooms as unordered sets of objects. We argue that this formulation is more natural, as it makes ATISS generally useful beyond fully automatic room layout synthesis. For example, the same trained model can be used in interactive applications for general scene completion, partial room rearrangement with any objects specified by the user, as well as object suggestions for any partial room. To enable this, our model leverages the permutation equivariance of the transformer when conditioning on the partial scene, and is trained to be permutation-invariant across object orderings. Our model is trained end-to-end as an autoregressive generative model using only labeled 3D bounding boxes as supervision. Evaluations on four room types in the 3D-FRONT dataset demonstrate that our model consistently generates plausible room layouts that are more realistic than existing methods. In addition, it has fewer parameters, is simpler to implement and train and runs up to 8x faster than existing methods.

heading: 1 Introduction
text: Generating synthetic 3D content that is both realistic and diverse is a long-standing problem in computer vision and graphics. In the last decade, there has been increased demand for tools that automate the creation of 3D artificial environments for applications like video games and AR/VR, as well as general 3D content creation [61, 16, 36, 4, 62]. These tools can also synthesize data to train computer vision models, avoiding expensive and laborious annotations. Generative models [28, 19, 13, 29, 56] have demonstrated impressive results on synthesizing photorealistic images [7, 1, 24, 8, 25] and intelligible text [46, 2], and are beginning to be adopted for the generation of 3D environments.
Recent works proposed to solve the scene synthesis task by incorporating procedural modeling techniques [45, 43, 23, 9] or by generating scene graphs with generative models [34, 57, 65, 35, 44, 64, 63, 27, 12]. Procedural modeling requires specifying a set of rules for the scene formation process,
∗Work done during Despoina’s internship at NVIDIA.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
but acquiring these rules is a time-consuming task, requiring skills of experienced artists. Similarly, graph-based approaches require scene graph annotations, which may be laborious to obtain.
Another line of research utilizes CNN-based [58, 47] and transformer-based [59] architectures to generate rooms by autoregressively selecting and placing objects in a scene, i.e. one after the other. These approaches represent scenes as ordered sequences of objects. Typically, the ordering is defined using the spatial arrangement of objects in a room (e.g. left-to-right) [22] or the object class frequency (e.g. most to least probable) [47, 59]. Such orderings impose unnatural constraints on the scene generation process, inhibiting practical applications. For example, in [47, 59], which order objects by class frequency, the probability of a bed (more common) appearing after an ottoman (less common) in the training set is zero. As a result, these methods cannot generate more common objects after less common objects, which makes them impractical for interactive tasks like general room completion and partial room re-arrangement, where input is unconstrained (e.g. Fig.1B).
To address these limitations, we pose scene synthesis as an unordered set generation problem and introduce ATISS, a novel autoregressive transformer architecture to model this process. Given a room type (e.g. bedroom, living room) and its shape, our model generates meaningful furniture arrangements by sequentially placing objects in a permutation-invariant fashion. We train ATISS to maximize the log-likelihood of all possible permutations of object arrangements in a collection of training scenes, labeled only with object classes and 3D bounding boxes, which are easier to obtain, than costly support relationship [57] or scene graph annotations [34]. Unlike existing works [58, 47, 59], we propose the first model to perform scene synthesis as an autoregressive set generation task. ATISS is significantly simpler to implement and train, requires fewer parameters and is up to 8× faster at run-time than the fastest available baseline [59]. Furthermore, we demonstrate that our model generates more plausible object arrangments without any post-processing on the predicted layout. Our formulation allows applying a single trained model to automatic layout synthesis and to a number of interactive scenarios with versatile user input (Fig.1), such as automatic placement of user-provided objects, object suggestion with user-provided constraints, and room completion. Code and data are publicaly available at https://nv-tlabs.github.io/ATISS.

heading: 2 Related Work
text: In this section, we discuss the most relevant literature on interior scene synthesis, as well as transformer architectures [56] in the context of generative modeling.
Procedural Modeling with Grammars: Procedural modeling describes methods that recursively apply a set of functions for content synthesis. Grammars are a formal instantiation of this idea and have been used for modeling 3D structures such as plants [52], buildings and cities [37, 40], indoor [45] and outdoor [43] scenes. [52] employed reversible-jump MCMC to control the output of stochastic context-free grammars. Meta-Sim [23] learned a model that modifies attributes of scene graphs sampled from a known probabilistic context-free grammar to match visual statistics between generated and real data. [9] extended this model to also learn to sample from the grammar, allowing context dependent relationships to be learnt. Concurrently, [44] employed Grammar-VAE [31] to generate scenes using a scene grammar generated from annotated data. In contrast, our model implicitly encapsulates inter-object relationships, without having to impose hand-crafted constraints.
Graph-based Scene Synthesis: Representing scenes as graphs has been extensively studied in literature [34, 57, 65, 35, 44, 64, 63, 27, 12]. Zhou et al. [65] introduced a neural message passing algorithm for scene graphs that predicts the category of the next object to be placed at a specific location. Similarly, [34, 64, 44, 35] utilized a VAE [28] to synthesize 3D scenes as parse trees [44], adjacency matrices [64], scene graphs [35] and scene hierarchies [34]. Concurrently, [57, 63] adopted a two-stage generation process that disentangles planning the scene layout from instantiating the scene based on this plan. Note that graph-based models require supervision either in the form of
relation graphs [57, 63, 35] or scene hierarchies [34]. In contrast, ATISS infers functional and spatial relations between objects directly from data labeled only with object classes and 3D bounding boxes.
Autoregressive Scene Synthesis: Closely related to our work are autoregressive indoor scene generation models [58, 47, 59]. Ritchie et al. [47] introduced a CNN-based architecture that operates on a top-down image-based representation of a scene and inserts objects in it sequentially by predicting their category, location, orientation and size with separate network modules. [47] requires supervision in the form of 2D bounding boxes as well as auxiliary supervision such as depth maps and object segmentation masks. In concurrent work, Wang et al. [59] introduced SceneFormer, a series of transformers that autoregressively add objects in a scene similar to [47]. Both [47, 59] use separate models to generate object attributes (e.g. category, location) that are trained independently and represent scenes as ordered sequences of objects, ordered by the category frequency. In contrast, we propose a simpler architecture that consists of a single model trained end-to-end to predict all attributes. We provide experimental evidence that our model generates more realistic object arrangements while being significantly faster. While [47, 59] assume a fixed ordering of the objects in each scene, our model does not impose any constraint on the ordering of objects. Instead, during training, we enforce that our model generates objects with all orderings, in a permutation invariant fashion. This allows us to represent scenes as unordered sets of objects and perform various interactive tasks such as rearranging any object in a room or suggesting new objects given any room.
Transformers for Set Generation: Transformer models [56] demonstrated impressive results on various tasks such as machine translation [50, 39], language-modeling [2, 10], object detection [33, 3, 66], image recognition [14, 53], semantic segmentation [60] as well as on image [42, 26, 5, 15, 54] and music [11] generation tasks. While there are works [32, 30] that utilize the permutation equivariance property of transformers for unordered set processing and prediction, existing generative models with transformers assume ordered sequences [2, 5, 6] even when there exists no natural order such as for pointclouds [38] and objects in a scene [59]. Instead, we introduce an autoregressive transformer for unordered set generation that enforces that the probability of adding a new element in the set is invariant to the order of the elements already in the set. We show that for the scene synthesis task, our model outperforms transformers that consider ordered sets of elements in every metric.

heading: 3 Method
text: Given an empty or a partially complete room of a specific type (e.g. bedroom) together with its shape, as a top-down orthographic projection of its floor, we want to learn a generative model that populates the room with objects, whose functional composition and spatial arrangement is plausible. To this end, we propose an autoregressive model that represents scenes as unordered sets of objects (Sec. 3.1) and describe our implementation using a transformer network (Sec. 3.2). Finally, we analyse the training and inference details of our method (Sec. 3.3).

heading: 3.1 Autoregressive Set Generation
text: Let X = {X1, . . . ,XN} denote a collection of scenes where each Xi = ( Oi,Fi ) comprises the unordered set of objects in the sceneOi = {oij}Mj=1 and its floor layout Fi. To compute the likelihood
of generating Oi we need to accumulate the likelihood of generating {oij}Mj=1 autoregressively in any order. This is formally written as
pθ(Oi|Fi) = ∑
Ô∈π(Oi) ∏ j∈Ô pθ(o i j | oi<j ,Fi), (1)
where pθ(oij | oi<j ,Fi) is the probability of the j-th object, conditioned on the previously generated objects and the floor layout, and π(·) is a permutation function that computes the set of permutations of all objects in the scene. As a result, the log-likelihood of the whole collection X is
log pθ(X ) = N∑ i=1 log  ∑ Ô∈π(Oi) ∏ j∈Ô pθ(o i j | oi<j ,Fi)  . (2) However, training our generative model to maximize the log-likelihood of (2) poses two problems: (a) the summation over all permutations is intractable and (b) (2) does not ensure that all orderings will have high probability. The second problem is crucial because we want our generative model to be able to complete any partial set in a plausible way, namely we want any generation order to have high probability. To this end, instead of maximizing (2), we maximize the likelihood of generating a scene in all possible orderings, p̂θ(·), which is defined as
log p̂θ(X ) = N∑ i=1 log  ∏ Ô∈π(Oi) ∏ j∈Ô pθ(o i j | oi<j ,Fi)  = N∑ i=1 ∑ Ô∈π(Oi) ∑ j∈Ô log pθ(o i j | oi<j ,Fi).
(3) Note that training our generative model with (3) allows us to approximate the summation over all permutations using Monte Carlo sampling thus solving both problems of (2).
Modelling Object Attributes: We represent objects in a scene as labeled 3D bounding boxes and model them with four random variables that describe their category, size, orientation and location, oj = {cj , sj , tj , rj}. The category cj is modeled using a categorical variable over the total number of object categories C in the dataset. For the size sj ∈ R3, the location tj ∈ R3 and the orientation rj ∈ R1, we follow [48, 55] and model them with mixture of logistics distributions
sj ∼ K∑ k=1 πsklogistic(µ s k, σ s k) tj ∼ K∑ k=1 πtklogistic(µ t k, σ t k) rj ∼ K∑ k=1 πrklogistic(µ r k, σ r k) (4)
where πsk, µ s k and σ s k denote the weight, mean and variance of the k-th logistic distribution used for modeling the size. Similarly, πtk, µ t k and σ t k and π r k, µ r k ans σ r k refer to the weight, mean and variance of the k-th logistic of the location and orientation, respectively. In our setup, the orientation is the angle of rotation around the up vector and the location is the 3D centroid of the bounding box.
Similar to prior work [47, 59], we predict the object attributes in an autoregressive manner: object category first, followed by position, orientation and size as follows:
pθ(oj | o<j ,F) = pθ(cj |o<j ,F)pθ(tj |cj , o<j ,F)pθ(rj |cj , tj , o<j ,F)pθ(sj |cj , tj , rj , o<j ,F). (5) This is a natural choice, since we want our model to consider the object class before reasoning about the size and the position of an object. To avoid notation clutter, we omit the scene index i from (5).

heading: 3.2 Network Architecture
text: The input to our model is a collection of scenes in the form of 3D labeled bounding boxes with their corresponding room shape. Our network consists of four main components: (i) the layout encoder that maps the room shape to a global feature representation F, (ii) the structure encoder hθ that maps the M objects in the scene into per-object context embeddings C = {Cj}Mj=1, (iii) the transformer encoder τθ that takes F, C and a query embedding q and predicts the features q̂ for the next object to be generated and (iv) the attribute extractor that predicts the attributes of the next object. Our model is illustrated in Fig. 2. The layout encoder is simply a ResNet-18 [20] that extracts a feature representation F ∈ R64 for the top-down orthographic projection of the floor.
Structure Encoder: The structure encoder hθ maps the attributes of the j-th object into a per-object context embedding Cj as follows:
hθ : RC × R3 × R3 × R1 → RLc × RLs × RLt × RLr
(c, s, t, r) 7→ [λ(c); γ(s); γ(t); γ(r)] (6)
where Lc, Ls, Lt, Lr are the output dimensionalities of the embeddings used to map the category, the size, the location and the orientation into a higher dimensional space respectively and [· ; ·] denotes concatenation. For the object category cj we use a learnable embedding λ(·), whereas for the size sj , the position tj and the orientation rj , we use the positional encoding of [56] as follows
γ(p) = (sin(20πp), cos(20πp), . . . , sin(2L−1πp), cos(2L−1πp)) (7)
where p can be any of the size, position or orientation attributes and γ(·) is applied separately in each attribute’s dimension. The set of per-object context vectors synthesizes the context embedding C that encapsulates information for the existing objects in the scene and is used to condition the next object to be generated. Before passing the output of (6) to the transformer encoder, we map each Cj to 64 dimensions using a linear projection.
Transformer Encoder: We follow [56, 10] and implement our encoder τθ as a multi-head attention transformer without any positional encoding. This allows us to learn a parametric function that computes features that are invariant to the order of Cj in C. We use these features to predict the next object to be added in the scene, creating an autoregressive model. The input set of the transformer is
I = {F} ∪ {Cj}Mj=1 ∪ q, with M the number of objects in the scene. q ∈ R64 is a learnable object query vector that allows the transformer to predict output features q̂ ∈ R64 used for generating the next object to be added in the scene. The use of a query token is akin to the use of a mask embedding in Masked Language Modelling [10] or the class embedding for the Vision Transformer [49].
Attribute Extractor: We autoregressively predict the attributes of the next object to be added in the scene using one MLP for each attribute. More formally, the attribute extractor is defined as follows:
cθ : R64 → RC q̂ 7→ ĉ (8) tθ : R64 × RLc → R3×3×K (q̂, λ(c)) 7→ t̂ (9)
rθ : R64 × RLc × RLt → R1×3×K (q̂, λ(c), γ(t)) 7→ r̂ (10) sθ : R64 × RLc × RLt × RLr → R3×3×K (q̂, λ(c), γ(t), γ(r)) 7→ ŝ (11)
where ĉ, ŝ, t̂, r̂ are the predicted attribute distributions and cθ, tθ, rθ and sθ are mappings between the latent space and the low-dimensional space of attributes. For the object category, cθ predicts C class probabilities, whereas, tθ, rθ and sθ predict the mean, variance and mixing coefficient for the K logistic distributions for each attribute. To predict the object properties in an autoregressive manner, we need to condition the prediction of a property on the previously predicted properties. Thus, instead of only passing q̂ to each MLP, we concatenate it with the previously predicted attributes, mapped in a higher-dimensional space using the embeddings λ(·) and γ(·) from (6).

heading: 3.3 Training and Inference
text: During training, we choose a scene from the dataset and apply a random permutation π(·) on its M objects. Then, we randomly select the first T objects to compute the context embedding C.
Conditioned on C and F, our network predicts the attribute distributions of the next object to be added in the scene and is trained to maximize the log-likelihood of the T +1 object from the permuted scene. A pictorial representation of the training process is provided in Fig. 3. To indicate the end of sequence, we augment the C object classes with an additional class, which we refer to as end symbol.
During inference, we start with an empty context embedding C = ∅ and the floor representation F of the room to be populated and autoregressively sample attribute values from the predicted distributions of (8)-(11) for the next object. Once a new object is generated, it is appended to the context C to be used in the next step of the generation process until the end symbol is generated. A pictorial representation of the generation process can be found in Fig. 2. In order to transform the predicted labeled bounding boxes to 3D models we use object retrieval. In particular, we retrieve the closest object from the dataset in terms of the euclidean distance of the bounding box dimensions.

heading: 4 Experimental Evaluation
text: In this section, we provide an extensive evaluation of our method, comparing it to existing baselines. We further showcase several interactive use cases enabled by our method, not previously possible. Additional results as well as implementation details are provided in the supplementary.
Datasets: We train our model on the 3D-FRONT dataset [17] which contains a collection of 6, 813 houses with roughly 14, 629 rooms, populated with 3D furniture objects from the 3D-FUTURE dataset [18]. In our evaluation, we focus on four room types: (i) bedrooms, (ii) living rooms, (iii) dining rooms and (iv) libraries. After pre-processing to filter out uncommon object arrangements and rooms with unnatural sizes, we obtained 5996 bedrooms, 2962 living rooms, 2625 dining rooms and 622 libraries. We use 21 object categories for the bedrooms, 24 for the living and dining rooms and 25 for the libraries. The preprocessing steps are discussed in the supplementary.
Baselines: We compare our approach to FastSynth [47] and SceneFormer [59] using the authors’ implementations. Note that both approaches were originally evaluated on the SUNCG dataset [51], which is now unavailable. Thus, we retrained both on 3D-FRONT. We also compare with a variant of our model that generates scenes as ordered sequences of objects (Ours+Order). To incorporate the order information to the input, we utilize a positional embedding [56] and a fixed ordering based on the object frequency as described in [59].
Evaluation Metrics: To measure the realism of the generated scenes, we follow prior work [47] and report the KL divergence between the object category distributions of synthesized and real scenes from the test set and the classification accuracy of a classifier trained to discriminate real from synthetic scenes. We also report the FID [21] between top-down orthographic projections of synthesized and real scenes from the test set, which we compute using [41] on 2562 images. We repeat the metric computation for FID and classification accuracy 10 times and report the average.

heading: 4.1 Scene Synthesis
text: We start by evaluating the performance of our model on generating plausible object configurations for various room types, conditioned on different floor plans. Fig. 4 provides a qualitative comparison of four scenes generated with our model and baselines. In some cases, both [47, 59] generate invalid room layouts with objects positioned outside room boundaries or overlapping. Instead, our model consistently synthesizes realistic object arrangements. We validate this quantitatively in Tab. 1, where we compare the generated scenes wrt. their similarity to the original data from 3D-FRONT. Synthesized scenes sampled from our model are almost indistinguishable from scenes from the test set, as indicated by the classification accuracy in Tab. 1, which is consistently around 50%. Our model also achieves lower FID scores for all room types and generates category distributions that are more faithful to the category distributions of the test set, expressed as lower KL divergence.
To showcase that our model generates diverse object arrangements we visualize 3 generated scenes conditioned on the same floor plan for all methods (Fig. 5). We observe that our generated scenes are consistently valid and contain diverse object arrangements. In comparison [47, 59] struggle to generate plausible layouts particularly for the case of living rooms and dining rooms. We hypothesize that these rooms are more challenging than bedrooms, for the baselines, due to their significantly smaller volume of training data, and the large number of constituent objects per scene (20 on average, as opposed to 8). To investigate whether our model also generates plausible layouts conditioned on floor plans with uncommon shapes that are not in the training set, we manually design unconventional floor plans (Fig. 6) and generate bedroom layouts. While both [47, 59] fail to generate valid scenes, our model synthesizes diverse object layouts that are consistent with the floor plan. Finally, we compare the computational requirements of our architecture to [47, 59]. Our model is significantly faster (Tab. 2), while having fewer parameters (Tab. 3) than both [47, 59]. Note that [47] is orders of magnitude slower because it requires rendering every individual object added in the scene.

heading: 4.2 Applications
text: In this section, we present three applications that greatly benefit by our unordered set formulation and are crucial for creating an interactive scene synthesis tool.
Scene Completion: Starting from a partial scene, the task is to populate the empty space in a meaningful way. Since both [47, 59] are trained on sorted sequences of objects, they first generate frequent objects (e.g. beds, wardrobes) followed by less common objects. As a result, incomplete scenes that contain less common objects cannot be correctly populated. This is illustrated in Fig. 7, where [47, 59] either fail to add any objects in the scene or place furnitures in unatural positions, thus resulting in bedrooms without beds (see 1st row Fig. 7) and scenes with overlapping furniture (see 2nd row Fig. 7). In contrary, our model successfullly generates plausible completions with multiple objects such as lamps, wardrobes and dressing tables.
Failure Case Detection and Correction: We showcase that our model is able to identify and correct unnatural object arrangements. Given a scene, we compute the likelihood of each object, according to our model, conditioned on the other objects in the scene. We identify problematic objects as those with low likelihood and sample a new location from our generative model to rearrange it. We test our model in various scenarios such as overlapping objects, objects outside the room boundaries and objects in unnatural positions and show that it successfully identifies problematic objects (highlighted in green in Fig. 8) and rearranges them into a more plausible position. Note that this task cannot be performed by methods that consider ordering because they assign very low likelihood to common objects appearing after rare objects e.g. beds after cabinets.
Object Suggestion: We now test the ability of our model to provide object suggestions given a scene and user specified location constraints. To perform this task we sample objects from our generative model and accept the ones that fullfill the constraints provided by the user. Fig. 9 shows examples of location constraints (red box in top row) and the corresponding objects suggested (bottom row). Note that even when the user provided region is partially outside the room boundaries (4th, 5th column), suggested objects always reside in the room. Moreover, if the acceptable region overlaps with another object, our model suggests adding nothing (6th column). This task requires computing the likelihood of an object conditioned on an arbitrary scene, which [59, 47] cannot perform due to ordering.

heading: 4.3 Perceptual Study
text: We conducted two paired Amazon Mechanical Turk perceptual studies to evaluate the quality of our generated layouts against [47] and [59]. We sample 6 bedroom layouts for each method from the same 211 test set floor plans. Users saw 2 different rotating 3D scenes per method randomly selected from 6 pre-rendered layouts. Random layouts for each floor plan were assessed by 5 different workers to evaluate agreement and diversity across samples for a total of 1055 question sets per paired study. Generated scenes of [47] were judged to contain errors like interpenetrating furniture 41.4% of the time, nearly twice as frequently as our method, while [59] performs significantly worse (Tab. 4). Regarding realism, the scenes of [47] were more realistic than ours in only 26.9% of the cases. We conclude that our method outperforms the baselines in the key metric, generation of realistic indoor scenes, by a large margin. Additional details are provided in the supplementary.

heading: 5 Conclusion
text: We introduced ATISS, a novel autoregressive transformer architecture for synthesizing 3D rooms as unordered sets of objects. Our method generates realistic scenes that advance the state-of-the-art for scene synthesis. In addition, our novel formulation enables new interactive applications for semi-automated scene authoring, such as general scene completion, object suggestions, anomaly
detection and more. We believe that our model is an important step not only toward automating the generation of 3D environments, with impact on simulation and virtual testing, but also toward a new generation of tools for user-driven content generation. By accepting a wide range of user inputs, our model mitigates societal risks of task automation, and promises to usher in tools that enhance the workflow of skilled laborers, rather than replacing them. In future work, we plan to extend order invariance to object attributes to further expand interactive possibilities of this model, and to incorporate style information. As any machine learning model, our model can introduce learned biases for indoor scenes, and we plan to investigate learning from less structured and more widely available data sources to make this model applicable to a wider range of cultures and environments.

Review summaries:
This paper proposes an transformer based framework for autoregressive indoor scene synthesis. A scene is modelled by an orthographic projection of the floor geometry (encoded by a CNN), and an unordered set of objects, characterized by category (encoded with a learned embedding), location, orientation and size (encoded with positional encoding). These attributes are then fed into a transformer encoder, which predicts a feature vector that is then decoded by a cascade of MLPs into attributes of the object to be predicted, where a categorical distribution is used for category, and a mixture of logistics is used for the remaining three attributes.   The proposed model is thus able to generate indoor scenes by autoregressively add the objects in arbitrary order. The scene synthesis results outperform relevant prior works convincingly with respect to multiple evaluation metrics. The paper also showcases a myriad of applications such as scene completion, failure detection and object suggestion, which are enabled by the design of the method.

The paper proposes an approach for 3D scene synthesis based on a transformer architecture.  The approach encodes 3D scenes from a top-down floorplan image and a set of per-object learned embeddings that capture the category, size, location, and orientation of each object.  Critically, the formulation addresses the permutation-invariant nature of object sets in 3D scenes, in contrast with prior work which relied on generation of scenes through an ordered sequence of object insertions.  The proposed approach is demonstrated on three tasks: 3D scene synthesis "from scratch", scene completion (i.e. conditioning synthesis on a partial scene), suggestion of objects to place at a particular location in a scene, and detection of implausibly placed objects.  The experiments contrast the proposed approach against baselines from prior work, as well as an ablation that requires object ordering.  Scenes generated using the approach are shown to be of higher quality (as measured by FID, classification, and KL divergence metrics) relative to scenes generated using the baseline methods.  Additional analysis experiments show that the approach also generalizes well to out-of-domain floorplan shapes, and that it is significantly faster than prior work.  Finally, a perceptual study carried out on Amazon Mechanical Turk shows that scenes generated by the approach are deemed to contain significantly fewer object placement errors and to be more realistic that scenes generated by the baselines.

The submission proposes a permutation-invariant autoregressive generative model based on transformers for populating 3D rooms with furniture objects.  The model is trained with labeled 3D bounding boxes and is trained and evaluated using the 3D-FRONT dataset.  The proposed model predicts attributes (category, size, location - centroid of bounding box, and orientation - rotation around up axis) for each object in an autoregressive manner using a transformer-based model until stop is predicted.  The category is modeled using a categorical distribution, and the other attributes are modeled using a mixture of logistic distributions.  To model the objects in a scene and a unordered set, the model does not encode the position of the object in the sequence.  To allow the model to handle different permutations of objects, the model is trained with different permutations.  The output prediction of the model is effectively a set of oriented bounding boxes with categories, which are then replaced with an appropriate CAD model with a matching category and closest in size.  Experiments comparing the proposed model against prior work (FastSynth and SceneFormer, both retrained on the 3D-FRONT dataset) shows the proposed model can produce higher quality furniture layouts given a room outline and room type.  There are also qualitative examples of the model used for scene completion, identifying and correcting poor object placement, and suggest objects to place at a given location.  Ablations on the number of logistic distributions, and layout encoder is provided in the supplemental.  Main contributions - Transformer-based model for indoor scene synthesis that treats the objects as unordered sets during training, and autoregressively generate each object with their attributes - Set of experiments comparing the proposed model against prior work