ATISS: Autoregressive Transformers for Indoor Scene Synthesis
Authors: Despoina Paschalidou
Amlan Kar
Maria Shugrina
Karsten Kreis
Andreas Geiger
Sanja Fidler
Venue: NeurIPS 2021 Poster (2021)
Keywords: Indoor Scene Synthesis
Layout Generation
Autoregressive Set Generation
Generative Models

Abstract: The ability to synthesize realistic and diverse indoor furniture layouts automatically or based on partial input, unlocks many applications, from better interactive 3D tools to data synthesis for training and simulation. In this paper, we present ATISS, a novel autoregressive transformer architecture for creating diverse and plausible synthetic indoor environments, given only the room type and its floor plan. In contrast to prior work, which poses scene synthesis as sequence generation, our model generates rooms as unordered sets of objects. We argue that this formulation is more natural, as it makes ATISS generally useful beyond fully automatic room layout synthesis. For example, the same trained model can be used in interactive applications for general scene completion, partial room re-arrangement with any objects specified by the user, as well as object suggestions for any partial room. To enable this, our model leverages the permutation equivariance of the transformer when conditioning on the partial scene, and is trained to be permutation-invariant across object orderings. Our model is trained end-to-end as an autoregressive generative model using only labeled 3D bounding boxes as supervision. Evaluations on four room types in the 3D-FRONT dataset demonstrate that our model consistently generates plausible room layouts that are more realistic than existing methods.
In addition, it has fewer parameters, is simpler to implement and train and runs up to 8 times faster than existing methods.

ATISS: Autoregressive Transformers for Indoor Scene Synthesis
Despoina Paschalidou 11,44 1,,3,
Amlan K5r4
Maria Shugrina
Karsten Kreis*
Andreas G.i
Sanja Fidler
Max Planck Institute for Intelligent Systems Tübingen University of Tübingen
Max Planck ETH Center for Learning Systems
NVIDIA University of Toronto Vectoo Institute
(firstname. wwstaaaeeetuueeee.....a.....aaaa www.uue........... mpg. famlank, mshugrina, kkreis, sfidler)dnvidia.com
Abstract
The ability synthesize realistic and diverse indoor furniture layouts automatically or based on partial input, unlocks many applications, from better interactive 3D tools to data synthesis for training and simulation. In this paper, we present ATISS, a novel autoregressive transformer architecture for creating diverse and plausible synthetic indoor environments, given only the room type and its floor plan. In contrast to prior work, which poses scene synthesis as sequence generation, our model generates rooms as unordered formulation is more natural, it makes ATISS generally useful beyond fully automatic room layout synthesis. For example, the same trained model can be used in interactive applications for general scene completion, partial room re- arrangement with any objects specified by the user, as well as object suggestions for any partial room. To enable this, our model leverages the permutation equivariance of the transformer when conditioning on the partial scene, and is trained to be permutation invaarian across object orderings. Our model is trained end-to-end as an autoregressive generative model using only labeled 3D bounding boxes as supervision. Evaluations four room types the 3D-FRB FD-FRONT dataset demonstrate that our model consistently generates plausible room layouts that are more realistic than existing methods. In addition, has fewer parameters, is simpler implement and train and runs up to 8x faster than existing methods.
of objects. We argue that this
Introduction
Generating synthetic 3D content that is both realistic and diverse is a long-standing problem in computer vision and graphics. In last decade, there has been increased demand for tools that automate the creation of 3D artificial environments for applications like video games and AR/VR. as well as general 3D content creation [61, 16, 36, 4, 62]. These tools can also synthesize data to train computer vision models, avoiding expensive and laborious annotations. Generative models [28, 19, 13, 29, 56] have demonstrated impressive results on synthesizing photorealistic images [7, 2,, 8, 25] and intelligible text [46, 2), and are beginning to be adopted for the generation of 3D) environments.
Recent works proposed to solve the scene synthesis task by incorporating procedural modeling techniques [45, 43, 23, 9] or by generating scene graphs with generative models [34, 57, 65, 35, 44, 64, 63, 27, 12]. Procedural modeling requires specifying a set of rules for the scene formation process,
Work done during Despoina's internship at NVIDIA.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

AT
ATISS
general
object
synthesis
completion
suggestion
Figure 1: Motivation In addition to fully automatic layout synthesis (A), our formulation in terms of unordered sets of objects allows our model to be used for novel interactive applications with versatile user control: scene completion given any number of existing furniture pieces of any class pinned to specific location by the user (B), and object suggestions with user-provided constraints (object centroid constraint shown in red) (C).
but acquiring these rules is a time-consuming task, requiring skills of experienced artists Similarly, graph-based approaches require scene graph annotations, which may be laborious obtain.
Another line of research utilizes CNN- [58, 47] and transformer- based [59] architectures to generate rooms by autoregressively selecting and placing objects in scene, i.e. one after the other. These approaches represent scenes as ordered sequences of objects. Typically, the ordering is defined using the spatial arrangement of objects in a room (e.g. left-to-right) [22] or the object class frequency (e.g. most to least probable) [47, 59]. Such orderings impose unnatural constraints on the scene generation process, inhibiting practical applications. For example, in [47, 59], which order objects by class frequency, the probability of a bed (more common) appearing after an ottoman (less common) in the training set is zero. As result, these methods cannot generate more common objects after less common objects, which makes them impractical for interactive tasks like general room completion and partial room re-arrangement, where input is unconstrained (e.g. IB).
To address these limitations, we pose scene synthesis as an unordered set generation problem and introduce ATISS, a novel autoregressive transformer architecture to model this process, Given room type (e.g. bedroom, living room) and its shape, our model generates meaningful furniture arrangements by sequentially placing objects in a corrmutatiooninvariinn to maximize the lig-liieldhood of all possible permutations of object arrangements in a collection of training scenes, labeled only with object classes and 3D bounding boxes, which easier to obtain, than costly support relationship [57] or scene graph annotations [34]. Unlike existing works [58, 47, 59], we propose the first model to perform scene synthesis as autoregressive set generation task. ATISS is significantly simpler to implement and train, requires fewer parameters and up to 8x faster run-time than the fastest available baseline [59]. Furthermore, we demonstrate that our model generates more plausible object arrangments without any post-processing on the predicted layout. Our formulation allows applying a single trained model automatic layout synthesis and to a number of interactive scenarios with versatile user input (Fig. 1), such as automatic placement of user-provided objects, object suggestion with user-provided constraints, and room completion. Code and data are publicaly available at https://nv-tlabss http:///wwllbb..iiiubbbooobo www.ggb.ittt.iiis....
fashion. We train ATISS
2 Related Work
In this section, discuss the most relevant literature on interior scene synthesis, as well as trans- former architectures [56] in the context of generative modeling.
Procedural Modeling with Grammars: Procedural modeling describes methods that recursively apply set of functions for content synthesis. Grammars are formal instantiation of this idea and have been used for modeling 3D structures such as plants [52], buildings and cities [37, 40], indoor [45] and outdoor [43] scenes. [52] employed reversible-jum MCMC to control the output of stochastic context-free grammars. Meta-Sim [23] learned model that modifies attributes of scene graphs sampled from a known probabilistic context-free grammar to match visual statisties between generated and real data. [9] extended this model to also learn to sample from the grammar, allowing context dependent relationships to be learnt. Concurrently, [44] employed Grammar-VAL [3 to generate scenes using scene grammar generated from annotated data. In contrast, our model implicitly encapsulates inter-object relationships, without having to impose hand-crafted constraints.
Graph-based Scene Synthesis: Representing scenes as graphs has been extensively studied in literature [34, 57, 65, 35, 44, 64, 63, 27, 12]. Zhou et al. [65] introduced a neural message passing algorithm for scene graphs that predicts the category of the next object to be placed at a specific location. Similarly, [34, 64, 44, 35] utilized a VAE [28] to synthesize 3D scenes as parse trees [44], adjacency matrices [64], scene graphs [35] and scene hierarchies [34]. Concurrently, [57, 63) adopted a two-stage generation process that disentangles planning the scene layout from instantiating the scene based on this plan. Note that graph- based models require supervision either in the form of

---

Atlaloute
Layort Basnde
Rrcoder
Objjct
TraanSoorroe
mpf
Object
:
Cyr
Object
Method Overview. Starting from scene with M objects and a floor layout, the layout encoder maps the floor into feature representation F and the structure encoder maps the objects into context embedding context embedding and a learnable query vector q are then passed to the transformer encoder that predicts q. Using q the attribute extractor autoregressively predicts the attribute distributions that are used to sample the attributes for the next object to be generated.
Figure 2:
MM [C The floor layout feature
=
relation graphs [57, 63, 35] or scene hierarchies [34]. In contrast, ATISS infers functional and spatial relations between objects directly from data labeled only with object classes and 3D bounding boxes.
work are autoregressive indoor scene generation models [58, 47, 59]. Ritchie et al. [47] introduced CNN-based architecture that operates on a top-down image-based representation of a scene and inserts objects in sequentially by predicting their category, location, orientation and size with separate network modules. [47] requires supervision in the form of 2D bounding boxes well auxiliary supervision such as depth maps and object segmentation masks. In concurrent work, Wang et al. [59] introduced SceneFormer, series of transformers that autoregressively add objects in scene similar to [47]. Both [47, 59] use separate models to generate object attributes (e.g. category, location) that are trained independently and represent scenes as ordered sequences of objects, ordered by the category frequency. In contrast, we propose simpler architecture that consists of single model trained end-to-end to predict all attributes. We provide experimental evidence that our model generates more realistie object arrangements while being significantly faster. While [47, 59] assume fixed ordering of the objects in each scene, our model does not impose any constraint on the ordering of objects. Instead, during training, we enforce that our model generates objects with all orderings, in a permutation invariant fashion. This allows us to represent scenes as unordered sets of objects and perform various interactive tasks such rearranging any object in
Autoregressive Scene Synthesis: Closely related
suggesting
objects given any room.
Transformers for Set Generation: Transformer models [56] demonstrated impressive results on various tasks such as machine translation [50, 39], language-modeling [2, 10], object detection [33, 66), image recognition [14, 53], semantic segmentation [60] as well as on image [42, 26, 15,54) 544 and music [11] generation tasks. While there are works [32, 30] that utilize the permutation cquivariance property of transformers for unordered set processing and prediction, cxisting generative models with transformers assume ordered sequences [2, 5, 6] even when there exists natural order such as for pointclouds [38] and objects transformer for unordered set generation that enforces that the probability of adding
scene [59]. Instead, we introduce an autoregressive
element in the set invariant to the order the elements already in the set. We show that for the scene synthesis task, our model outperforms transformers that consider ordered sets of elements in every metric.
3 Method
Given an empty or partially complete room of a specific type (e.g. bedroom) together with its shape, top-down orthographic projection of its floor, we want to learn generative model that populates the room with objects, whose functional composition and spatial arrangement is plausible. To this end. we propose an autoregressive model that represents scenes as unordered sets of objects (Sec. 3.1) and describe our implementation using transformer network (Sec. 3.2). Finally, we analyse the training and inference details of our method (Sec. 3.3).
3.1 Autoregressive Set Generation
Let X = {Xi.... X' denote a collection of scenes where each X; (O,, comprises the unordered set of objects in the scene O {or M.M. and its floor layout F'. To compute the likelihood

---

of generating O we need to accumulate the likelihood of generating 10 autoregressively in any order. This is formally written as
po (O, F*)
Σ
Il Po(oo 1o2y, F'),
ŌEE(O
where po(o) | o22y F') is the probability of the j-th object, conditioned the previously generated objects and the floor layout, and x()) is permutation function that computes the set of permutations of all objects in the scene. As a result, the log- lig-liiel the whole collection X' is
N
(X) = Σ log
Σ (EE
P
polo) I lee (o,
(2)
2=1
However, training our generative model to maximize the log- log-likelihood of (2) poses two problems: (a) the summation over all permutations is intractable and (b) (2) does not ensure that all orderings will have high probability. The second problem is crucial because we want our generative model to be able to complete any partial set in plausible way, namely we want any generation order have high probability. To this end, instead of maximizing (2), we maximize the likelihood of generating scene all possible orderings, (.).), which defined as
N
N
log Po (2) = Σ log
II
II Palo) o22.. F1)
log Po ozzz F'). (EE)
Σ
Σ
(E=(OO)) JEO
(3)
Note that training our generative model with (3) allows us to approximate the summation over all permutations using Monte Carlo sampling thus solving both problems of (2).
Modelling Object Attributes: We represent objects in a scene as labeled 3D bounding boxes and model them with four random variables that describe their category, size, orientation and location, Oj Sc, ty, The category Cj is modeled using a categorical variable over the total number of object categories C' in the dataset. For the size s, E R², the location tú E IR$ and the orientation ry E IR_, we follow [48, 55] and model them with mixture of logistics distributions
K
K
Sj TElogistic(H&, of) tj #logistti(uft.of) of) I2
n #flogisti((u&.of) (4)
where Ti. 1g and of denote the weight, mean and variance of the k-th logistic distribution used for modeling the size. Similarly, Tf. 1 and of and The /4 ans of refer to the weight, mean and variance of the k-th logistic of the location and orientation, respectively. angle of rotation around the up vector and the location is the 3D centroid of the bounding box.
our setup, the orientation the
Similar to prior work [47, 59], predict the object attributes in autoregressive manner: object category first, followed by position, orientation and size as follows:
pe (o, ozj , F) = pe(c, 0zz F) (t, cy,, ozj, F) (r, ty, ozj. F)po ry, ozソ F).
(5)
This is natural choice, since we want our model to consider the object class before reasoning about the size and the position of object. To avoid notation clutter, omit the scene index í from (5).
3.2 Network Architecture
The input to our model is collection of scenes in the form of 3D labeled bounding boxes with their corresponding room shape. Our network consists of four main components: (i) the layout encoder that maps the room shape to global feature representation F, (ii) the structure encoder he that maps the M objects in the scene into per-object context embeddings = {C (iii) the transformer encoder Ta that takes F, C and a query embedding q and predicts the features q for the next object to be generated and (iv) the altribute extractor that predicts the attributes of the next object. Our model illustrated Fig. The layout encoder is simply a ResNet- 18 [20] that extracts a feature representation F e 61 for the top-down orthographic projection of the floor.

---

Wh Li hii Iunbi e
F
ATTSS
NLL(Cs (LL +NLL(Ss, Pa)
01
03
04
05
01 04
02
tc.
+NLL( ( +NLL(tt, Ida.)
Figure 3: Training Overview: Given scene with M objects (coloured squares), we first randomly permute them and then keep the first T' objects (here T 3). We task our network to predict the next object in the scene given the subset of kept objects (highlighted with grey) and its floor layout feature F. Our loss function is the negative log-likelihood (NIL) of the next object in the permuted sequence (green square).
added
Structure Encoder: The structure encoder he maps the attributes of the j-th object into a per-object context embedding Cj as follows:
R x R R R x R x R ha IR x
R
x
(6)
(c,s, s,s,t,)
(t))
where Le, L,, Lt, Ly are the output dimensionalities of the embeddings used to map the category, the size, the location and the orientation into a higher dimensional space respectively and [-; denotes concatenation. For the object category Cj we use a learnable embedding A(.). whereas for the size sj. the position ty and the orientation F, use the positional encoding of [56] as follows
T(p) = (sin(2 Tp), coss rp)...., sin (pp) cos(22 mp))
(7)
where can be any of the size, position or orientation attributes and 7(-) is applied separately in each attribute's 's dimension. The set of per-object context vectors synthesizes the context embedding C that encapsulates information for the existing objects in the scene and is used to condition the next object
be generated. Before passing the output of (6) to the transformer encoder, we map each C, to 64 dimensions using linear projection.
We follow [56, 10] and implement encoder To multi-head attention transformer without any positional encoding. This allows us to learn a parametric function that computes features that are invariant the order of Cj in C. We these features to predict the next object to be added in the scene, creating an autoregressive model. The input set of the transformer is
Transformer Encoder:
[F) (C,) 1 U.q with M the number of objects in the scene. q R is learnable object query vector that allows the transformer predict output features q 1864 used for generating the next object to be added in the scene. The use of a query token is akin to the use of a mask embedding in Masked Language Modelling [10] or the class embedding for the Vision Transformer [49].
Attribute Extractor: We autoregressively predict the attributes of the next object to be added in the scene using MLP for each attribute. More formally, the attribute extractor is defined as follows:
Cy : IR - IRC
q +
(8)
to 64 x IR Le - 3x3x
(()) +
(9)
IR 64 x R Lo x R - RR. 1x3xx
(G)) +
(10)
8g RR64 L
R x R - R 3
(q. (t)) S
(11)
where ê, i, i are the predicted attribute distributions and co. lo. and 8p are mappings between the latent space and the low-dimensional space of attributes. For the object category, Co predicts C class probabilities, whereas, (g. l'a and 88 predict the mean, variance and mixing coefficient for the K logistic distributions for each attribute. To predict the object properties in an autoregressive manner, we need to condition the prediction of property on the previously predicted properties. Thus, instead of only passing q to each MLP, we concatenate it with the previously predicted attributes, mapped in nigher-dimensionn space using the embeddings X(-) and 7(-) from (6).
3.3 Training and Inference
During training, we choose scene from the dataset and apply a random permutation T(-) on its M objects. Then, randomly select the first T objects to compute the context embedding C.
5

---

Scene Layout
Training Sample
FastSynth
SceneFormer
Ours
Figure Qualitative Scene Synthesis Results. Synthesized scenes for room types: bedrooms (Ist+22nd row), living room (3rd row), dining room (4th row) using FastSynth, SceneFormer and our method. To showcase the generalization abilities of our model we also show the closest scene from the training set (2nd column).
Conditioned on C and our network predicts the attribute distributions of the next object to be added in the scene and is trained to maximize the log- likelihood of the T T+1 object from the permuted scene. pictorial representation the training process provided in Fig. 3. To indicate the end of sequence, we augment the C' object classes with an additional class, which we refer to as end symbol.
During inference, start with an empty context embedding C and the floor representation F of the room to be populated and autoregressively sample attribute values from the predicted distributions of (8)-(1)) for the next object. Once new object is generated, it is appended to the context C to be used in the next step of the generation process until the end symbol is generated. A pictorial be found in Fig. 2. In order to transform the predicted
representation of the generation process labeled bounding boxes to 3D models we use object retrieval. In particular, we retrieve the closest object from the dataset in terms of the euclidean distance of the bounding box dimensions.
Experimental Evaluation
In this section, we provide an extensive evaluation of our method, comparing to existing baselines. enabled by method, not previously possible.
We further showease several interactive Additional results as well as implementation details are provided in the supplementary.
Datasets: We train our model the 3D-FRONT dataset [17] which contains a collection of 6, 813 houses with roughly 14, 629 rooms, populated with 3D furniture objects from 3D-FUTURE dataset [18]. In our evaluation, we focus on four room types: (i) bedrooms, (ii) living rooms, (iii) dining rooms and (iv) libraries. After pre-processing to filter out uncommon object arrangements and rooms with unnatural sizes, obtained 5996 bedrooms, 2962 living rooms, 2625 dining rooms and 622 libraries. We use 21 object categories for the bedrooms, 24 for the living and dining rooms and 25 for the libraries. The preprocessing steps discussed in the supplementary.
Baselines: We compare our approach FastSynth [47] and SceneFormer [59] using the authors' implementations. Note that both approaches were originally evaluated on the SUNCG dataset [51], which is now unavailable. Thus, we retrained both on 3D-FRONT. We also compare with a variant of our model that generates scenes as ordered sequences of objects (Ours+Order). To incorporate the order information to the input, we utilize positional embedding [56] and fixed ordering based on the object frequency described in [59].

---

FastSynth
Scene Former
Ours
Figure 5: Scene Diversity. We show three generated scenes conditioned on three different floor plans for bedrooms and dining rooms. Every triplet of columns corresponds to different floor plan.
Secne Clasalication
Culegory Soetellormr
FastSynth
SceneCurmer
Ours+Order
Ours
F'sttSynhh
SconeFoomer
Ouzs+Order
Ours
PassSSync
Ouns+Order
Ours
Bedrooms
40.89
43,17
38.67
38,39
0.883
0.945
0.760
0,562
0,0064
0,0052
0.0533
0,0085
Living 61.07
99.54
35.37
33.14
0.945
0.972
0.694
0.516
0.0176
0,0313
0.0372
0.0834
Dining 55.83
67.04
35.79
29.25
0.915
0.941
0.623
0.477
0,0518
0,0368
0.0278
0.0061
Librar
57.72
55,34
35.60
35.24
0.815
0,850
0,512
0.521
0,1431
0.0232
0.0183
0.0098
Table 1: Quantitative Comparison. We report the FID score (L) at 256 pixels, the KI. divergence (L) between the distribution of object categories of synthesized and real scenes and the real vs. synthetic classification accuracy for all methods. Classification accuracy closer to 0.5 is better.
Evaluation Metries: To measure the realism the generated scenes, we follow prior work [47] and report the KI divergence between the object category distributions of synthesized and real scenes from the test set and the classification accuracy of classifier trained to discriminate real from synthetic scenes. We also report the FID [21] between top-down orthographic projections of synthesized and real scenes from the test set, which we compute using [41] on 256² images. We repeat the metric computation for FID and classification accuracy 10 times and report the average.
4.1 Scene Synthesis
We start by evaluating the performance of our model generating plausible object configurations for various room types, conditioned on different floor plans. Fig. 4 provides a qualitative comparison of four scenes generated with our model and baselines. In some cases, both [47, 59] generate invalid room layouts with objects positioned outside room boundaries or overlapping. Instead, our model consistently synthesizes realistic object arrangements. We validate this quantitatively in Tab. where we compare the generated scenes wrt. their similarity to the original data from 3D-FRONT. Synthesized scenes sampled from our model are almost indistinguishable from scenes from the test set, as indicated by the classification accuracy in Tab. 1, which is consistently around 50%. Our model also achieves lower FID scores for all room types and generates category distributions that are more faithful to the category distributions of the test set, expressed as lower KL divergence.
Scene Layout FastSynth
SceneFormer
Ours
Scene Layout
FastSynth
SceneFormer
Ours
Figure 6: Generalization Beyond Training Data. We show four synthesized bedrooms conditioned on four room layouts that we manually designed.

---

Partial Scene
FastSynth
SceneFormer
Ours+Order
Ours
Figure 7: Scene Completion. Given partial scene (left column), we visualize scene completions using our model and our baselines. Our model consistently generates plausible layouts.
Bedroom
Living
Dining
Library
FastSynth [47]
SceneFormer [59]
Ours
FastSyuth [47]
13193.77
30578.54
26596.08
10813.87
38.180
129.298
36.053
SceneFormer [59]
849.37
731.84
9U1. 17
369.74
Ours [47]
102.38
201.59
201.84
88.24
Table 3: Network Parameters Comparison. We
Table 2: Generation Time Comparison. We measure time (ms) to generate a scene, conditioned on a floor plan.
report the number of network parameters in millions.
To showcase that model generates diverse object arrangements visualize 3 generated scenes conditioned on the same floor plan for all methods (Fig. 5). We observe that our generated scenes consistently valid and contain diverse object arrangements. In comparison [47, 59] struggle to generate plausible layouts particularly for the case of living rooms and dining rooms. We hypothesize that these rooms are more challenging than bedrooms, for the baselines, due their significantly smaller volume of training data, and the large number of constituent objects per scene (20 on average, as opposed to 8). To investigate whether our model also generates plausible layouts conditioned on floor plans with uncommon shapes that are not in the training set, we manually design unconventional floor plans (Fig. 6) and generate bedroom layouts. While both [47, 59] fail to generate valid scenes, our model synthesizes diverse object layouts that are consistent with the floor plan. Finally, we compare the computational requirements of our architecture [47, 59]. Our model significantly faster (Tab. 2), while having fewer parameters (Tab. 3) than both [47,59]. 591. Note that [47] is orders of magnitude slower because it requires rendering every individual object added in the scene.
4.2 Applications
In this section, we present three applications that greatly benefit by our unordered set formulation and are crucial for creating interactive scene synthesis tool.
Scene Completion: Starting from partial scene, the task is populate the empty space in meaningful way. Since both 59] are trained on sorted sequences of objects, they first generate frequent objects (e.g. beds, wardrobes) followed by less common objects. As result, incomplete scenes that contain less common objects cannot be correctly populated. This is illustrated in Fig. 7, where [47, 59] either fail to add any objects in the scene or place furnitures in unatural positions, thus resulting in bedrooms without beds (see 1st row Fig. 7) and scenes with overlapping furniture (see 2nd row Fig. 7). contrary, our model successfullly generates plausible completions with multiple objects such as lamps, wardrobes and dressing tables.
Figure 8: Failure Case Detection and Correction. We use partial room with unnatural object arrangements. Our model identifies the problematic objects (first row, in green) and relocates them into meaningful positions.

---

Lamp
Double Bed
Cabinet
TV-stand
Wardrobe
Nothing
Figure Object Suggestion. user specifies region acceptable positions to place an object (marked as red boxes, 1st row) and our model suggests suitable objects (2nd row) to be placed in this location.
Failure Case Detection and Correction: We showcase that our model is able to identify and correct unnatural object arrangements. Given scene, we compute the likelihood of each object, according to our model, conditioned on the other objects in the scene. We identify problematic objects as those with low likelihood and sample new location from our generative model to rearrange it. We test our model various scenarios such as overlapping objects, objects outside the room boundaries and objects in unnatural positions and show that it successfully identifies problematic objects (highlighted in green in Fig. 8) and rearranges them into a more plausible position. Note that this task cannot be performed by methods that consider ordering because they assign very low likelihood to common objects appearing after rare objects e.g. beds after cabinets.
Object Suggestion: We now test the ability of our model to provide object suggestions given a scene and user specified location constraints. To perform this task we sample objects from our generative model and accept the ones that fullfill the constraints provided by the user. Fig. 9 shows examples of location constraints (red box top row) and the corresponding objects suggested (bottom row). Note that even when the user provided region is partially outside the room boundaries (4th, 5th column). suggested objects always reside in the room. Moreover, if the acceptable region overlaps with another object, our model suggests adding nothing (6th column). This task requires computing the likelihood of an object conditioned on an arbitrary scene, which [59, 47] cannot perform due to ordering.
4.3 Perceptual Study
We conducted two paired Amazon Mechanical Turk perceptual studies to evaluate the quality of our generated layouts against [47] and [59]. We sample 6 bedroom layouts for each method from the same 211 test set floor plans. Users saw different rotating 3D scenes per method randomly selected from 6 pre-rendered layouts. Random layouts for each floor plan were assessed by 5 different workers to evaluate agreement and diversity across samples for total of 1055 question per paired study. Generated scenes of [47] were judged to contain errors like interpenetrating furniture 41.4% of the time, nearly twice frequently
method, while [59] performs significantly worse (Tab. 4). Regarding realism, the scenes of [47] were more realistic than ours in only 26.9% of the cases. We conclude that method outperforms the baselines in the key metric, generation of realistic indoor scenes, by a large margin. Additional details are provided in the supplementary.
More Realistic
Method
Condition
Frequency
CI 99%
FastSyuth (47)
0.414
0.269
0.235, 0.306
SceneFoemer [59] vs. Ours
0.713
0.165
0.138. 0.196
Ours
Both
0.232
0.783
0.759, 0.805]
Table 4: Perceptual Study Results.
Aggregated results two A/B paired tests. Our method was judged more
realistic with high confidence (binomial confidence interval with 0.01 reported) and contained fewer errors.
Conclusion
We introduced ATISS, novel autoregressive transformer architecture for synthesizing 3D rooms as unordered sets of objects. Our method generates realistic scenes that advance the state-of-the-art for scene synthesis. In addition, our novel formulation enables new interactive applications for semi-automated scene authoring, such general scene completion, object suggestions, anomaly

---

detection and more. We believe that our model is an important step not only toward automating the generation of 3D environments, with impact on simulation and virtual testing, but also toward a new generation of tools for user-driven content generation. By accepting wide range of user inputs, our model mitigates societal risks of task automation, and promises to usher tools that enhance the workflow of skilled laborers, rather than replacing them. In future work, we plan to extend order invariance to object attributes to further expand interactive possibilities of this model, and to incorporate style information. As any machine learning model, our model can introduce learned biases for indoor scenes, and we plan to investigate learning from less structured and more widely available data sources to make this model applicable to wider range of cultures and environments.
References
[1] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Proc. of the International Conf. on Learning Representations (ICLR), 2019.
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πlya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proc. of the European Conf. on Computer Vision (ECCV), 2020.
[4] Siddhartha Chaudhuri, Evangelos Kalogerakis, Stephen Giguere, and Thomas A. Funkhouser. Attribit: content creation with semantic attributes. In The 26th Annual ACM Sympusium on User Interface Software and Technology. U3, St. Andrews, United Kingdom, October 8-11, 2013, 2013.
[5] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proc. of the International Conf. on Machine learning (ICML), 2020.
[6] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXivoor. 2019.
[7] Yunjey Choi, Min-Je Choi, Munyoung Kim, Jung- Woo Ha. Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain maaeetto-mmage translation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
[8] Yunjey Choi. Youngjung Uh. Jaejun Yoo. and Jung-Wor Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
[9] Jeevan Devaranjan, Amlan Kar, and Sanja Fidler. Meta-sim2: Unsupervised learning scene structure for synthetic data generation. In Proc. of the European Conf. on Computer Vision (ECCV), 2020.
[10] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 4171 771-4 Association for Computational Linguistics, 2019.
[11] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv.org, 2020.
[12] Xinhan Di, Penggian Yu, Hong Zhu, Lei Cai, Qiuyan Sheng, and Changyu Sun. Structural plan of indoor scenes with personalized preferences. arXivoorg, 2020.
[13] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In Proc. of the International Conf. on Learning Representations (ICLR), 2017.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In Proc. of the International Conf. on Learning Representations (ICLR), 2021.
[15] Patrick Esser, Robin Rombach, and Björ Ommer. Taming transformers for high- resolution image synthesis. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
[16] Matthew Fisher, Manolis Savva, and Pat Hanrahan. Characterizing structural relationships in scenes using graph kernels. ACM Trans. Graphics, 2011.
10

---

[17] Huan Fu, Bowen Cai, Lin Gao. Lingxiao Zhang, Cao Li. Zenggi Xun, Chengyue Sun, Yiyun Fei, Yu Zheng. Ying Li, Yi Liu, Peng Liu, Lin Ma, Le Weng. Xiaohang Hu, Xin Ma, Qian Qian, Rongfei Jia, Bingiang Zhao, and Hao Zhang. 3d-front: 3d furnished rooms with layouts and semantics. arXiv.org, abs/2011.09122 2020.
[18] Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve J. Maybank, and Dacheng Tao. 3d-future: 3d furniture shape with texture. arXivoorg, abs/2009 09633, 2020.
[19] Ian J. Goodfellow, Jean Pouget-Abaae Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2014.
[20] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016.
[21] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems (NIPS), 2017.
[22] Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochastic scene layout generation from label set. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
[23] Amlan Kar, Aayush Prakash, Ming- Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler. Meta-sim: Learning to generate synthetic datasets. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
[24] Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
[25] Tero Karras, Samuli Laine. Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. 2020.
[26] Angelos Katharopoulos, Apoory Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In Proc. of the International Conf. on Machine learning (ICMI.), 2020.
[27] Mohammad Keshavarzi. Aakash Parikh, Xiyu Zhai, Melody Mao, Luisa Caldas, and Allen Y. Yang. Scenegen: Generative contextual scene augmentation using scene graph priors. arXiv.org, 2020.
[28] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. Proc. of the International Conf. on Learning Representations (ICLR), 2014.
[29] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems (NIPS), 2018.
[30] Adam R. Kosiorek, Hyunjik Kim, and Danilo J. Rezende. Conditional set generation with transformers. arXiv.org, 2020.
[31] Matt J Kusner, Brooks Paige, and José Miguel Hernández-LLoatto Grammar variational autoencoder. In Proc. the International Conf. on Machine learning (ICMI.), pages 1945-1954. PMLR, 2017.
[32] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proc. of the International Conf. on Machine learning (ICML), 2019.
[33] Lingyun Luke Li, Bin Yang, Ming Liang, Wenyuan Zeng, Mengye Ren, Sean Segal, and Raquel Urtasun. End-to-end contextual perception and prediction with interaction transformer. In Proc. IEEE International Conf. on Intelligent Robots and Syxiems (IROS), pages 5784-5791, 2020.
[34] Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, and Hao (Richard) Zhang, GRAINS: generative recursive autoencoders for indoor seenes. ACM Trans. on Graphics, 2019.
[35] Andrew Luo. Zhoutong Zhang. Jiajun Wu, and Joshua B. Tenenbaum. End-to-end optimization of scene layout. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
[36] Paul Merrell, Eric Schkufza, Zeyang Li, Maneesh Agrawala, and Vladlen Koltun. Interactive furniture layout using interior design guidelines. ACM Trans, on Graphics, 2011.
[37] Pascal Müller, Peter Wonka, Simon Haegler, Andreas Ulmer, and Lue Van Gool. Procedural modeling of buildings. ACM Trans. on Graphics, 2006.
[38] Charlic Nash, Yaroslav Ganin, S. M. Ali Eslami, and Peter W. Battaglia. Polygen: An autoregressive generative model of 3d meshes. In Proc. of the International Conf. Machine learning (ICML), 2020.
[39] Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018, Belgium. Brussels, October 31 - November 1, 2018 2018
11

---

[40] Yoav I. H. Parish and Pascal Müller. Procedural modeling of cities. ACM Trans. on Gruphicx, 2001.
[41]
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu, On buggy resizing libraries and surprising subtleties in FID calculation. arXiv.org, 2021.
[42]
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In Proc. of the International Conf. on Machine learning (ICML), 2018.
[43] Aayush Prakash, Shaad Boochoon, Mark Brophy, David Acuna, Eric Cameracci, Gavriel State, Omer Shapira, and Stan Birchfield. Structured domain randomization: Bridging the reality gap by context-aware synthetic data, In Proc. IEEE International Conf. on Robotics and Automation (ICRA). 2019.
[44] Pulak Purkait, Christopher 7ach, and Ian Reid. SG-VAE: scene grammar variational autoencoder to generate new indoor scenes, In Proc. of the European Conf. on Computer Vision (ECCV). 2020.
[45] Siyuan Qi, Yixin Zhu, Siyuan Huang, Chenfanfu Jiang. and Song-Chun Zhu. Human-centrie indoor scene synthesis using stochastic grammar. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2018.
[46] Alee Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. arXiv.org, 2019.
[47] Daniel Ritchie, Kai Wang, and Yu-An Lin. Fast and flexible indoor scene synthesis via deep convolutional generative models. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
[48] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. In Proc. of the International Conf. on Learning Reprexentations (ICLR), 2017.
[49] Gilad Sharir, Asaf Noy. and Lihi Zelnik-Manor. An image is worth 16x 16 words, what is a video worth? In Proc. of the International Conf. on Learning Representations (ICLR), 2021.
[50] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chopter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), 2018
[51] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
[52] Jerry O. Talton. Lou, Steve Lesser, Jared Duke, Radomír Mech, and Vladlen Koltun. Metropolis procedural modeling. ACM Trans. on Graphics, 2011.
[53] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data- efficient image transfommers & distillation through attention. arXiv.org, 2020.
[54] Shubham Tulsiani and Abhinav Gupta. Pixeltransfrrerr: Sample conditioned signal generation. In Proc. of the International Conf. on Machine learning (ICML), 2021.
[55] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In The 9th ISCA Speech Synthesis Workshop, Sunnyvale, CA, USA, 13-15 September 2016, 2016.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), pages 5998-6005 2017.
[S7] Kai Wang, Yu-An Lin, Ben Weissmann, Manolis Savva, Angel X. Chang, and Daniel Ritchie. Planit: planning and instantiating indoor seenes with relation graph and spatial prior networks. ACM Trans. on Graphics, 2019.
[58] Kai Wang, Manolis Savva, Angel X. Chang, and Daniel Ritchie. Deep convolutional priors for indoor scene synthesis. ACM Trans. on Graphics, 37(4):70:177:1 14, 2018.
[59] Xinpeng Wang, Chandan Yeshwanth, and Matthias NicBner. Sceneformer: Indoor scene generation with mransoormess. arXivorg. 2020.
[60] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2019.
[61] Lap-Fai Yu, Sai Kit Yeung. Chi-Keung Tang, Demetri Terzopoulos. Tony F. Chan, and Stanley Osher. Make it home: automatic optimization of furniture arrangement. ACM Trans. on Graphics, 2011.
[62] Lap-Fai Yu, Sai Kit Yeung, and Demetri Terzopoulos. The clutterpalette: An interactive tool for detailing indoor scenes, IEEE Trans. Vis. Comput. Graph., 2016.
[63] Song- Hai Zhang, Shaokui Zhang, Wei- Yu Xie, Cheng- Luo, and Hong- Bo Fu, Fast 3d indoor scene synthesis with discrete and exact layout pattern extraction. arXiv.org, 2020.
12

---

[64]
Zaiwei Zhang, Zhenpei Yang, Chongyang Ma, Linjie Luo, Alexander Huth, Etienne Vouga, and Qixing Huang. Deep generative modeling for scene synthesis via hybrid representations. ACM Trans. Graphics, 2020.
[65] Yang Zhou, Zachary While, and Evangelos Kalogerakis. Scenegraphnet: Neural message passing for 3d indoor scene augmentation. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
[66] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers end-to-end object detection. Proc. of the International Conf. on Learning Representations (ICLR), 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Yes, we discuss this in the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss this in Sec.5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In Sec.4 and the supplementary material.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] FID and classification accuracy metrics were computed as the average of 10 runs (Sec.4; Table 1); perceptual study analysis includes confidence intervals (Tab, 4).
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] This information is provided in the supplemental material.
4. If you are using existing assets (e.g., code, data, models) or curating/releusing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] (b) Did you mention the license of the assets? [N/A] The 3D-FRONT Data Sets Use
License Agreement available together with the dataset that we cite; there is no easy
way to summarize the license in the paper.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent
obtained from people whose data you're
using/curating? [Yes] The data does not include personally identifiable information and collected from interior design website by the creators of 3
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Yes, this data does not contain any personally identifiable info/offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] We include a screenshot of the interface, questions and additional details in the supplementary.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] The participation risks involved only the regular risks associated with using a computer, which are mentioned the study description in the supplementary.
13

---

(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] We include wage per question set instead, as this is the compensation model used by Amazon Mechanical Turk in the supplementary.
14


Reviews:
[{"review_id": "tY9PpY3nQUm", "reviewer": "Reviewer_DWcp", "summary": "This paper proposes an transformer based framework for autoregressive indoor scene synthesis. A scene is modelled by an orthographic projection of the floor geometry (encoded by a CNN), and an unordered set of objects, characterized by category (encoded with a learned embedding), location, orientation and size (encoded with positional encoding). These attributes are then fed into a transformer encoder, which predicts a feature vector that is then decoded by a cascade of MLPs into attributes of the object to be predicted, where a categorical distribution is used for category, and a mixture of logistics is used for the remaining three attributes. \n\nThe proposed model is thus able to generate indoor scenes by autoregressively add the objects in arbitrary order. The scene synthesis results outperform relevant prior works convincingly with respect to multiple evaluation metrics. The paper also showcases a myriad of applications such as scene completion, failure detection and object suggestion, which are enabled by the design of the method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like this paper overall. The authors demonstrate clear knowledge of relevant works of scene synthesis and works that apply transformers to computer graphics. The design choices make intuitive sense and suits the task well e.g. the use of positional encoding for continuous input attributes and the use of mixture of logistics for predicting these attributes. The quality of the outputs are impressive: not just in terms of numbers (these can be hard to judge for such generative models), but also in terms of qualitative examples. The output of the proposed method seems to be handling challenging cases (which, as far as I know, are not handled by prior works well) pretty convincingly e.g. symmetrical chairs around dining tables, placement of sofa around coffee tables, etc. It is also very impressive that the proposed method can avoid collisions without explicit collision handling. While I do think the baselines are not retrained particularly well, judging from the results from the original works, the results still appear to be more convincing.\n\nThe most outstanding limitation of this work has to do with novelty: the autoregressive scene generation formulation is similar to multiple prior works. The claims that this work is novel in terms of being able to handle random ordering and being able to train everything in one model, to me, is not particular convincing. The use of transformers for generating 2D/3D outputs, again, is not a new concept, which the authors also acknowledge themselves. \n\nI cannot be more positive on this paper primarily due to the novelty concerns. Still, I think this is a very well engineered piece of work and clearly advances state of the art (though on a topic traditionally more of interest in graphics/vision community), and can inspire future works both in 3D generative models and in using transformer for autoregressive generation. I lean towards acceptance, but won't be too bothered if others decide against this paper.\n\nAdditional comments and questions below.\n\n=====Method=====\n\n-Orientation can be ambiguous for symmetrical objects. Will that cause problems with training, or do the output distribution naturally handles that?\n\n-Also for orientation: how is SO(2) mapped to R? Are all orientations mapped onto [0, 2pi) or can they fall anywhere? Subsequently, how do the mixture of logistics handle the \"wrap-around\" for decoding?\n\n-Why predicting the 3D cuboid centroid instead of the 2D one? It appears to me that predicting the 3D centroid will hurt disentanglement between location & size, since 2location_y = size_y for all ground-attached objects? What is the behavior in practice, and do the network consistently put everything on ground level?\n\n-I wonder if the current pipeline can be naturally extended to scenes with hierarchy e.g. objects on desks. \n\n-What information is carried in the transformer output q hat? Is it carrying a summary of the current partial scene or is it carrying the \"object feature\" as claimed in the paper? It appears to me that if the latter is the case, then this vector need to carry a lot of information for it to handle all these distributions when the sampled next object can differ in category/location/orientation? Any intuitions on this?\n\n=====Results & Evaluation=====\n\n-I am very curious of the behavior of the proposed model in Figure 6, bottom left. It seems that it is learn to match the orientation of the objects with boundaries rarely seen in the training set? Does this work for all wall orientations or is it something a bit more specific to this input?\n\n-I think figure 7 is not fair. Baselines trained on fixed sequences certainly won't be able to handle such cases. A fairer comparison would be to compare this against versions of the baselines that are trained on random orderings, which they should be able to do.\n\n-I would like to see some visualizations of the distributions predicted, in addition to full generated scenes. This will help judging if the model is overfitting to specific location/orientation/size or is it learning something that is more general. I get that figure 5 partially showcases diversity, but this might be due to difference in generation order, not due to the model capture all possible modes at every single step. A bit more evaluations on this would be nice.\n\n-Would also like to see just a bit more qualitative results (sampled randomly), perhaps in the supplementary?\n\n=====Post-Rebuttal Comment=====\n\nI thank the authors for their details response to my concerns. I am convinced by most of them, though there are a few that I am still left unconvinced with, which I list below. I have raised my score from 6 to 7 in response to the rebuttals, and the fact that other reviewers do not express too much concerns regarding novelty.\n\n- Ordering & Novelty: I still don't feel that it is the best idea to claim random ordering as a main novelty - in the case of short sequences it isn't really that challenging to do, and there's plenty of prior works that handle much more challenging cases of set generation e.g. [53]. While I do agree that it is nice that this model can do autoregressive scene synthesis with random ordering, personally I think the outstanding quality of results is the more impressive part.\n- Compare against prior works on ordering (Q8 &  uRVi Q5): If I understand it correctly, it should be easy to train a version of the existing works with random sequences of objects. It should be a relatively straightforward modification with their data loader. I don't doubt that they will perform worse with this modification, but just feel that adding it will make the claims in the paper much clearer and remove the need for indirect speculations.\n- Predicted distributions (Q9): I think there's quite a bit of overfitting going on, but I guess this is OK as the emergent final layouts are indeed diverse. Still, this might be worth discussing a bit in the paper and might serve as a good direction for future works. Besides, some visualizations with a partial scene instead of an empty room might be more illustrative.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not that I am aware of.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "alQ9PvvB7pj", "reviewer": "Reviewer_urVi", "summary": "The submission proposes a permutation-invariant autoregressive generative model based on transformers for populating 3D rooms with furniture objects.  The model is trained with labeled 3D bounding boxes and is trained and evaluated using the 3D-FRONT dataset.  The proposed model predicts attributes (category, size, location - centroid of bounding box, and orientation - rotation around up axis) for each object in an autoregressive manner using a transformer-based model until stop is predicted.  The category is modeled using a categorical distribution, and the other attributes are modeled using a mixture of logistic distributions.  To model the objects in a scene and a unordered set, the model does not encode the position of the object in the sequence.  To allow the model to handle different permutations of objects, the model is trained with different permutations.  The output prediction of the model is effectively a set of oriented bounding boxes with categories, which are then replaced with an appropriate CAD model with a matching category and closest in size.  Experiments comparing the proposed model against prior work (FastSynth and SceneFormer, both retrained on the 3D-FRONT dataset) shows the proposed model can produce higher quality furniture layouts given a room outline and room type.  There are also qualitative examples of the model used for scene completion, identifying and correcting poor object placement, and suggest objects to place at a given location.  Ablations on the number of logistic distributions, and layout encoder is provided in the supplemental.\n\nMain contributions\n- Transformer-based model for indoor scene synthesis that treats the objects as unordered sets during training, and autoregressively generate each object with their attributes\n- Set of experiments comparing the proposed model against prior work", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: While idea of using transformers is not new (it has been explored in the unpublished work SceneFormer), this work provides convincing experiments showing the proposed model outperforms prior work on indoor 3D scene synthesis.  It also point out that while the scene synthesis can be formulated as a sequence generation problem, it is important to allow for different ordering during training as applications such as scene completion may require generation from a random set of objects.  \n\nQuality: The submission appears to be technically sound with reasonable experimental setup and results (see weaknesses for missing comparisons). \n\nClarity: The paper is well written and clear.  When describing various parts of the model, the rationale for choosing certain design decision is provided.\n\nSignificance: The ability to rearrange furniture in realistic layouts will allow for automatic generation of indoor scene data at scale.  This work is a step toward that direction.  It is also one of the first works to move away from the SUNCG dataset which is no longer available for download.\n\nStrengths:\n- The proposed model shows how to use transformers to model scenes as unordered sets of objects\n- The experiments are thorough with both automated evaluation and perceptual study with users.\n- Experiments show that the proposed model generates higher-quality rooms and is faster than prior work\n\nWeaknesses:\n- The train/test split used is unclear (see \"Other comments and questions\")\n- There is limited ablation studies\n- Some of the claims made in the paper is inaccurate (see \"Other comments and questions\")\n\n\nMissing References:\nThe submission is missing discussion of older/alternative lines of work on indoor scene synthesis.  Some examples are provided below:\n- Interactive Furniture Layout Using Interior Design Guidelines, Merrell et al, SIGGRAPH 2011 \n- Example-based synthesis of 3D object arrangements, Fisher et al, SIGGRAPH ASIA 2012  \n- Learning spatial knowledge for text to 3D scene generation, Chang et al, EMNLP 2014 \n- Language-driven synthesis of 3d scenes from scene databases, Ma et al, SIGGRAPH Asia 2018\n- Activity-centric Scene Synthesis for Functional 3D Scene Modeling, Fisher et al, SIGGRAPH Asia 2015\n- Adaptive synthesis of indoor scenes via activity-associated object relation graphs, Fu et al, SIGGRAPH Asia 2017\n- Fast and Scalable Position-Based Layout Synthesis, Weiss et al, Transactions on visualization and computer graphics 2018\n\nThe references should also be proofread for errors:\n- Citation for Metropolis procedural modeling is duplicated ([50,51])\n- Ideally when a preprint is superseded by a refereed publication at a conference/journal, the citation should be for the refereed publication (not the preprint).\n\n\nOther comments and questions:\n- Equation 6: Should the R^M be R^C since it corresponds to the c (category of object j)? \n\n- The notation for the per-object contextual embedding C_j is a bit confusing.  It is unclear if the C_j refers to the 64-dimensional embedding (as in supplement line 32), or to the embedded attributes (as suggested by the main paper Lines 151,159) before it is projected to a 64-dimensional vector.\n\n- Lines 242-249: The description for Scene Completion states that both the baseline methods [45, 58] fails to generate any objects in Figure 7.  This is not true as SceneFormer [58] has clearly added some objects to both scenes.  \n\n- Line 197: I'm not sure that the 3D-FRONT dataset is \"professionally designed\".  The scenes from 3D-FRONT are based on a set of scenes designed using the Alibaba Topping Homestyler tool (it's unclear if the users were professional or just regular users), and due to copyright restriction, the CAD models in the scenes are algorithmically replaced/removed.  So the final dataset could not be said to be \"professionally designed\".\n\n- The split of scenes into training and test set is not really specified.  Line 270 mention there is 211 test set plans, but it is not that clear where this split comes from and how is it split.  Is this a standard split from the 3D-FRONT dataset?  Is the split done so that the rooms for the training and test set come from different houses?  Is the set of objects shared between the splits?  Is there a validation split for hyperparameter tuning?\n\n- Since the results for Table 1 are for 10 runs, it would be good to report the error bars for those.\n\n- The paper points out that one limitation of prior work is that they are trained on a sorted sequence of objects and thus fails to fail to generate any objects giving an arbitrary set of objects as input. It's not actually clear to me why the FastSynth[45] model requires a ordering during training.  According to the FastSynth[45] paper, it is trained by randomly removing a set of objects in the scene but for some reason they require a stable ordering because otherwise \"there are too many valid possible categories at each step\".  Do you think that a version of the FastSynth model that is trained with random removal of objects (without the ordering) would be able to generate addition objects for the cases presented in Figure 7?      ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of the method is described in the submission.  There is not much discussion of how the limitations would translate to negative societal impact.   \n", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "GJvOPFop_A_", "reviewer": "Reviewer_z3Jv", "summary": "The paper proposes an approach for 3D scene synthesis based on a transformer architecture.  The approach encodes 3D scenes from a top-down floorplan image and a set of per-object learned embeddings that capture the category, size, location, and orientation of each object.  Critically, the formulation addresses the permutation-invariant nature of object sets in 3D scenes, in contrast with prior work which relied on generation of scenes through an ordered sequence of object insertions.  The proposed approach is demonstrated on three tasks: 3D scene synthesis \"from scratch\", scene completion (i.e. conditioning synthesis on a partial scene), suggestion of objects to place at a particular location in a scene, and detection of implausibly placed objects.\n\nThe experiments contrast the proposed approach against baselines from prior work, as well as an ablation that requires object ordering.  Scenes generated using the approach are shown to be of higher quality (as measured by FID, classification, and KL divergence metrics) relative to scenes generated using the baseline methods.  Additional analysis experiments show that the approach also generalizes well to out-of-domain floorplan shapes, and that it is significantly faster than prior work.  Finally, a perceptual study carried out on Amazon Mechanical Turk shows that scenes generated by the approach are deemed to contain significantly fewer object placement errors and to be more realistic that scenes generated by the baselines.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe paper's use of a transformer architecture with a permutation-invariant object set encoding is novel, and sets it apart from prior work. \n On the other hand, the discussion of related work on graph-based scene synthesis is missing early work that introduced graph-based representations for 3D scenes (​\"Characterizing Structural Relationships in Scenes using Graph Kernels\" [Fisher et al. 2011]), and work that adopted such representations for 3D scene synthesis (\"Interactive furniture layout using interior design guidelines\" [Merrell et al. 2011], \"Make it home: automatic optimization of furniture arrangement\" [Yu et al. 2011], \"Example-based Synthesis of 3D Object Arrangements\" [Fisher et al. 2012]).  I would recommend that the authors incorporate and discuss these references as well.\n\nQuality:\nThe submission is technically sound, as far as I can tell.  The experiments are fairly thorough and support the claims well.  In particular, the attention to detail in clarifying the permutation-invariant nature of the scene synthesis problem statement is valuable.  The careful analysis of of model capacity differences (network parameter comparison) is also quite valuable, and will hopefully inform future work.\n\nClarity:\nThe paper is clearly written overall.  There are a few minor low-level writing issues that can be fixed through a proofreading pass.  A specific example that caught my eye in L129-L130: the order of the category, size, orientation, and location is not consistent with the order of the symbols immediately following the description.  Also \"logistics distributions\" should be \"logistic distributions\".\n\nSignificance:\nI believe this paper is likely to inspire follow up work in the area of 3D scene synthesis.  This is a solid paper, and I am in favor of acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss some directions for future work and connect them limitations of the proposed approach (in the main paper, and in the supplementary material).  Societal impacts of the work are mentioned briefly in the conclusion to the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree, and I concur, that this paper is solid. It has a well executed, relatively scaled up model and idea with good results. I also agree with the relative lack of novelty or insight, so I recommend acceptance as a poster.
Decision: Accept (Poster)