COFS: COntrollable Furniture layout Synthesis
Authors: Wamiq Reyaz Para
Paul Guerrero
Niloy Mitra
Peter Wonka
Venue: Submitted to ICLR 2023 (2023)
Keywords: generative modelling
conditional generation
layouts
transformers

Abstract: Realistic, scalable, and controllable generation of furniture layouts is essential for many applications in virtual reality, augmented reality, game development and synthetic data generation. The most successful current methods tackle this problem as a sequence generation problem which imposes a specific ordering on the elements of the layout, making it hard to exert fine-grained control over the attributes of a generated scene. Existing methods provide control through object-level conditioning, or scene completion, where generation can be conditioned on an arbitrary subset of furniture objects. However, attribute-level conditioning, where generation can be conditioned on an arbitrary subset of object attributes, is not supported. We propose COFS, a method to generate furniture layouts that enables fine-grained control through attribute-level conditioning. For example, COFS allows specifying only the scale and type of objects that should be placed in the scene and the generator chooses their positions and orientations; or the position that should be occupied by objects can be specified and the generator chooses their type, scale, orientation, etc. Our results show both qualitatively and quantitatively that we significantly outperform existing methods on attribute-level conditioning.

Under review as a conference paper at ICLR 2023
COFS CONTROLLABLE FURNITURE LAYOUT SYNTHESIS
Anonymous authors
Paper under double-blind review
ABSTRACT
Realistic, scalable, and controllable generation of furniture layouts is essential for many applications in virtual reality, augmented reality, game development and synthetic data generation. The most successful current methods tackle this problem as a sequence generation problem which imposes specific ordering on the elements of the layout, making it hard to exert fine-grained control over the attributes of generated scene. Existing methods provide control through object- level conditioning, or scene completion, where generation can be conditioned on arbitrary subset of furniture objects. However, attribute- conditioning, where generation can be conditioned on an arbitrary subset of object attributes, is not supported. We propose COFS, method generate furniture layouts that enables fine- fine-grained control through attribute-level conditioning, For example, COFS allows specifying only the scale and type of objects that should be placed the scene and the generator chooses their positions and orientations; or the position that should be occupied by objects can be specified and the generator chooses their type, scale, orientation, etc. Our results show both qualitatively and quantitatively that we significantly outperform existing methods on attribute-level conditioning.
INTRODUCTION
Automatic generation of realistic assets enables content creation at a scale that is not possible with traditional manual workflows. It is driven by the growing demand for virtual assets in both the creative industries, virtual worlds, and increasingly data-hungry deep model training. In the context of automatic asset generation, 3D scene and layout generation plays central role much of the demand is for the types of real-world scenes we see and interact with every day, such as building interiors.
Deep generative models for assets like images, videos, 3D shapes, and 3D scenes have come long way to meet this demand. In the context of 3D scene and layout modeling, in particular auto-regressive models based on transformers enjoy great success. Inspired by language modeling, these architectures treat layouts as sequences of tokens that are generated one after the other and typically represent attributes of furniture objects, such the type, position, scale of object. These architectures particularly well suited for modeling spatial relationships between elements of a layout. For example, (Para et al., 2021) generate two-dimensional interior layouts with transformers, one for furniture objects and one for spatial constraints between these objects, while SceneFormer (Wang et al., 2021) and ATISS (Paschalidou et al., 2021) extend interior layout generation to 3D.
A key limitation of basic autoregressive approach is that it only provides limited control over the generated scene. It enforces a sequential generation order, where new tokens can only be conditioned on previously generated tokens and in addition it requires consistent ordering of the token sequence. This precludes both object-level conditioning, where generation conditioned on a partial scene, e.g., an arbitrary subset of furniture objects, and attribute-level conditioning, where generation is conditioned on an arbitrary subset of attributes of the furniture objects, e.g., class or position of target objects. Most recently, ATISS (Paschalidou et al., 2021) partially alleviates this problem by randomly permuting furniture objects during training, effectively enabling object-level conditioning. However, attribute-level conditioning still remains elusive.
We aim to improve these results by enabling attribute-level conditioning, in addition to oeject-level conditioning. For example, a user might be interested to ask for a room with a table and two chairs, without specifying exactly where these objects should be located. Another example is to perform

---

Under review as a conference paper at ICLR 2023
Prior Work
Proposed
C1
bed table
D1
Enc
Dec
wardrobe
Decodar
Encoder
Decoder
C2
table
D2
bed, 90"
Enc
Dec
wardrobe
Figure I: Motivation. Current autoregressive layout generators (A) provide limited control over the generated result, since any generated value (denoted by black triangles) can only be conditioned on values that occur carlier in the sequence (values that are given as condition are denoted with c). Our proposed encoder-decoder architecture (B) adds bidirectional attention through encoder, allowing the model to look ahead, that all values the sequence can be given as condition. This enables conditioning on an arbitrary subset of objects or object attributes a layout. In C1, only the position an object, shown as pink cuboid, is given as condition and COFS performs context-aware generation of the remaining attributes. In D1, only object types are provided as condition, and D2 adds the bed orientation to the condition. Note how the layout adapts to fit the updated condition.
object queries for given geometry attributes. The user could specify the location of an object and query the most likely class, orientation, and size of an object at the given location. Our model thereby extends the baseline ATISS with new functionality while retaining all its existing properties and performance.
The main technical difficulty in achieving attribute-level conditioning is due to the autoregressive nature of the generative model. Tokens in the sequence that define a scene are generated iteratively, and each step only has information about the previously generated tokens. Thus, the condition only be given at the start of the sequence, otherwise some generation steps will miss some of the conditioning information. The main idea of our work is allow for attribute-level conditioning using two mechanisms: (i) Like ATISS, we train our generator to be approximately invariant to object permutations by randomly permuting furniture objects training time. This enables object-level conditioning since an arbitrary subset of objects can be given as the start of the sequence. To condition on a partial set of object attributes however, the condition is not restricted the start of the sequence. Attributes that are given as condition follow unconstrained attributes that need to be generated. (ii) To give our autoregressive model knowledge of the entire conditioning information in each step, we additionally use transformer encoder that provides cross-altention over the complete conditioning information in each step. These two mechanisms allow us to accurately condition on arbitrary subsets of the token sequence, for example, only on tokens corresponding to specific object attributes.
In our experiments, we demonstrate four applications: (i) attribute-Ieeve conditioning, (ii) attribute- level outlier detection, (iii) object-level conditioning, and (iv) unconditional generation. We compare to three current state-of-the-art layout generation methods (Ritchie et al., 2019; Wang et al., 2021; Paschalidou al., 2021) and show performance that is par or superior unconditional generation and object-level conditioning, while also enabling attribute-level conditioning, which, to the best of our knowledge, is currently not supported by any existing layout generation method.
RELATED WORK
We discuss recent work that we draw inspiration from. In particular, we build on previous work in Indoor Scene Synthesis, Masked Language Models, and Set Transformers.
Indoor Scene Synthesis: Before the rise of deep-learning methods, indoor scene synthesis methods relied on layout guidelines developed by skilled interior designers, and an optimzation strategy such that the adherence to those guidelines maximized ( (Yu et al., 2011; Fisher al., 2012; Weiss et al., 2019). Such optimization is usually based on sampling methods like simulated annealing, MCMC, or rjMCMC. Deep learning based methods, e.g. ( (Wang et al., 2019; Ritchic et al., 2019; Wang et al., 2021; Paschalidou et al., 2021) are substantially faster and can better capture the variability of the design space. The of-te-of-the-as methods among them are autoregressive in nature. All of these operate on a top-down view of partially generated scene. PlanIT and FastSynth then autoregressively generate the rest of the scene. FastSynth uses separate CNNs+MLPs to create probability distributions over location, size and orientation and categories. PlanIT the other hand generates graphs where
2

---

Under review as a conference paper at ICLR 2023
nodes objects and edges are constraints on those objects. Then scene is instantiated by solving CSP on that graph.
Recent methods, SceneFormer (Wang et al., 2021) and ATISS (Paschalidou et al., 2021) use trans- former based architectures to sidestep the problem of rendering partial scene which makes PlanIT and FastSynth slow. This is because using a transformer allows the model to accumulate information from previously generated objects using the attention mechanism. SceneFormer flattens the scene into a structured sequence of the object attributes, where the objects are ordered lexicographically in terms of their position. It then trains separate model for each of the attributes. ATISS breaks the requirement of using a specific order by training on all possible permutations of the object order and removing the position encoding. In addition, it uses single transformer model for all attributes and relies on different decoding heads which makes it substantially faster than other models while also having significantly fewer parameters.
Masked Language Models: Masked Language Models (MLMs) like BERT (Devlin et al., 2019), ROBERTA (Liu et al., 2019), and BART (Lewis et al., 2020) have been very successful in pre- training for language models. These models are pretrained on large amounts of unlabeled data in an unsupervised fashion, and are then fine-tuned on a much smaller labeled dataset. These fine- tuned models show impressive performance on their corresponding downstream tasks, However, the generative capability of these models has not been much explored except by Wang et al. in (Wang & Cho, 2019), which uses a Gibbs- sampling approach to sample from a pre-trained BERT model. Follow up work in Mansimov et al. (Mansimov et al., 2020), proposes more general sampling approaches. However, the sample quality is still inferior to autoregressive models like GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020). More recently, MLMs have received renewed interest especially in the context of 2022) shows that with a carefully designed masking schedule, high quality image samples can be generated from MLMs with parallel sampling which makes them much faster than autoregressive models. Edi-BERT (Issenhuth et al., 2021) shows that the BERT masking objective can be succesfully used with a VQGAN (Esser et al., 2021) representation of an image to perform high quality image editing. Our model most closely resembles BART when used generative model.
(Issenhuth ct al., 2021; Chang et al., 2022). MaskGit (Chang et al.,
Zaheer et al. (Zaheer et al., 2017) introduced framework called DeepSets providing a mathematical foundation for networks operating on set-structured data. A key insight is that operations in the network need to be permutation invariant. Methods based on such formulation were extremely successful, especially in the context of point-could processing (Charles al., 2017; Ravanbakhsh et al., 2016). Transformer models without any form of positional encoding are permutation invariant by design. Yet, almost all the groundbreaking works in transformers use some from of positional encoding, in objection detection (Carion al., 2020), language generation (Radford et al., 2019; Brown et al., 2020), and of the early attempts use a truly permutation invariant set transformer was in Set Transformer (Lee et al., 2019), who methodically designed principled operations that are permutation invariant but toy-problems. However, recent work based on (Lee
Set Transfomers:
(Chang et al., 2022). One
could only achieve respectable performance et al., 2019) shows impressive performance in 3d-Object Detection (Chenhang He & Zhang, 2022), 3d Pose Estimation (Ugrinovic et al., 2022), and SFM (Moran et al., 2021).
3 METHOD
Our goal is to design generative model of object layouts that allows for both object-level and attribute- level conditioning, Attribute-leve conditioning enables more flexible partial layout specification, for example specifying only the number and types of objects in a layout, but not their positions, or exploring suggestions for plausible objects at given positions in the layout. An overview of our architecture given in Figure 2.
3.1 LAYOUT REPRESENTATION
We focus on 3D layouts in our experiments. A 3D layout C= = (I. B) is composed of two elements a top-down representation of the layout boundary I. such as the walls of a room, and a set of
k hiree-dimensional oriented bounding- boxes B = {B, {B.)*= of the objects in the layout. The boundary is given binary raster image and each bounding box is represented by four attributes:
B, = (Ti, t, C,, r:), representing the object class, center position, size, and orientation, respectively.
The orientation is rotation about the up-axis, giving total of 8 scalar values per bounding box.
3

---

Under review as a conference paper at ICLR 2023
Encoder
Decoder
Decoder
Encoder
R
R2
100
12
0 0
O
Figure 2: COFS Overview. (Left): The model a BART-like encoder-dceoder model, with bidirectional attention in the encoder and an autoregressive decoder. The encoder encodes the layout as a set without ordering information and therefore does not receive (absolute) position tokens. However, to disambiguate a single object, the encoder receives additional information in the form of Relative Position Tokens R', and the Object Index Tokens O'. During training, object order randomly permuted replaced with a [MASK] token. The decoder outputs sequence representation of the set Absolute Position Tokens P'. It performs two tasks 1. copy-pane paste: the decoder copies the unmasked attributes to their proper location 2: mask prediction: the decoder predicts the actual value of the token corresponding to [MASK] token in the encoder input. (Right): During inference, measure likelihood, create copy of the sequence with each token masked out. The decoder outputs probahility distribution over the possible values of the masked tokens.
layout and random proportion of tokens trained with
The layout is arranged into a layout sequence S by concatenating bounding box parameters. Additionally, special start and stop tokens SOS and EOS are added to mark the start and the end of sequence: S [SOS; : Bx: BOS]. where [: denotes concatenation. The layout boundary is not generated by our method, but it is used condition, Section 3.3 provides details.
3.2 GENERATIVE MODEL
We use a transformer-ban generative model, as these types of generative models have shown great performance in the current state of the art. Originally proposed as a generative model for language, transformer- -basdd generative models represents layouts as sequence of tokens S = (81,..., ,.))) ,,) that are generated auto-regressively; one token is generated at a time, based on all previously generated tokens:
((3 = fo(
(1)
where p(s: Sex) the probability distribution over the value of token 8s. computed by the generative model So given the previously generated tokens Sez ( 8;-1). We sample from p(s| to obtain the token s,. Each token represents one attribute of an object, and groups of adjacent tokens correspond to objects. More details the layout representation described in Section 3.1.
Limitations of traditional conditioning. To condition a transformer-based generative model on a partial sequence C (Co.... Cm), we replace tokens of with the corresponding tokens of C, giving us the sequence SC, This is done after each generation step, so that the probability for the token each step conditioned on SC instead of Szji
pl S(, = fo(S
(2)
Each generated token in SC (i.e. tokens that are not replaced by tokens in C)) needs to have knowledge of the full condition during its generation step, otherwise the generated value may be incompatible with some part of the condition. Therefore, since each generated token s; only has information about the partial sequence SE, of tokens that are closer to the start of the sequence, the condition only be given start of the sequence:
( c; if i S |C|
sc =0=
otherwise.
Typically both the objects and the attributes the objects the sequence are consistently ordered according some strategy, for example based raster order of the object positions (Para et al., 2021), or on the object size (Wang et al., 2019). Therefore, a generative model f fonderel that is only trained to generate sequences in that order cannot handle different orderings, that in general:
poddrrdd (5C) # #18 pordered (To (m,s))))
4

---

Under review as a conference paper at ICLR 2023
where Wo is random permutations of the objects in sequence Sei. The consistent ordering improves the performance of the generative model, but also presents a challenge for conditioning: it limits the information that can appear in the condition. In bedroom layout, for example, if beds are always generated before nightstands in the consistent ordering, the layout can never be conditioned on nightstands only, this would preclude the following tokens from containing bed.
Object-level conditioning. Recent work (Paschalidou et al., 2021) tackles this issue by forgoing consistent object ordering, and instead training the generator to be approximately invariant to permutations of objects in the sequence:
fo(S (o()o((
This makes generation more difficult, but enables object-level conditioning by allowing conditioning on arbitrary subset of objects, as now arbitrary objects can appear at the start of the sequence. However, since only objects are permuted and not their attributes, it does not allow conditioning on subsets of object attributes. Permuting object attributes to appear arbitrary positions in the sequence is not good solution to enable attribute-level conditioning, as this would make it very hard for the generator to determine which attribute corresponds to which object.
Attribute-level conditioning. We propose extend previous work to allow for attribute-level conditioning by using two different conditioning mechanisms, in addition to the approximate object permutation invariance: First, similar to previous work, we provide the condition as partial sequence C. However, unlike previous work, some tokens in the condition are unconstrained and will not be used to replace generated tokens. We introduce special mask tokens .M in C' mark these unconstrained tokens. For example, if all tokens corresponding to object positions and orientations in C' are mask tokens, the positions and orientations will be generated by our model, only the remaining tokens: object types and sizes will be constrained by the condition. The constrained sequence S is then defined as:
Sc = (c; c; if S |C] and a * M
8; otherwise.
Second, to provide information about the full condition each generated token, modify fo to use transformer encoder that encodes the condition C' into a set of feature vectors that each generated token has access
p S C) fo(S C2) where C (),(ci,)),... .)...).,(999)) C)}-
where C is the output of the encoder, set of encoded condition tokens. We use standard transformer encoder-decoder setup (Vaswani et al., 2017) for fo and Js. implementation details are provided in Section 3.3, and the complete architecture is described in detail in the appendix.
Parameter probability distributions. The generative model outputs probability distribution over one scalar component of the bounding box parameters in each step. Probability distributions over the discrete object class are represented vectors of logits i, over discrete choices that can be converted to probabilities with the softmax function. Similar to previous work (Paschalidou et al., 2021; Salimans et al., 2017), we represent probability distributions continuous parameters, like the center position, size, and orientation, as mixture of logistic distributions.
p(b) = Σ. Σ ag Logoiiiti o;),
p(T) = softmax(1,))
(8)
where b is a single scalar attribute from fú c; or r,, The mixture weight, mean and variance of the logistic distribution components are denoted a, μu, respectively. Each probability distribution over a continuous scalar component is parameterized by -d-diennso vector, and probability distributions over the object class represented dimeennnsi vectors, where n,, is the number of object classes.
3.3 IMPLEMENTATION
Condition encoder 9ø: To encode the condition C into a set of encoded condition tokens C", we Transformer encoder with full bidirectional attention. As positional encoding, provide two additional sequences: object index tokens Oi provide for each token the object index in the permuted sequence of objects; and relative position tokens RD provide for each token the element index inside
5

---

Under review as a conference paper at ICLR 2023
the attribute tuple of object. Since the attribute tuples are consistently ordered the index can be used to identify the attribute type of a token. These sequences are used as additional inputs to the encoder. The encoder architecture is based on BART (Lewis et al., 2020), details are provided in the appendix.
Boundary encoder g.: To allow conditioning the layout boundary I, prepend feature vector encoding 2σ of the boundary to the input of the condition encoder, as shown in Figure 2, so that the encoder receives both zg and the condition sequence C. Similar to ATISS, we use an untrained ResNet-18 18 (He et al., 2016) to encode a top-down view of the layout boundary into an embedding vector.
Generative model fg: The generative model is implemented as a Transformer decoder with a causal attention mask. Each block of the decoder performs pross-attention over the encoded condition tokens C9. As positional encoding, we provide absolute position tokens P. which provide for each token the absolute position in the sequence S. This sequence used as additional input to the generative model. The output of the generative model in each step is one of the parametric probability distributions described in Eq. 8. Since the probability distributions for discrete and continuous values have a different numbers of parameters, we use a different final linear layer in the generative model for continuous and discrete parameters. Similar to the encoder, the architecture of the generative model is based on BART (Lewis et al., 2020).
During training, we create a ground truth sequence SGT with randomly permuted objects. We generate the condition C as copy of SGT and mask out random percentage of the tokens by replacing them with the mask token M. The boundary encoder g the condition encoder Jo and the generative model fe are then trained jointly, with the task to generate the full sequence SGT For unmasked tokens in C, this is copy task from C to the output sequence S. For masked tokens, this is a scene completion task. We use the negative log-likelihood loss between the predicted probabilities p(s:) and ground truth values .दT for tokens corresponding continuous parameters, and the cross-entropy loss for the object category т. The model is trained with teacher-forcing.
Training:
Sampling: We generate a sequence auto-regressively. one token at time, by sampling the probability distribution predicted by the generative model (as defined in Eq. 7) in each step. We use the same model for both conditional and unconditional generation. For unconditional generation, we start with a condition C' where all tokens are mask token M. To provide more complete information about the partially generated layout the encoded condition tokens C we update the condition C after each generation step by replacing mask tokens with the generated tokens. Empirically, we observed that this improves generation performance. An illustration and the full algorithm of this approach shown in the supplementary. Once layout has been generated, we populate the bounding boxes with objects from the dataset with a simple retrieval scheme. For each bounding box, we pick the object of the given category T that best matches the size of the bounding box. In the supplementary, we present an ablation of the tokens Oi, R', and P that we add to the conditional encoder and generative model.
4 RESULTS
Datasets: We train and evaluate our model on the 3D-FRONT dataset (Fu et al., 2021). It consists of of about 10k indoor furniture layouts created by professional designers. We train on the BEDROOM category and follow ATISS preprocessing which removes a few problematic layouts that have intersections between objects, mislabeled objects, layouts that have extremely large or small dimensions. For further details on the preprocessing, we refer the reader to ATISS (Paschalidou et al., 2021). This yields approximately 6k/224, 0.6k/125, 3k/516 and 2.6k/583 total/test set layouts for BEDROOM, LIBRARY, DINING, LIVING, respectively.
Baseline: ATISS (Paschalidou et al., 2021) is the most recent furniture layout generation method that provides the largest amount of control over the generated layout, and is therefore most related to our method. However, ATISS does not provide pretrained models, hence we train their models using the official code matching their training settings closely as possible. While ATISS does not support attribute-Ievel conditioning, we can still use it as a bascline by applying the sampling procedure defined in Eq. 6: we sample tokens usual, but when reaching token that is given as condition (i.e. a token in C' that is not mask token), we use the value given as condition instead of the sampled value.
hhtps://gthuuhccomnnaabbbbaaaaaa commit 0cce45b

---

Under review as a conference paper at ICLR 2023
500
A1155
COFS
ATISS
ATISS-attrib
COFS
COFS-attri
100
30
Unconditional
Object
Attribute
Realism
error
Figure 3: NLL comparison. We compare the NLL of our method to ATISS in three settings: (Uncond.) We measure unconditional generation performance the NLL of BEDROOM test set layouts in our model. (Object) We measure the NLL of layouts gen- crated with object-level conditioning and (Attribute) attribute-lewe conditioning. Note how our method performs slightly better than ATISS on unconditional generation and object-level conditioning, while show- ing clear advantage in attribute-level conditioning.
Figure 4: Perceptual Study. We compare the percent- age of comparisons in which users found either method more realistic, and the percentage of results in which users did not find any obvious errors such as object in- tersections. Results show a large advantage for COFS realism layouts generated with attribute- ditioning (-attrib), and a smaller, but still significant advantage in the percentage of error-fiee free layouts.. This advantage is also present in the unconditional setting.
con-
4.1 QUANTITATIVE RESULTS
Metrics: For unconditional generation, we use the negative log-likelihood of test set sequences in our model as main quantitative metric. A small NLL shows that model approximates the dataset distribution well. For both object-level and attribute-leve conditioning, we use the NLL of the generated sequences metric. This includes the condition tokens that come from the test set. If the generated layout does not harmonize with the condition, the NLL will be high. Additionally, we performed perceptual study for attribute-level conditioning and unconditional generation (details are given below and in the supplementary material, respectively).
For object-level conditioning, we remove three random objects from each test sequence obtain condition sequences C. For attribute-level conditioning, conditions C obtained from test set sequences by replacing all tokens except size and position tokens with mask tokens, effectively conditioning the sizes and positions of all objects, and letting the generator infer the types and orientations of all objects.
Choosing conditions:
Figure 3 shows NLL results on the BEDROOM category. For unconditional generation, we can see that we perform on par or slightly better than ATISS. We believe that our slight advantage here might be due more fine-grained sequence representation of the layout on our side, which allows for more detailed attention. For object-level conditioning, our performance slightly better than ATISS, again because of detailed attention. Our main contribution, however, lies in attribute-level conditioning, where we can see a clear advantage for our method. Since ATISS cannot look ahcad in the sequence, any generated token cannot take into account future condition tokens. The bidirectional attention of our encoder enables look-ahead and gives the generator knowledge of all future condition tokens, giving us generated layouts that can better adapt the supplied condition.
Discussion:
4.2 PERCEPTUAL STUDY
We conducted two perceptual studies to further evaluate the quality of generated furniture layouts compared to ATISS. One of the studies focused on unconditionally generated layouts and the other on layouts generated with attribute-level conditioning. For this purpose, we randomly sampled layouts from the BEDROOM layouts evaluated in the previous section for both COFS and ATISS. Subjects were shown pair of layouts generated from the same floorplan boundary by COFS and ATISS, and asked three questions: which of the layouts looked more realistic, and for each of the two layouts, if it showed obvious errors like intersections. A total of 9 subjects participated in the unconditional study, and 8 subjects participated in the attribute-level study. More details about setup can be found in the supplementary. Figure shows the results. We can see that the our method produces significantly more realistic layouts compared to ATISS. The error plots on the right-hand side show that this only in some part due to avoiding obvious errors such as intersections.
4.3 QUALITATIVE RESULTS
A few examples of furniture layouts generated with attribute-level conditioning are shown in Figure 1 and in Figure 5. See the captions for details. Figure 6 additionally shows how attribute-level conditioning can be used perform sequential edits of furniture layout.
7

---

Under review as a conference paper at ICLR 2023
GT
Figure 5: Attribute-level conditioning: On the left, we show GT floorplan. We set the condition to include two beds facing opposite directions and sample. The model generates two plausible layouts for this challenging case (see supplementary). On the right, we constrain the location of the next object be sampled. The location highlighted in pink. In this example, force the inferred size into a narrow range, such that the chair even matches the style of the chairs in the example on the left, even though we use a simple object-rctrieval scheme.
network automatically infers the proper class and size. The constraints
Figure 6: Sequential edits with attribute-level conditioning: We show how COFS can be used to selectively edit parts of a scene. Left shows GT and the other two are samples with classes and orientation as condition. When we change orientation of few objects, COFS produces realistic layouts affecting only a part of the scene. More details in the supplementary.
Figure 7: Object-level conditioning, In the top row, we show examples of object-level conditions that used to condition generation of the scenes shown below. The generated layouts all plausibly combine the generated objects with the objects given as condition into realistic layouts.
In Figure 7, we show layouts generated with object- level conditioning, providing the objects shown the top row as condition. Note how our method generates plausible layouts in each case.
4.4 OUTLIER DETECTION
We can also use COFS to perform outlier detection. To estimate the likelihood of each token, we follow (Salazar et al., 2020) and replace the token at ith position with [MASK]. This can be performed in parallel by creating batch in which only one element is replaced with [MASK]. J. The likelihood
of one object is then the product likelihoods of all its attributes. Attributes or objects with low likelihood can then be resampled. Results on several of the layout categories of our dataset are shown in Figure& 8. This can be thought of a form of surributed-condiiionn generation.
4.5 ADDITIONAL EXPERIMENTS WITH UNCONDITIONAL GENERATION
Here we present additional experiments with unconditional generation. We include two additional state-of-the art methods for unconditional generation, FastSynth (Ritchie et al., 2019) and Scene- Former (Wang et al., 2021), in our quantitative experiments.
We use a set of metrics that mostly derive from (Ritchic et al., 2019; Paschalidou et al., 2021). They are defined in greater detail in the supplementary. Following (Ritchie et al., 2019), we report the KL- divergence between the distribution of the classes of generated objects and the distribution classes of the objects in the test set. We further report the Classification Accuracy Score (CAS) (Paschalidou et al., 2021). 1). Additionally, we compute the FID by rendering the populated layout from a top-down view using orthographic camera at resolution of 256 x 256. We report the FID computed between
8

---

Under review as a conference paper at ICLR 2023
Figure Outlier detection: Our model can utilize bidirectional attention reason about unlikely arrangements of furniture. We can then sample new attributes that create a more likely layout. In contrast, ATISS can only sample whole objects. Top row: An object is perturbed to create outlier (highlighted in blue). Bottom row: The object can be identified by its low likelihood, and new attributes sampled which place it more naturally.
these rendered top down images of sampled layouts and the renders of the ground truth layouts. Additional details are given in the supplementary.
Results are shown in Table 1. slightly superior to ATISS, with slightly inferior results in the CAS metric, comparable results in the FII) metrics, and more substantially improved results in the KL--divergence metric. Examples of unconditionally generated layouts are shown in the supplementary.
The results suggest that overall, COFS performs roughly on par or
Table 1:
Comparison on Unconditional Generation: We provide floorplan boundaries from the Ground Truth an input to the methods and compare the quality of generate layouts. We retrain the ATISS model and report metries. The retrained model is called ATTSS*
CAS (1)
KL Divergence x x10" (1)
FID (L)
BEDROOM
LIVING DINING LIBRARY REDROOM LIVING DINING LIBRARY
BEDROOM LIVING DINING LIBRARY
FastSynth
93.5
81.5
6.4
17.6
58.9
SceneFormer
94.5
97.2
94.1
88.0
5.2
31,3
36.8
23.2
90.6
68.1
60.1
89.1
ATTSS*
61.1
76.4
69.1
61.77
8.6
14.1
15.6
10.1
73.0
43.32
47.66
75,34
Ours
61.0
78.9
76.1
66.2
5.0
8.1
9.3
6.7
73.2
35.9
43.12
75.72
CONCLUSIONS
We proposed new framework to produce layouts with auto-regressive transformers with arbitrary conditioning information. While previous work was only able to condition on a set of complete objects, we extend this functionality and also allow for conditioning on individual attributes of objects. Our framework thereby enables several new modeling applications that cannot be achieved by any published framework.
I Limitations and Future Work. We now discuss limitations of our model. The first is related to our simple object retrieval scheme based only bounding box sizes. This often leads to stylistically different objects in close proximity even if the bounding box dimensions are only slightly different. We show such example in the inset (left).
The second is related to the training objective of the model we only consider the cross entropy/NLL. Thus, the network does not have explicit knowledge of design principles such non-intersection, or object co-occurrence. This means that the model completely relies on the data being high-quality to ensure such output. In the supplementary. we highlight the fact that certain scenes in the dataset have problematic layouts, and our method cannot filter them out. We show an example of intersections in the inset (center). Thirdly, the the performance on the LIVING and DINING datasets not as good as the other classes, which is clear from the CAS scores. This is in part because the datasets are small but also have significantly more objects than BEDROOM or LIBRARY. This leads to accumulated errors. We would like to explore novel sampling strategies to mitigate such errors. Lastly, while the conditioning works well, it is not guaranteed to generate a good layout. For example, in the inset (right), set the condition to be two beds opposite each other, but the network is unable to place them in valid locations. Adding explicit design knowledge would help mitigate such arrangements, but we leave that extension future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Huiwen Chang,
Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In The IEFE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
R. Qi Charles, Hao Su, Mo Kaichun, and Leonidas J. Guibas. Pointnet: for 3d classification and segmentation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77-85, 2017.
Deep learning point sets
doi: 10. N..119//VPP.201..00
Shuai Li Chenhang He, Ruihuang Li and Lei Zhang. Voxel set transformer: A set-to-set approach to 3d object detection from point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Jacob Devlin, Ming-Weiree Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4188 Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 10.18653/1//N11--4223. URL https: // //aclanthology org/N19-142
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), doi: 10.1109/CVVP44433...22...00
pp. 12868-12878, 2021.
Matthew Fisher, Daniel Ritchie, Manolis Savva, Thomas Funkhouser, and Pat Hanrahan. Example- based synthesis of 3d object arrangements. ACM Trans. Graph. (1(6), Nov 2012. ISSN 0730-0301, doi: 10.1 145/236614452222 25.6661144 URL https: ttw. org 1000011.0000 1441 100...1111/166110001 2366145.6555 2366154.
Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, and Hao Zhang. 3d-front: 3d furnished rooms with layouts and semantics. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. doi: 10.10000CCV489922222.000000
10913-10922, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
doi: 10. 10.1109/CVR.22016.990
Thibaut Issenhuth, Ugo Tanielian, Jérémie Mary, and David Picard. Edibert, a generative model for image editing. arXiv preprint arXiv:2111.1526 2021.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: framework for attention-based permutation-invarian neural networks. In Proceedings of the 36th International Conference on Machine Learning, pp. 3744-3753, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sseueeee pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 787 1881-7880, Online, July 2020. Association for Computational Linguistics. doi: 18.18863/v 006653/v1/22220 1/2020. acl- main. 703. URL https: / /ccl anthology orgoo/g//2222 acl20.cll--1 acl-main 703.
10

---

Under review as a conference paper at ICLR 2023
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1XX7::9977.11922 2019.
Elman Mansimov, Alex Wang, and Kyunghyun Cho. A generalized framework of sequence generation with application to undirected sequence models, 2020. URL https: httpp://///peeeeveew net/
forum?
Dror Moran, Hodaya Koslowsky, Yoni Kasten, Haggai Maron, Meirav Galun, and Ronen Basri. Deep permutation equivariant structure from motion. In 2021 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 5956-5966, 2021.
doi: 10.1 N..100CCC4499222.00 1.00992.
Wamiq Para, Paul Guerrero, Tom Kelly, Leonidas Guibas, and Peter Wonka. Generative layout modeling using constraint graphs. In 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6670-6680, 2021. doi: 10.1 N0.11009CCC449922222..000
Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos, Deep learning with sets and point clouds. arXiv preprint 1XXv:1611.005000 2016
Daniel Ritchie, Kai Wang, and Yu-An Lin. Fast and flexible indoor scene synthesis via deep convolutional generative models. In 2019 JEEE/CVF Conference on Computer Vision and Pattern doi: 10. 1ww.1099VVR...299.00000
Recognition (CVPR), pp. 6175-6183, 2019.
Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. Masked language model scoring. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 2699-2712, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.1863//v1/2020. acl-main. 240.
URL http:::tt///ooo / ac aallannnooooooo loyy org/ 2 2020 aacl-main ma.n.240..
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelenn++: A pixelenn implementation with discretized logistic mixture likelihood and other modifications. ICLR. 2017.
Nicolas Ugrinovic, Adria Ruiz, Antonio Agudo, Alberto Sanfeliu, and Francese Moreno-Noguer. commutation-invariann relational network for multi-person 3d pose estimation. arXiv preprint arXiv:2204.04913, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proceedings of the 31st International Conference Neural Information Processing Systems. NIPS 17, pp. 6000-6010, Red Hook, NY, USA, 2017, Curran Associates Inc. ISBN 1781510860964.
Alex Wang and Kyunghyun Cho. Bert has mouth, and must speak: Bert as markov random field language model. arXiv preprint arXiv: 190.29992.9944.. 2019.
Kai Wang, Yu-An Lin, Ben Weissmann, Manolis Savva, Angel X. Chang, and Daniel Ritchie. Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Trans. Graph., 38(4), jul 2019. ISSN 0730-0301. doi: 10.11457396344433224.. URL
patpss//doli www.oogg........... .1.0..110 111.1 /33/3333334663 3223344433322940 3322941.
Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner. Sceneformer: Indoor scene generation with transformers. In 2021 International Conference on 3D Vision (3DV), pp. 106-115. IEEE, 2021.
Tomer Weiss, Alan Litteneker, Noah Duncan, Masaki Nakada, Chenfanfu Jiang, Lap-Fai Yu, and Demetri Terzopoulos. Fast and scalable nosition-based layout synthesis. IEEE Transactions on Visualization and Computer Graphics, 25(12):3231-3243, 2019. doi: 10. 20.1.0.. 2866436.
11

---

Under review as a conference paper at ICLR 2023
Lap-Fai Yu, Sai-Kit Yeung, Chi-Keung Tang, Demetri Terzopoulos, Tony F. Chan, and Stanley J. Osher. Make it home: Automatic optimization of furniture arrangement. ACM Trans. Graph., 30 (4), Jul 2011. ISSN 0730-0301. doi: 00.114492010324.19699811. URL https://doi.org:
(0.1145/201032. 964981.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander Smola. Deep sets. In Guyon, U. Luxburg, S. Bengio, H. Wallach, Fergus. S. Vishwanathan, and R. Garnett Advances
Advances in Neural Information Processing Systems 30,
pp. 3391-3401. Curran Associates, Inc., 2017. URL http:/ hatp://paapee ppppppaa.......//p.ppppeaea nips. cc/ cc/paper/ @931-deeps
1.31-dest seppssets.... sff..
12


Reviews:
[{"review_id": "OPYfud2E2n", "reviewer": "Reviewer_iswK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper improves upon prior approaches that generates furnitures. The proposed method is more controllable, able to generate / complete a scene from either object-level conditioning (think rendered, physically placed objects) or attribute-level conditioning (think word, stylized language description of the scene). The evaluation is thorough. ", "summary_of_the_review": "I personally like this paper. It addresses a realistic problem, and it is well evaluated and solid. My only concern is this work might not be the right fit for ICLR. ", "strength_and_weaknesses": "strength: this paper is well presented, the method is clear to read, and the evaluation is solid. Using data likelihood and omitting either objects / attributes and having the model regenerating them is a good way to evaluate, and it performs better than the baseline. I especially liked the user study on realistic-ness, where we can see COFS generates more realistic scenes rated by end-users -- this is gold standard.\n\nweakness: the technical aspect of this work (by nature) isn't very novel. A naive interpretation is simply this model can be conditioned by a richer context (object + attribute) as opposed to objects alone. Therefore, one would like to see more works on down-stream usage of the system. The proposed system can be used as an augmented design tool, where designers can interact with the system to create, or re-create realistic scenes. This is more of a HCI / Siggraph aligned work, and I wonder if ICLR isn't quite the right fit for it. \n\nAn amazing user study would be give designers a picture, and ask them to re-construct it in 3D using your tool. How might this process look? I feel data on interacting with these generative tools are vastly lacking, but is extremely valuable as that is their ultimate use-case.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NFafC123q_R", "reviewer": "Reviewer_6pUr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an indoor scene generation algorithm that supports conditioning on a set of object attributes. By randomly permuting furniture objects at training time, the method is trained to be approximately invariant to object permutations. A transformer encoder is added to provide cross-attention over the complete conditioning information in each step. The experiments show that the proposed method generates realistic scenes in both conditional and unconditional cases.", "summary_of_the_review": "Overall I feel the paper is not fully ready considering its presentation and novelty. I would like to hear the authors' response on these comments.", "strength_and_weaknesses": "Strenghs:\n- The problem being studied is important. The authors point out the weakness of existing method that they are usually not flexible enough.\n- The generated scenes look realistic.\n\nWeaknesses:\n- The presentation of the paper could be improved. For example, in Sec 3.2, it is unclear whether each s_i corresponds to B_i in Sec 3.1. In Sec 3.3, it is unclear what is the exact form of C. \n- Overall, the method is close to ATISS, hence the novelty could be limited. As mentioned by the authors, ATISS can also generate scenes conditioning on object attributes by the replacement technique (Sec 4, baseline). The overall mechanism is similar, and the proposed method is also auto-regressive.\n- The performance of the method is close to ATISS in unconditional generations. For conditional generations, it is difficult to tell if the advantage of the proposed method still holds if ATISS also does attribute conditioning during training.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KmOw5LwFFU", "reviewer": "Reviewer_CiDE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose a model for generating layouts of furniture in rooms. Unlike existing works in this area (e.g. ATISS), the proposed method allows conditioning on combinations of attributes (e.g. \"place a chair somewhere, and some other object at this particular location\"). The proposed model is a novel combination of transformer encoder and decoder, allowing arbitrary attributes of arbitrary objects to be fixed, and retaining permutation invariance. Evaluation is conducted on the 3D-FRONT dataset, including various metrics and a perceptual user study.", "summary_of_the_review": "The paper addresses a clear gap in functionality of layout generation models, and proposes a sensible and novel approach. However, the empirical evaluation is currently marginally insufficient to convince me that the method indeed solves this problem well.", "strength_and_weaknesses": "- authors identify a potential weakness of existing furniture layout generation methods, and propose a system that addresses it\n\n- the proposed method is architecturally novel, utilising transformers like ATISS but in a significantly different configuration to support more sophisticated conditioning\n\n- unconditional samples from the proposed model are no worse that those from ATISS\n\n- attribute-conditioned samples have higher likelihood than those from ATISS, and the few qualitative examples given look reasonable\n\n- a user study finds that both conditioned and unconditioned samples are significantly preferred (by a small group of humans) vs those of ATISS\n\n- the qualitative results on fixing-of-outliers are a nice example of the power of this kind of approach\n\n- there are insufficiently many qualitative results (in paper and supplementary) to convince me of the quality of the generations. Indeed, it is not clear whether the qualitative examples (even in supplementary) are curated or random. This could be mitigated by providing (e.g.) 100 uncurated samples per room type from each method\n\n- the most critical results in the paper are those on attribute-level conditioning (since this is the main motivation, and the only feature that existing methods do not support). However, qualitative results are restricted to very few examples (fig. 5 + 6). This is insufficient evidence to support the key claims of the work. Moreover, some of these would be tractable with ATISS – either by rejection sampling (when the constraints do not restrict to an overly small subset of possible configurations), or (fig. 5, right) exhaustively finding the most-likely position, since there is only a single attribute and the likelihood is tractable. If rejection sampling ATISS in fact takes prohibitively long for fig. 5 (left) and fig. 6, then this should be stated and measured. I do however appreciate that the authors went to the effort of including a user study, including the attribute-conditional setting.\n\n- it is not clear to me how NLL results with attribute conditioning are calculated for the baseline ATISS (which doesn't support such conditioning, hence the need for the proposed method). Please clarify this.\n\n- there is no quantitative metric for outlier prediction / fixing; it seems like the former (at least) should be straightforward to measure quantitatively, roughly as a binary classification problem. This would avoid any impression that the figure just contains cherry-picked examples\n\n- the negative log-likelihood (NLL) results in fig. 3 are given only for one room type. The full results should be given for all rooms (maybe in supplementary), to give a fair and balanced view of how well the two models perform.\n\n- similarly, NLL is apparently only given when locations/sizes are conditioned on, not classes. This is unfortunate since I think a more likely practical application is (e.g.) \"create a layout with three chairs\", rather than \"create a layout with arbitrary objects at these particular locations\". Please consider adding these results. Also, quantitative evaluation in more diverse conditioning scenarios would be valuable – e.g. mixtures of position and class conditioning.\n\n- fig. 7 should compare against ATISS, since that is noted to support the object-level conditioning task (i.e. completing partial layouts). In this case, a handful of uncurated samples from each method should be given\n\n- the introduction clearly states that attribute-conditional generation is impossible (or at best very computationally expensive) with existing methods like ATISS. However, it doesn't give a convincing motivation of why attribute-conditioned generation is actually useful – i.e. in what practical use-cases one might have a complex constraint on classes and/or positions, but one wouldn't simply build the scene by hand\n\n- some of the cited 'pre deep learning' methods do in fact support the kind of conditioning described in the paper. In particular, it is straightforward within an MCMC sampling scheme to fix certain variables (position of one object, etc.), resulting in a posterior sample that correctly accounts for this constraint. This should be made clear, as currently the paper reads like it is the first to support such conditioning.\n\n- fig. 5 caption advertises the fact that the sampled objects are of a style that matches the ground-truth. However, from the method description this must simply be a coincidence, since the method only predicts bounding boxes, not particular object shapes / instances! Also, there aren't actually any chairs in the left scene, so far as I can see – maybe this comment refers to the right-hand one?\n\n- as noted in the limitations section, and in common with related works, the method is limited in terms of output quality by the quality of the ground-truth dataset (3D-FRONT), which is well known for containing object intersections and other issues. While it may seem unfair to raise this as a criticism of the present work, it means that generated scenes are not suitable for most practical uses – a fact that has to weigh slightly negatively given the rather 'applied' focus of the paper.\n\n- there does not appear to be any analysis of whether the model is memorising training layouts. This should be measured, given that relatively small size of 3D-FRONT compared with the relatively large transformer model (I appreciate it's smaller than ATISS, which is a good thing, but still there are millions of parameters, and overfitting is a potential issue). This could be resolved fairly easily by doing a search for nearest neighbors (in the training set) to a large set of sampled layouts.\n\n- there is no test of statistical significance for the user study results. This is particularly problematic given the small number of participants. Relatedly, for the \"326 responses\" in the user study, is this 326 individual comparisons, or 326 x 24 comparisons, or something else?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G1OU1XPRtX", "reviewer": "Reviewer_J17x", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends existing transformer-based autoregressive indoor scene synthesis method by introducing a new conditioning mechanism that allow the input sequence to contain, at any location, an arbitrary number of masked tokens, whose values are to be predicted. Three set of positional encodings are introduced to enable this form of conditioning: object index and attribute types for the encoder, and global absolute positions for the decoder. It is shown that the proposed method performances better than prior works, especially with respect to human perceptual studies. Limited discussions are provided that demonstrate that the proposed framework is more flexible thanks to the conditioning mechanism.", "summary_of_the_review": "I have mixed feeling about this work. On one side, I am impressed by the quality of the results and really like the amount of flexibility provided by this new conditioning formulation. On the other side, the amount of novelty in this work is rather limited - both the transformer based scene synth part and the set generation / arbitrary conditioning part. My rating of this paper would have been higher if the authors provide more fine-grained analysis on what *exactly* does the new ideas bring. However, since the current evaluation focuses on overall quality, which does not necessarily origin from the novel (in my mind) part of the paper, I am hesitant to recommend acceptance and lean towards rejection for now.\n\n=====Post Rebuttal Comment=====\n\nThanks for the response to my concerns. I appreciate the clarification regarding the experiment setup and the positional tokens. The additional experiments do showcase the strength of this method, though I still don't think it showcases scenarios where previous method *cannot* work.\n\nI thus still am quite ambivalent about this paper. However, after the reviewer meeting, I do agree with the AC and other reviewers that audiences of this conference can get some inspiration should this work be accepted. I was a bit hesitant in arriving at this conclusion because I might be quite biased as I have worked on related problems for a long time. However, the discussion did sway me slightly towards the acceptance side and I thus raise my score from a 5 to a 6. I still won't champion for this paper, and am fine with it going either way.", "strength_and_weaknesses": "# Strengths\n- Simple yet effective idea that extends existing autoregressive scene synthesis models and enable more flexible applications.\n- Good empirical performance: although the performance w.r.t the three distributional similarity metrics is not too much different from ATISS, the perceptual study results are much better. I definitely think the human perceptual study tells much more in this case.\n\n# Weaknesses\n### Novelty\n- Amount of novelty is rather limited, both on the scene synth side and on the set generation side. \n\n### Motivation & Design Choices\n- Some additional discussions regarding the positional encoding tokens would be nice. I get that they helps training, as shown in the supplementary, but I am unsure what's the intuition behind these, especially the absolute position tokens on the decoder side.\n- I think it might make sense to highlight the difference between \"what was not possible with prior work\" versus \"what was possible, but only in a very complicated way\". I get that the conditioning in this work can be very flexible, but it appears to me that most of the stuff can also be accomplished by prior works, by following a \"traditional\" sequence and hardcode some of the info when needed. Would be more helpful if the authors can highlight something unique that only this work can do, and showcase these a bit more.\n\n### Evaluation\n- My main issue lies with the set of evaluations the authors chose to perform. In the case of this work, where main novelty lies in the ability to enable more flexible forms of conditions, I do not think overall scene quality is the right metric to use here. Granted, these metrics are important as they show that the method generates reasonable outputs, they do no highlight the benefits of the new ideas in this paper as much. There's a few qualitative results in the supplementary that showcases such flexibility, however, I think it more evaluations on this respect are needed. A few ideas here: 1. visualize the distributions for a single token between ATISS and COFS, and show that the extra conditioning of COFS actual makes these distributions more reasonable; 2. A user study that task someone to edit a scene given a natural language instruction, and hopefully demonstrates that COFS allows a better editing experience; 3. A more fine-grained quantitative analysis showing that performance of ATISS decreases further when the form of conditioning is more complex/irregular.\n- I am slightly concerned with the perceptual study setup - judging from Fig 2 in the supplementary, it seems that it might be hard for the participants to make good decisions due to the choice of camera angles and the rendering quality.\n\n### Misc\n- Figure 5: \"even matches the styles of the chairs\": IIUC the models are retrieved based on bbox dimensions, so a claim regarding styles is probably not the best one here? (it's probably a byproduct of predicting a bbox of exactly the right size?)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject