Multimodal Attention for Layout Synthesis in Diverse Domains
Authors: Kamal Gupta
Vijay Mahadevan
Alessandro Achille
Justin Lazarow
Larry S. Davis
Abhinav Shrivastava
Venue:  (2021)
Keywords: layout generation
layout synthesis
multimodal attention
transformers
document layouts
generative model
3D

Abstract: We address the problem of scene layout generation for diverse domains such as images, mobile applications, documents and 3D objects. Most complex scenes, natural or human-designed, can be expressed as a meaningful arrangement of simpler compositional graphical primitives. Generating a new layout or extending an existing layout requires understanding the relationships between these primitives. To do this, we propose a multimodal attention framework, MMA, that leverages self-attention to learn contextual relationships between layout elements and generate novel layouts in a given domain. Our framework allows us to generate a new layout either from an empty set or from an initial seed set of primitives, and can easily scale to support an arbitrary of primitives per layout. Further, our analyses show that the model is able to automatically capture the semantic properties of the primitives. We propose simple improvements in both representation of layout primitives, as well as training methods to demonstrate competitive performance in very diverse data domains such as object bounding boxes in natural images (COCO bounding boxes), documents (PubLayNet), mobile applications (RICO dataset) as well as 3D shapes (PartNet).

Under review as a conference paper at ICLR 2021
MULTIMODAL ATTENTION FOR LAYOUT SYNTHESIS IN DIVERSE DOMAINS
Anonymous authors Paper under double-blind review
ABSTRACT
We address the problem of scene layout generation for diverse domains such as images, mobile applications, documents and 3D objects. Most complex scenes, natural or human-designed can be expressed as a meaningful arrangement of sim- pler compositional graphical primitives. Generating new layout or extending an existing layout requires understanding the relationships between these primitives. To do this, we propose multimodal attention framework, MMA, that leverages self-attention to learn contextual relationships between layout elements and gen- erate novel layouts in a given domain. Our framework allows us to generate a new layout either from an empty set or from an initial seed set of primitives, and can easily scale to support an arbitrary of primitives per layout. Further, our analyses show that the model is able to automatically capture the semantic properties of the primitives. We propose simple improvements both representation of layout primitives, well training methods to demonstrate competitive performance in very diverse data domains such as object bounding boxes in natural images (COCO bounding boxes), documents (PubLayNet), mobile applications (RICO dataset) as well as 3D shapes (PartNet).
INTRODUCTION
In the real world, there exists strong relationship between different objects that are found in the same environment (Torrllba & Sinha 2001 Shrivastava & Gupta, 2016). For example, dining table usually has chairs around it, surfboard is found near the sea, horses do not ride cars, etc. Biederman (017) provided strong evidence in cognitive neuroscience that perceiving and under- standing a scene involves two related processes: perception and comprehension. Perception deals with processing the visual signal or the appearance of a scene. Comprehension deals with under- standing the schema of a scene, where this schema (or layout) can be characterized by contextual relationships (e.g., support, occlusion, and relative likelihood, position, and size) between objects. For generative models that synthesize scenes, this evidence underpins the importance of two factors that contribute the realism plausibility of generated scene: layout, i.e., the arrangement of different objects, and their appearance (in terms of pixels). Therefore, generating realistic scene necessitates both these factors to be plausible.
The advancements in the generative models for image synthesis have primarily targeted plausibility of the appearance signal by generating incredibly realistic images often with single entity such as faces (Karras et al. 2019 (017), or animals (Brock et al.) 2018 Zhang et al. 2018). In the case of large and complex scenes, with a lot of strong non-local relationships between different elements, most methods require proxy representations for layouts to be provided as inputs (e.g., scene graph, segmentation mask, sentence). We argue that to plausibly generate large and complex scenes without such proxies, it is necessary to understand and generate the layout of scene, in terms of contextual relationships between various objects present in the scene.
The layout of a scene, capturing what primitives occupy what parts of the scene, is incredibly rich representation. Learning to generate layouts itself a challenging problem due to the variability of real-world human-designed layouts. Each layout is composed of small fraction of possible objects, objects can be present a wide range of locations, the number of objects varies for each scene and do the contextual relationships between objects.

---

Under review as a conference paper at ICLR 2021
Sufficard
(e) Auteragrasivve 2D layoui geveratinn and dowrstresm Loynnth---mgg- applictioon
Goventing 3D objects antreerassively
Figure I: Our framework can synthesize layouts in diverse natural as well as human designed data domains such as natural scenes or 3D objects in sequential manner.
Formally, scene layout can be represented as an unordered set of graphical primitives. The primi- tive itself can be discrete or continuous depending on the data domain. For example, in the case of layout of documents, primitives can bounding boxes from discrete classes such as 'text', 'image', or 'caption', and in case of 3D objects, primitives can be 3D occupancy grids of parts of the object such as "arm', 'leg'. or 'back' in case of chairs. Additionally, in order to make the primitives com- positional, we represent each primitive by a location vector with respect to the origin, and scale vector that defines the bounding box enclosing the primitive. Again, based on the domain, these location and scale vectors can be 2D 3D. A generative model for layouts should be able look at all existing primitives and propose the placement and attributes of a new one. We propose a novel Multimodal Attention framework (MMA) that first maps the different parameters of the primitive independently to fixed-length continuous latent vector, followed by masked Transformer decoder to look representations of existing primitives in layout and predict the next parameter. Our gen- erative framework can start from an empty set, or set of primitives, and can iteratively generate new primitive one parameter at a time. Moreover, by predicting either to stop or generate the next primitive, our sequential approach can generate variable length layouts.
Our approach can be readily plugged into scene generation frameworks (e.g.. Layout2lmage (Zhao et al. 2019), GauGAN (Park et al. 2019b)) or stand-alone applications that require generating lay- outs or templates with/without user interaction. For instance, the UI design of mobile apps and websites, automated model for generating plausible layouts can significantly decrease the manual effort and cost of building such apps and websites. Finally, a model to create layouts can potentially help generate synthetic data for various tasks tasks (Yang al., 2017 Capobianco & Marinai 2017 Chang et al. 2013] Wu et al. 2017b]a).
To the best of our knowledge, MMA is the first framework to perform competitively with the state- of-the-art approaches in 4 diverse data domains. We evaluate our model using existing metrics proposed for different domains such as Jensen-Shannon Divergence, Minimum matching distance, and Coverage in case of 3D objects, Inception Score and Fréchet Inception Distance for COCO, and Negative likeelildooo of the test set in case of app wireframes and documents. Qualitative analy- sis of the framework also demonstrates that our model captures the semantic relationships between objects automatically (without explicitly using semantic embeddings like word2vec Mikolov et al. 2013)
2 RELATED WORK
Generative models. Deep generative models based CNNs such as variational auto-encoders (VAEs) (Kingma& & Welling 2013). and generative adversarial networks (GANs) (Goodfellow et al.)
2014) have recently shown great promise in terms of faithfully learning given data distribution and sampling from it. There has also been research on generating data sequentially (Oord et al.)
2016 Chen et al.] (2020) even when the data has no natural order (Vinyals et al.] (2015). Many of these approaches often rely on low-level information (Gupta et 20. such as pixels while generating images (Brock al., 20188 Karras et al. 2019), videos (Vondrick et al. 2016), or 3D objects (Wu et al.) 2016; Yang et al. 2019 Park et al.) 2019a Gupta et al. 2020a) and not on semantic and geometric structure in the data.
Scene generation. Generating 2D or 3D scenes conditioned on sentence (Li et al. 2019dd Zhang al. 2017 Reed af., 2016), ) scene graph (Johnson et al., 2018 Li et al. 2019a] Ashual & Wolf 2019). layout (Dong et al.) 2017 Hinz al.) 2019, Isola et al. 2016 Wang et al.) (018b) or an existing image (Lee et al. 2018) has drawn a great interest in vision community. Given the input, some works generate fixed layout and diverse scenes (Zhao et al. 2019), while other works generate diverse layouts and scenes @Jhnsson et al. 2018 Li et al. 2019d). These methods involve pipelines often trained and evaluated end-to-end, and surprisingly little work has been done to eval- uate the layout generation component itself. Layout generation serves as a complementary task to these works and can be combined with these methods. In this work, we evaluate the layout modeling
2

---

Under review as a conference paper at ICLR 2021
Input Sequence
Latents
Multimodal Masked Attention
Oulput
Feed-forward
Predictions
(bos)
E
0oos
ioca
Pos
(sky)
sky
(sky)
E
++
81
01
0
x: (0)
2: (0)
E
++
02
02
0₂
(0)
(0)
E
0g
0g
0g
h: (16)
Softmax
h: (16)
E
04
04
T
04
(32)
(32)
05
0s
0g
(grass))
(grass)
8
0.
0g
sky
person
6... 0..
0
(eos)
gress
person
L
Figure 2: The architecture for MMA depicted for a toy example. It takes layout elements as input and predicts the next layout elements as output. During training, we use teacher forcing. iee, use the ground-truth layout tokens as input to a multi-head decoder block. The first layer of this block is a masked self-attention layer, which allows the model to see only the previous elements in order to predict the current element. We pad each layout with special (bos) token in the beginning and (cos) token in the end. To generate new layouts, perform nucleus sampling starting with just the (bos) token or a partial sequence.
capabilities of two of the recent works (Johnson al. 2018 et al. 2019d) that have layout gener- ation as intermediate step. We also demonstrate the results of our model with Layout21m (Zhao ct al. 2019) for image generation.
Layout generation. The automatic generation of layouts is important problem in graphic design. Many of the recent data diiten approaches use data specific constraints order to model the layouts. For example, WWang et al. (2018a) 2019): Li et al. (2019c); Ritchie et al. (2019) generates top- down view indoor rooms layouts but make several assumptions regarding the presence of walls, roof etc., and cannot be easily extended to other datasets. In this paper,
focus approaches that have fewer domain-specifii constraints. LayoutGAN (Li et al., (209b)) uses a GAN framework to generate semantic and geometrie properties of fixed number of scene elements. LayoutVAE (Jyothi et al., (019)) starts with a label set, i.e., categories of all the elements present in the layout, and then generates a feasible layout of the scene. Zheng et al. (209)) attempt to generate document layouts given the images, keywords, and category of the document. Patil et al. (2019) proposes method to construct hierarchies of document layouts using a recursive variational autoencoder and sample new hierarchies to generate new document layouts. Manandhar et al. (2020) develops an auto-encoding framework for layouts using Graph Networks. 3D-PRNN (Zou et al., (017). PQ-
Net (Wu et al.) 2020) and ComplementMe [Sung et al. (2017), generates 3D shapes sequential part assembly. While 3D-PRNN generates only bounding boxes, PQ-Net and ComplementMe can synthesize complete 3D shapes starting with partial or input shape.
Our approach offers several advantages over current layout generation approaches without sacrific- ing their benefits. By factorizing primitives into structural parameters and compositional geometric parameters, we can generate high righ-resolution primitives using distributed representations and con- sequently, complete scenes. The autoregressive nature of the model allows us to generate layouts of arbitrary lengths as well as start with partial layouts. Further, modeling the position and size of primitives as discrete values (as discussed in 13.1) helps us realize better performance on datasets, such as documents and app wireframes, where bounding boxes of layouts are typically axis-aligned. We evaluate our method both quantitatively and qualitatively with state-of-the-art methods specific to each dataset and show competitive results in very diverse domains.
OUR APPROACH
In this section, we introduce our attention network in the context of the layout generation problem. We first discuss our representation of layouts for primitives belonging to different domains. Next,
3

---

Under review as a conference paper at ICLR 2021
discuss the Multimodal Attention (MMA) framework and show how leverage previous advances such as Transformers (Vaswani et al. 2017) to model the probability distribution of lay- outs. MMA allows to learn non-local semantic relationships between layout primitives and also gives us the flexibility to work with variable length layouts.
3.1 LAYOUT REPRESENTATION
Given dataset of layouts, a single layout instance can be defined as graph & with nodes, where each node E {l....,y n is a graphical primitive. We assume that the graph is fully-connected, and let the attention network learn the relationship between nodes. The nodes can have structural or semantic information associated with them. For each node, we project the information associated with it d dimmessionl space represented by feature vector s;. Note that the information itself can be discrete (e.g.- part category), continuous (e.g., color), or multidimensional vectors (e.g., signed distance function of the part) on some manifold. Specifically, in our ShapeNet experiments, we use MLP to project part embedding to fimaee-dimensional space, while in the 2D layout experiments, we use a learned dmodel dimensional category embedding which is equivalent to using an MLP with zero bias to project one-hot encoded category vectors to the latent space.
Each primitive also carries geometric information g: which we factorize into a position vector and a scale vector. For the layouts in IR2 such as images documents, gi [xta, ys, h4, Wz], where (ar, y) are the coordinates of the centroid of primitive and (h, w) are the height and width of the bounding box containing the primitive, normalized with respect the dimensions of the entire layout.
Representing geometry with discrete variables. We apply an 8-bit uniform quantization on each of the geometric fields and model them using Categorical distribution. Discretizing continuous signals is a practice adopted in previous works for image generation such as PixeICNN+ (Salimans et al. 2017), however, the best of our knowledge, has been unexplored in the layout modeling task. We observe that even though discretizing coordinates introduces approximation errors, it allows us to express arbitrary distributions which we find particularly important for layouts with strong symmetries such as documents and app wireframes. We project each geometric field of the primitive independently to the same domoea-dimension such that jth primitive in IR² can be represented as (st, Xi, yi, h,, w:). We concatenate all the elements in a flattened sequence of their parameters. We also append embeddings of two additional parameters (bos) and S(enc) to denote start and end of sequence. Our layout in ?R? can now be represented by sequence of 5n + 2 latent vectors
g = (ho))s) $1; X1; Y1; ;;; h Wi;.. Yn; V hn; Wn ((os)
For brevity, we use ,,. j f1, 5n + 2} to represent any element in the above sequence. We can now pose the problem of modeling this joint distribution as product over series of conditional distributions using chain rule:
5n-+2
= II
p(0,10 1-j-1)
3.2 MODEL ARCHITECTURE AND TRAINING
Our overall architecture is surprisingly simple and shown in Fig. 2 Given initial set of K visible primitives (where K can be 0 when generating from scratch), our attention based model takes as input, random permutation of the visible nodes, of dimess-dimensoonal vectors (0I...., 0sk). ). We find this to be an important step since by factoriz- ing primitive representation into geometry and structure fields, our attention module can explicitly assign weights to individual coordinate dimensions. The attention module is similar to Transformer Decoder (Vaswani al., 2017) and consists of L attention layers. each of which consists of (a) a masked multi-head attention layer (halm)). ( and (b) fully connected feed forward layer (h"*), Each sublayer also adds residual connections (He et al. (2166) and LayerNorm (Ba et al. (016).
(II,.... Tx) ) and consequently a sequence
0 LayerNorm 6) + h (0
2n42))
8/ = LayerNorm 0, + (0)))
(3)
where denotes the layer index. Masking is performed such that @ only attends to all the input latent vectors well previous predicted latent vectors. The output at the last layer corresponds to next
4

---

Under review as a conference paper at ICLR 2021
parameter. At training and validation time, teacher forcing, i.e., instead of using output of previous step, we use groundtruth sequences to train our model efficiently.
Loss. We use softmax layer to get probabilities if the next parameter is discrete. Instead of using a standard pross-entrppy loss, we minimize KL-Divergence between softmax predictions and output one-hot distribution with Label Smoothing (Szegedy et al. 2016), which prevents the model from becoming overconfident. the next parameter is continuous, we use an loss.
L = Disc. DEL (SofMMaxx (0)) p(0')) XCm |10 - 0'11
(4)
3D Primitive Auto-encoding, PartNet dataset Yu et al. 2019) consists of 3D objects decomposed into simpler meaningful primitives, such as chairs are composed of back, arms, 4 legs, and so on. We pose the problem of 3D shape generation as generating a layout of such primitives. We use Chen & Zhang 2 ' approach to first encode voxel-based represent of primitive to dimass-dimensional latent space using 3D CNN. An MLP based implicit parameter decoder projects the latent vector to the surface occupancy grid of the primitive.
Order of primitives. One of the limitations of an autoregressive modeling approach is that sequence of primitives is an important consideration, in order to train the generative model, even if the layout doesn't 't have natural defined order Vinyals et al. (2015). To generate layout from any partial layout, we use a random permutation of primitives as input to the model. For the output, we always generate the sequences in raster order of centroid of primitives, ie.,, order the primitives in ascending order of their (2r, y. 2) coordinates. In our experiments, we observed that the ordering of elements is important for model training. Note that similar limitations
faced by contemporary works in layout generation (Jyothi et al.) [2019] Li et al.] 2019d; Hong et al. 2018 [Wang et al.) 2018a), image generation (Salimans et al.; 2017 Gregor et al.) 2015) and 3D shape generation (Wu et al.) 2020; Zou et al., 2017). Generating a distribution over an order- invariant set of an arbitrary number of primitives is an exciting problem and we will explore it in future research.
Other details. In our base model, we use dmodel = 512, L = 6, and Tlaad = 8 (number of multi- attention heads). Label smoothing uses an c = 0.1, and À 1. We use Adam optimizer (Kingma & Ba 2014) with 31 0.9, 32 0.99 and learning rate 10-4 (10-5 for PartNet). We use early stopping based on validation loss. In the ablation studies provided in $ B we show that our model is quite robust to these choices, as well other hyperparameters (layout resolution, ordering of elements, ordering of fields). To sample a new layout, we can start off with just a start sequence embedding or initial set of primitives. Several decoding strategies are possible to recursively gen- crate primitives from the initial set. In samples generated for this work, unless otherwise specified, we have used nucleus sampling (Holtzman al. (2019), with top-p = 0.9 which has been shown to perform better as compared to greedy sampling and beam search (Steinbiss et al.) 1994).
4 EXPERIMENTS
In this section, we discuss the qualitative and quantitative performance of our model on different datasets. Evaluation of generative models hard, and most quantitative measures fail in providing a good measure of novelty and realism of data sampled from generative model. We will use dataset- specific quantitative metrics used by various baseline approaches and discuss their limitations wher- ever applicable. We will provide the code and pretrained models to reproduce the experiments.
4.1
3D SHAPE SYNTHESIS (ON PARTNET DATASET)
PartNet is a large-scale dataset of common 3D shapes that are segmented into semantically mean- ingful parts. We use two of the largest categories of PartNet Chairs and Lamp. We voxelize the shapes into 64 and train autoencoder to learn part embeddings similar to the procedure followed by PQ-Net (Wu et al. 2020). Overall, we had 6305 chairs and 1188 lamps in our datasets. We use the official train, validation, and test split from PartNet in experiments. Although it is fairly triv- ial to extend our method to train for the class conditiona generation of shapes, in order to compare with baselines fairly, we train separate models for each of the categories.
Generated Samples. Fig. showws examples of shape completion from the PartNet dataset. Given a random primitive, use our model to iteratively predict the latent shape encoding of the next part,
5

---

Under review as a conference paper at ICLR 2021
Figure 3: Generated objects. Top row shows input primitives to the model. Bottom row shows layout
obtained with greedy decoding.
Table I: Evaluation of generated shapes in Chair category.
Method
JSDL
MMD (CD)
MMD (EMD)
Cov? (CD)
Covt
1-NNAL
I-NNAL
(MMD)
(CD)
(MMD)
PointFlow Yang et al.
2019
1.74
2.42
7.87
46.83
46.98
60.88
59.89
StructureNet Mo 2019
4.77
0.97
15.24
29.67
31.7
75.32
74.22
IM-Net
Chen Zhang
2019
0.84
0.74
12.28
52.35
54.12
68.52
67.12
PQ-Net (Wu et al. 2020
0.83
0.83
14.16
54.91
60.72
71.31
67.8
Ours
1.50
1.92
7.38
55.25
55.44
60.67
60.46
as well its position and scale in 3D. We then use the part decoder to sample points the surface of the object. For visualization, we use the marching cubes algorithm to generate a mesh and render the mesh using fixed camera viewpoint.
Quantitative Evaluation. The output of our model point clouds sampled on the surface of the 3D shapes. We use Chamfer Distance (CD) and Earth Mover's Distance (EMD) to compare two point clouds. Following prior work, we use different metrics to compare the distribution of shapes generated from the model and shapes in the test dataset: (i) Jensen Shannon Divergence (JSD) computes the KL divergence between marginal distribution of point clouds in generated set and test set, (ii) Coverage (Cov)- compares the distance between each point generated set to its nearest neighbor in test set, (iii) Minimum Matching Distance (MMD) computes the average distance of each point in test
its nearest neighbor generated set, and (iv) 1-nearest neighbor accuracy (1-NNA) uses I-NN classifier see if the nearest neighbor of a generated sample is coming from generated set or test set. Our model performs competitively with existing approaches to generate point clouds.
4.2 LAYOUTS FOR NATURAL SCENES (COCO BOUNDING BOXES)
COCO bounding boxes dataset is obtained using bounding box annotations in COCO Panoptic 2017 dataset (Lin et al. 2014). We ignore the images where the isCrowd flag is true following the Lay- outVAE (Jyothi et al. 2019) approach. The bounding boxes come from all 80 thing and 91 stuff categories. Our final dataset has 118280 layouts from COCO train split with a median length of 42 elements and 5000 layouts from COCO valid split with a median length of 33. We use the of- ficial validation split from COCO as test set our experiments, and use 5% of the training data as validation.
Baseline Approaches. We compare our work with previous methods. LayoutGAN (Li et al., 2019h) is GAN based layout generation framework, starting with noise vector sampled from gaussian distribution generate bounding box layours. Since the method always generate fixed number of bounding boxes, uses non mon-maximum suppression (NMS) remove duplicates.
LayoutVAE (Jyothi et al. 2019) uses consists two separate autoregressive VAE models. The method assumes categories of elements present in generated layout to be known. First, CountVAE generates counts of each of the elements of the label set. and then Bounding Box VAE, generates the location and size of each occurrence of the bounding box. ObjGAN (Li et al. 2019d) is GAN framework for text to image synthesis. An intermediate step in their image synthesis approach is generate bounding box layout given sentence using BIL (trained independently). We

---

Under review as a conference paper at ICLR 2021
Ours LayoutVAE Real +121m + L2mm +L2mm
Iepust Laynun
Completed Lagout
Figure 5: Downstream task. Image genera-
tion with layouts (Zhao et al. 2019).
Figure 4: Generated layouts. Top row shows seed layouts input to the model. Bottom row shows the layout obtained with greedy decoding. We skip the 'stuff" bounding boxes for clarity.
adopt this step the ObjGAN framework to our problem setup by provide categories of all layout elements as input to the ObjGAN and synthesize all the elements' bounding boxes. sg2im (Johnson jet al.] 2018)) attempts to generate images given scene graph of the image by first generating a layout of the scene using graph convolutions and then using the layout to generate complete scene using GANs. Since sg2im requires a scene graph input, following the approach of (Jyothi et al. (2019), we create scene graph from the input and reproduce the input layout using the scene graph.
Since Layout and LayoutGAN are not open source, we implemented our own version of these baseline. Note that, like many GAN models, LayoutGAN was notoriously hard train and our im- plementation (and hence results) might differ from author's implementation despite our best efforts. We were able reproduce LayoutVAE's results COCO dataset as proposed in the original paper and train our own models for different datasets using the same hyperparameters. We also re-purpose ObjGAN and sg2im using guidelines mentioned Layout VAE. Although evaluating generative models is challenging. we attempt to do a fair comparison to the best of our abilities. For our model (and others), we keep architecture hyperparameters same across the datasets. We also train different baselines for same number of epochs in corresponding datasets. Some of the ablation studies are provided in the appendix.
Generated Samples. Fig- 4 shows layout completion task using our model COCO dataset. Although the model is trained with all 171 categories, in the figure we only show 'thing' categories for clarity. We also use the generated layouts for a downstream application of scene generation (Zhao ctt al. 2019).
Semantics Emerge via Layout. We posited earlier that capturing layout should capture contextual relationships between various elements. We provide further evidence of our argument in Fig. 6 We visualize the 2D-tsne plot of the learned embeddings for categories. We observe that super- categories from COCO are clustered together in the embedding space of the model. Certain cate- gories such window-blind and curtain (which belong to different super-categories) also appear close to each other. These observations are in line with observations made by Gupta et al. (2019) who use visual co-occurence learn category embeddings.
Quantitative evaluation. Following the approach of LayoutVAE, we compute negative log- likelihoods (NLL) of all the layouts in validation data using importance sampling. NLL approach is good for evaluating validation samples, but fails for generated samples. Ideally, we would like to evaluate the performance of generative model on a downstream task. To this end, we employ Lay- out2Im (Zhao et al. 2019) to generate an image from the layouts generated by each of the method. We compute Inception Score (IS) and Fréchet Inception Distance (FID) to compare quality and diversity of generated images. Our method is competitive with existing approaches in both these metrics, and outperforms existing approaches in terms of NLL.
Note that ObjGAN and LAy are conditioned on the label set. So provide labels of objects present in the each validation layout as input. The task for the model is to then predict the number and postition of these objects. Hence, these methods have unfair advantage over method and ObjGAN indeed performs better than our method and LayoutGAN, which are unconditional. We clearly outperform LayoutGAN IS and FID metrics.

---

Under review as a conference paper at ICLR 2021
Seccac rodB
Figure Quantitative Evaluations on COCO. Negative log oge-likeihhoo (NLL) of all the layouts in the validation set (lower the better). We use the importance sampling ap- proach descrihed in [yothi al. (2019) compute. We also generated images layout using Zhao et al.] (2019) and com- pute IS and FID. Following Johnson al.) 2018), randomly split test samples groups and report standard deviation across the splits. The mean is reported using the combined test set.
Tor
coma?
ButerCatogey
accestory
elocrono
celing
sold
pareen
2207a
applaace
LayeuriAN
2019h
3.2 (0)
LayorVALE MOUNE OsGGAN
Figure 6: TSNE plot of learned category embeddings. Words are colored by their super-categories provided in the COCO. Observe that semantically similar categories cluster together. Cats and dogs are closer as compared to sheep, zebra, or cow.
2g2im
3.3.8155)
7.1.0130) (3.5
Figure RICO layouts. Layouts obtained with greedy decoding for the RICO dataset. We skip the categories of bounding boxes for the sake of clarity.
Figure 9: Document Layouts. Generated samples VAE (top) and our method (bottom). Our method produces aligned bounding boxes for various elements.
4.3 MOBILE APP WIREFRAMES (RICO) AND DOCUMENT LAYOUTS (PUBLAYNET)
Rico Mobile App Wireframes. Rico mobile app dataset (Deka et al. 2017 Liu al. 2018) consists of layout information of more than 66000 unique UI screens from over 9300 android apps. Each layout consists of one or more of the 25 categories of graphical elements such text, image, icon elc. A complete list of these elements is provided in the supplementary material. Overall, we get 62951 layouts in Rico with a median length of 36. Since the dataset doesn't 't have official splits, use 5% of randomly selected layouts for validation and 15% for testing.
PubLayNet. PubLayNet (hhon al. 2019) is large scale document dataset consisting of over 1.1 million articles collected from PubMed Central. The layouts are annotated with 5 element categories text, title, list, label, and figure. We filter out the document layouts with over 128 elements. Our final dataset has 335703 layouts from PubLayNet train split with median length of 33 elements and 11245 layouts from PubLayNet dev split with a median length of 36. We use the provided dev split as our test set and 5% of the training data for validation.
Generated layout samples. Fig. g and 9 shows some of the generated samples of our model from RICO mobile app wireframes and PubLayNet documents. Note that both the datasets share similarity in terms of distribution of elements, such as high coverage in terms of space, very little collision of elements, and most importantly alignment of the elements along both and y-axes. Our method is able to preserve most these properties as we discuss in the next section. Fig. 10 shows multiple completions done by model for the same initial element.
Comparison with baselines. We use the same baselines for evaluation as discussed previously in Fig. shows that method able preserve alignment between bounding boxes better
4.2
8

---

Under review as a conference paper at ICLR 2021
Intial Layout
Completion
Comolction
Completion
Figure 10: Multiple completions from same initial element
Table 2: Spatial distribution analysis for the samples generated using model trained on RICO and PubLayNet dataset. Closer the Overlap and Coverage values to real data, better is the performance. All values in the table percentages (std in parenthesis)
RICO
PubLayNet
NLLL
Coverage
Overlap
NL.4
Coverage
Overlap.
sg2im Johnson et
2018
7.43
25.2 (46)
16.5
7.12
30.2 (26)
3.2)
ObjGAN 2019
4.21
39.2 (33)
36.4 (29)
4.20
38.9
8.2
Vay Tyathi
2019
2.54
41.5 (29)
34.1
2.45
40.1
(415
LayoutGAN Li 20196
37.3 (31)
3124
45.3 (19)
810)
Ours
1.07
33.6 (27)
23.7 (33)
1,10
(12)
01.55
Real Data
36.6 (27)
22.4
57.1 (10)
0.1 (0.6)
than competing methods. Note that haven't used any post-processing in order to generate these layouts. Our hypothesis is that (I) discretization of size/position, and (2) decoupling geometric fields in the attention module, are particularly useful in datasets with aligned boxes.
To measure this performance quantitatively, we introduce important statistics. Overlap represents the intersection over union (IoU) various layout elements. Generally in these datasets, elements do not overap with each other and Overlap is small. Coverage indicates the percentage of canvas covered by the layout elements. Table shows that layouts generated by our method resemble real data statistics better than LayoutGAN and LayoutVAF.
5 CONCLUSION.
We propose MMA, multimodal attention framework to generate layouts of graphical elements. Our model uses self-attention model to capture contextual relationship between different layout elements and generate novel layouts, or complete partial layouts. We show that our model performs competitively with the state-of-the-art approaches for very diverse datasets such as Rico Mobile App Wireframes, COCO bounding boxes, PubLayNet documents, and 3D shapes. There a few limitations of our approach. First, our model requires a layout or scene to be decomposed into compositional primitives. In many cases, such primitives might not be most data-driven approaches, generated layouts are dominated by high frequency objects or shapes in the dataset. We can control the diversity some extent using improved sampling techniques, however, generating diverse layouts that not only learn from data, but also from human priors or pre-defined rules is important direction of research which we will continue explore.
defined. Second, like

---

Under review as a conference paper at ICLR 2021
REFERENCES
Oron Ashual and Lior Wolf. Specifying object attributes and relations in interactive scene genera- tion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4561-4569, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1XXvv:607.004500 2016.
Irving Biederman. On the semantics of glance at scene. In Perceptual organization, pp. 213-253. Routledge, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1rXiv: 10000..11990 1106, 2018.
Samuele Capobianco and Simone Marinai. Docemul: a toolkit to generate structured historical 03474 74
documents. CoRR, abs//710,,3474, 2017. URL htt.hhtpp//// www.wawwwww....oo.oo.t.oooo.aaaa
Angel X. Chang, Will Monroe, Manolis Savva, Christopher Potts, and Christopher D. Manning. Text to 3d scene generation with rich lexical grounding. CoRR, abs/1505.06289, 2015. URL
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5939-5948, 2019.
Biplab Deka, Zifeng Huang, Chad Franzen. Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrry Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applica- tions. In Proceedings of the 30th Annual Symposium on User Interface Software and Technology, UIST 17, 2017.
Hao Dong, Simiao Yu, Chao Wu, and Yike Guo. Semantic image synthesis via adversarial learning. In Proceedings of the IEFE International Conference on Computer Vision, pp. 5706-5714, 2017.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. arXiv preprint arXiv:1502.04623 2015.
Kamal Gupta, Susmija Jabbireddy, Ketul Shah, Abhinav Shrivastava, and Matthias Zwicker. Im- proved modeling of 3d shapes with multi-view depth maps. arXiv preprint urXiv:2009.03299 2020a.
Kamal Gupta, Saurabh Singh, and Abhinav Shrivastava. PatchVAE: Learning Local Latent Codes for Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4746-4755, 2020b. 2020b.
Tanmay Gupta, Alexander Schwing, and Derek Hoiem. Vico: Word embeddings from visual co- occurrences. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7425- 7434, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Generating multiple objects at spatially distinct locations. CoRR, 153/1901000866. 2019.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degener- ation. arXiv preprint 1rXivv/904.0999 2019.
10

---

Under review as a conference paper at ICLR 2021
Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7986-7994, 2018.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. arxiv, 2016.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1219-1228 1228, 2018.
Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochas- tic scene layout generation from a label set. arXiv preprint arXiv:1907. 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv:1710.101900 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4416 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412,6990, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013.
Donghoon Lee, Sifei Liu, Jinwei Gu, Ming-Yu Liu, Ming-Hsuan Yang, and Jan Kautz. Context- aware synthesis and placement of object instances. CoRR, abs/1812.02350, 2018. URL BRLhttp:
02350
Boren Li, Boyu Zhuang, Mingyang Li, and Jian Gu. Seq-sg2sl: Inferring semantic layout from scene graph through sequence sequence learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7435-7443, 2019a.
Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. Layoutgan: Generating graphic layouts with wireframe discriminators. arXiv preprint
2019b.
Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, and Hao Zhang. Grains: Generative recursive autoencoders for indoor scenes. ACM Transactions on Graphics (TOG), 38(2):1-16, 2019c.
Wenbo Li, Pengchuan Zhang. Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao. Object-driven text-to-image synthesis via adversarial training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12174-12182, 2019d.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. Learn- ing design semantics for mobile apps. In The 31 Annual ACM Symposium on User Interface Software and Technology, UIST 18, pp. 569-579, New York. NY, USA, 2018. ACM. ISBN 978- 1-4503-5948-1 doi: 10. 00.1145/2242587.22266663 URL http: acm. wttp::/..........0..0.......
3242587 3242650
Dipu Manandhar, Dan Ruta, and John Collomosse. Learning structural similarity of user interface layouts using graph networks. In ECCV, 2020.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv: 2013.
Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas J Guibas. Struc-
turenet: Hierarchical graph networks for 3d shape generation. arXiv preprint #XXv:1908.00055 2019.
11

---

Under review as a conference paper at ICLR 2021
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? CoRR, abs/1906.02629, 2019. URL http://tpp//t wwwpwwwwwwgg//////....oo. 02629
Charlie Nash, Yaroslav Ganin, SM Eslami, and Peter W Battaglia. Polygen: An autoregressive generative model of 3d meshes. arXiv preprint 108800,002.10880, 2020.
Aaron den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arxiv: 1601.06777 2016.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2019a.
Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu, Gaugan: semantic image synthesis with spatially adaptive normalization. In ACM SIGGRAPH 2019 Real-Time Live!, pp. 2. ACM, 2019b.
Akshay Gadi Patil, Omri Ben-Eliczer, Or Perel, Hadar Averbuch-Elor and Cornell Tech. Read: Recursive autoencoders for document layout generation. arXiv preprint arXiv: wet.:/999000000 2019.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lec. Generative adversarial text to image synthesis. arXiv preprint
2016.
Daniel Ritchie, Kai Wang, and Yu-an Lin. Fast and flexible indoor scene synthesis via deep con- volutional generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1182-6190, 2019.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications, 2017. arXiv preprint arXiv:1701.05517. 2017.
Abhinav Shrivastava and Abhinav Gupta. Contextual priming and feedback for faster r-cnn. In European Conference on Computer Vision, pp. 330-348. Springer, 2016.
Volker Steinbiss, Bach-Hiep Tran, and Hermann Ney. Improvements in beam search. In Third International Conference on Spoken Language Processing, 1994.
Minhyuk Sung, Hao Su, Vladimir Kim, Siddhartha Chaudhuri, and Leonidas Guibas. Com- plementme: weakly supervidd component suggestions for 3d modeling. ACM Transactions Graphics (TOG), 36(6):1-2,, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818-2826, 2016.
Antonio Torralba and Pawan Sinha. Statistical context priming for object detection. In Proceedings Eighth IEEE International Conference Computer Vision. ICCV 2001, volume pp. 763-770. IEEE, 2001.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. arXiv preprini arXiv: /0/i::5///.0339 2015.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances in neural information processing systems, pp. 613-621, 2016.
Kai Wang, Manolis Savva, Angel X Chang, and Daniel Ritchie. Deep convolutional priors for indoor scene synthesis. ACM Transactions on Graphics (TOG), 37(4):70, 2018a.
Kai Wang. Yu-An Lin, Ben Weissmann, Manolis Savva, Angel X Chang, and Daniel Ritchie. Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Transactions Graphics (TOG), 38(4): 2019.
12

---

Under review as a conference paper at ICLR 2021
Ting- Mogg-Chn Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference Computer Vision and Pattemn Recognition, 2018b.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning prob- abilistic latent space of object shapes via 3d generative-adversaria modeling. In Advances in neural information processing systems, pp. 82-90, 2016.
Jiajun Wu, Erika Lu, Pushmeet Kohli, William Freeman, and Joshua Tenenbaum. Learning to see physics via visual de-animation. In Advances in Neural Information Processing Systems, 2017a.
Jiajun Wu, Joshua Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In IEEE Confer- on Computer Vision and Pattern Recognition (CVPR), 2017b.
Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen. Pq-net: generative part seq2seq network for 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 829-838, 2020.
Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. PointFlow: 3D point cloud generation with continuous normalizing flows. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4541-4550, 2019.
Xiao Yang, Mehmet Ersin Yümer, Paul Asente, Mike Kraley, Daniel Kifer, and C. Lee Giles. Learn- ing to extract semantic structure from documents using multimodal fully convolutional neural network. CoRR, abs/1706.02337, 2017. URLhttp: www.aaiivvoo.... abs abs/ibsse/e 1766. 02337
Fenggen Yu, Kun Liu, Yan Zhang, Chenyang Zhu, and Kai Xu. Partnet: A recursive part decompo- sition network for fine-grained and hierarchical shape segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9491-9500, 2019.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris Metaxas. Stackgan: Text to phato-realistic image synthesis with stacked generative adver- sarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp- 5907-5915, 2017.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv:1805.083/1 2018.
Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. Image generation from layout. In CVPR, 2019.
Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH Lau. Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphies (TOG), 38, 2019.
Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for docu- ment layout analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1022- IEEE, 2019.
Chuhang Zou, Ersin Yumer, Jimei Yang, Duygu Ceylan, and Derek Hoiem. 3d-prnn: Generat- ing shape primitives with recurrent neural networks. Proceedings of the IEEE International Conference on Computer Vision, pp. 900-909, 2017.
13

---

Under review as a conference paper at ICLR 2021
Appendix
A ARCHITECTURE AND TRAINING DETAILS
In all our R experiments, our base model consists of dmmodel = 512, L 6, Rhead 8, precision 8 and du = 2048. We also use a dropout of 0.1 at the end of each feedforward layer for regularization. We fix the the maximum number of elements in each of the datasets 128 which covers 99.9% of the layouts each of the COCO, Rico and Publ.ayNer datasets. We also used Adam optimizer Kingma & Ba (2014) with initial learning rate of 10-4 We train our model for 300 epochs for each dataset with early stopping based on maximum log likelihood on validation layouts. Our COCO Bounding Boxes model takes about day train on a single NVIDIA GTX 1080 GPU, Batching matters a lot to improve the training speed. We want to have evenly divided batches, with absolutely minimal padding. We sort the layouts by the number of elements and search over this sorted list to use find tight batches for training.
In all our R experiments, we change dmnddll 128, and learning rate to 10
B ABLATION STUDIES
We evaluate the importance of different model components with negative log-likelihood on COCO layouts. The ablation studies show the following:
Small, medium and large elements: NLL of our model for COCO large, medium, and small boxes is 2.4, and 1.8 respectively. We observe that even though discretizing box coordinates introduces approximation errors, it later allows our model be agnostic to large vs small objects.
Varying precision: Increasing allows us to generate finer layouts but at the expense of a model with more parameters. Also, increase the precision, NLL increases, suggesting that we might need to train the model with more data to get similar performance (Table
Size of embedding: Increasing the size of the embedding dmodel improves the NLL, but at the cost of increased number of parameters (Table
Model depth: Increasing the depth of the model I,, does not significantly improve the results (Ta- 5p. We fix the L = 6 in all our experiments.
Ordering of the elements: Adding position encoding, makes the self-attention layer dependent to the ordering of elements. In order to make it depend less the ordering of input elements, take randomly permute the sequence. This also enables our model to be able to complete any partial layout. Since output is predicted sequentially, our model is not invariant sequence also. In our experiments, we observed that predicting the elements in simple raster sean order of their position improves the model performance both visually and in terms of negative log-likelihood. This is intuitive as filling the elements in a pre-defined order is an easier problem. We leave the task of optimal ordering of layout elements generate layouts for future research. (Table
the order of output
Diseretization strategy: Instead of the factorizing location in x-coordinates and y-coordinates, tried predicting them at once (refer to the Split-xy column of Table This increases the vocabulary size of the model (since we use H H possible locations instead of H alone) and in turn the number of hyper- parameters with decline in model performance. An upside of this approach is that generating new layouts takes less time as we have make half many predictions for each element of the layout (Table
2019) and NLL. Although opti- Loss: We tried two different losses, label smoothing Müller et al.
mizing using NLL gives better validation performance in terms of NLL (as is expected), we do not find much difference in the qualitative performance when using either loss function. (Table 6
14

---

Under review as a conference paper at ICLR 2021
Table 3: Effect of Maschr on NLLL
Table 4: Effect of d on NLL
Rinchor
params
COCO
Rico
PubLayNet
d
params
COCO
Rico
Publ Net
32 x 32
19.2
1.07
512
19.2
2.28
1.07
1.10
x 8
19.1
1.69
0.98
0.88
32
0.8
2.51
1.56
1.26
16 x 16
19.2
1.97
1.03
0.95
2.43
1.40
64 x 64
19.3
2.67
1.26
1.28
128
3.6
2.37
1.29
1.57
128 128
19.6
3.12
1.44
256
8.1
2.32
1.20
1,56
Table 5: Effect of L on NLL
Table 6: Effect of other hyperparameters on NLL
params
COCO
Rico
PabLayNet
Order
Split-XY
Loss
params
COCO
Rico
PubLayNet
6
19.2
2.28
1.07
1.10
raster
Yes
NLL
19.2
2.28
1.07
1.10
2.31
1.18
1.13
random
19.2
2.68
1.76
1.46
No
21.2
3.74
2.12
1.87
4
12.9
2.30
1.12
1.07
LS
19.2
1.96
0.88
0.88
8
25.5
2.28
1.11
1.07
wide edrener
Figure 11: Visualizing attention. (a) Image source for layout (b) each row, the model is predicting one element at a time (shown in a green bounding box). While predicting that element, the model pays the most attention to previously predicted bounding boxes (in red). For example, in the first row, "snow" gets the highest attention score while predicting "skis". Similarly in the last column, "skis" get the highest attention while predicting "person".
C VISUALIZING ATTENTION
The self-attention based approach proposed enables us to visualize which existing elements being attending to while the model is generating a new element. This is demonstrated Figure
Brgw
546-serson
porsor
Original Image
Original Layout INLL 2.873)
Left right (NIL 3.203|
(NLL 4.536)
Figure 12: We observe the impact of operations such left right flip, and up down flip log likelihood of the layout. We observe that unlikely layouts (such fog the bottom of image have higher NLL than the layouts from data.
15

---

Under review as a conference paper at ICLR 2021
Table 7: Bigrams and trigrams. We consider the most frequent pairs and triplets of (distinct) categories in real vs. vs.generated layouts.
Table 8: Analogies. We demonstrate lin- guistic nuances being captured by our cat- egory embeddings by attempting to solve word2vee (Mikolov ct al. 2013) style analo- gies.
other person
gerson persen clothes
person other person
person clothles person clothes hie tree other person
person other
clothes
clothes tie
person hundlag
other penon clathes
Analogy
Nearest neighbors
cloties person
saw.burdssww::sttboort cartond:erain:?
waterdrops, sea, sand muiload, platfom, gravel net, cage, wall wall-panel panell
chsir penson
dining
persan chair persen
wall-cunencie other person
person chair
uae grass
cluis per was chair
grass other cow tree other person
sky-athercloodss/ playsiiiiiinggiieeed mouse boardd
sky-other ure
wall-cnnerete other
person other clothes
knife, York, potted plant, mirve-steee staff
brckpncck persan person
person handhag
sky-nther
dising
frui:table:foowrr?
handbag person clothes person
person skis person
person other person
D LAYOUT VERIFICATION
Since in our method is straightforward to compute likelihood of a layout, we can use our method to test if given layout is likely not. Figure 12 shows the NLL given by our model by doing left- right and top-down inversion layouts in COCO (following Li et al. (2019b)). In case of COCO, flip layout left-right, decreases the likelihood (or increases the NLL of the layout). This is intuitive since it unlikely to fog in the bottom of image, while skis top of person.
observe that layout remains likely, however flipping the layout upside
E MORE SEMANTICS IN LEARNED CATEGORY EMBEDDINGS
Table 7 captures the most frequent bigrams and trigrams (categories that co-occur) real and syn- thesized layouts. Table S shows word2vec (Mikolov et al. 2013) style analogies being captured by embeddings learned by our model. Note that the model was trained to generate layouts and we did not specify any additional objective function for analogical reasoning task.
real
generated
sky-other
1.00
Bbox center (y)
airplane
0.75
clouds
0.50
person
0.25
road
0.00
car
sky-other airplane clouds
person sea
road
car
0.0
0.5
1.0
Bbox center (x)
Figure 13: Distribution of xy-coordssates of bounding boxes centers. Distributions for generated layouts and real layouts is similar. The y-coordinate tends to be more informative (e.g., sky on the top, road and sea at the bottom)
F DATASET STATISTICS
In this section, we share statistics different elements and their categories our dataset. In partic- ular, we share the total number of occurrences of element in the trai ning dataset (in descending
16

---

Under review as a conference paper at ICLR 2021
Table 9: Category statistics for Rico
Category
occurrences
# layouts
Category
# occurrences
# layouts
Text
387457
50322
Modal
3248
3248
Image
179956
38958
Pager Indicator
2041
1528
Icon
160817
43380
Slider
1619
954
Text Button
118480
33908
On/Off Switch
1260
683
List Item
72255
9620
Button Bar
577
Input
18514
8532
Toolbar
444
395
Card
12873
3775
Number Stepper
369
147
Web View
10782
5808
Multi-Tab
284
275
Radio Button
4890
Date Picker
217
Drawer
4138
4136
Map View
186
94
Checkhox
3734
1126
Video
Advertisement
3695
3365
Bottom Navigation
75
27
order) and the total number of distinct layouts element present in throughout the training data. show the statistics for Rico wireframes, and table I0 show the statistics for PubLayNet
Tables 9 9 documents.
Table 10: Category statistics for PubLayNet
Category
# occurrences
# layouts
text
2343356
334548
title
627125
255731
figure 109292
91968
table
102514
86460
list 80759
53049
G COORDINATE EMBEDDING
Just like in Fig. project the embedding learned by model on COCO in 2-d space using TSNE. the absence of explicit constraints on the learned embedding, the model learns to cluster together all the coordinate embedding in distinet space, in ring-like like manner.
Figure 14: TSNE plot for dimension embedding (256 of them) and category embedding for COCO.
17

---

Under review as a conference paper at ICLR 2021
H COMPARISON WITH POLYGEN
We would like to highlight some similarities of our framework with the recently proposed autore- gressive generative model for meshes, PolyGen Nash al. (2020). While both works adopt Transformer Decoder for autoregressive modeling, the usage differs in the following aspects:
PolyGen models mesh vertices as nodes. Advantage this method is that allows mod- elling high resolution 3D objects. However challenge is that sequence lengths for high reso- lution meshes can be very high and it can be very difficult to model them using self-attention (whose memory requirements grow proportionally to the square of sequence length).
We on the other hand separate out attributes (not just coordinates but also height, width, category and (or) SDF encoding) of parts of 3D objects which are typically fewer num- ber. Deep Networks based SDF encoding are an active area of research and the current SOTA methods don't provide high resolution results as mesh based methods.
Our model predicts future elements in order, but we randomize the order of the input ele- ments. This allows to do partial layout completion.
NEAREST NEIGHBORS
To see our model memorizing the training dataset, we compute nearest neighbors of generated layouts using chamfer distance on top-left and bottom-right bounding box coordinates of layout elements. Figure [5 [5] shows the nearest neighbors of some of the generated layouts from the training dataset. We note that nearest neighbor search for layouts is active area of research.
Generated
NN1
NN2
NN3
Funi
Sioee
Eivil
Figure 15: Nearest neighbors from training data. Column shows samples generated by model. Column 3, 4 show the closest neighbors from training dataset. We use chamfer distance on bounding box coordinates to obtain the nearest neighbors from the dataset.
18

---

Under review as a conference paper at ICLR 2021
J MORE EXAMPLES FOR LAYOUT TO IMAGE
Layouts for natural scenes are cluttered and hard to qualitatively evaluate even for a trained user. Here share some more sample layouts generated from methods used in the paper. Figure [6 shows some extra sample layouts and corresponding images generated using Layout2lm tool. Ex- isting layout to image methods don't it work as well free-form image generation methods but are arguably more beneficial in downstream applications. We hope that improving layout generation will aid the research community develop better scene generation tools both terms of diversity and quality.
(a) LayoutVAE layouts (top) and images generated with Layout2lm (bottom)
DORig SENT
(b) Our layouts (top) and images generated with Layoul2lm (bottom)
Figure Some sample layouts corresponding images
19


Reviews:
[{"review_id": "n-NeB8W_Mv2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an auto-regressive method for generating layouts by sequentially synthesizing new elements. The architecture is not dramatically new, but it is well-justified and analyzed, and there are some interesting tweaks. The results are strongest in that they show good performance of essentially the same architecture and hyperpameters across quite different domains: to my knowledge such variety has not really been demonstrated for any of the assembly-based generative models I'm familiar with.\n\nThe discretization aspect is not completely clear. As I understand it, each scalar geometric attribute of a node (but not the feature vector s_i) in the layout (x_i, y_i, w_i, h_i) is quantized to 8 bits, so that it can be thought of as selecting from one of 256 discrete options. How is the projection to d_model dimensions then performed, if d_model is 512 in one domain and 128 in another? What is the \"categorical distribution\" referred to here? Could you please explain this section more clearly?\n\nAlso, would it be possible to do an ablation study where no discretization is done at all?\n\nThe following paper is relevant as a baseline which also does sequential part assembly for shape generation. It would be nice if the authors could compare to this work.\n\nSung et al., \"ComplementMe: Weakly-Supervised Component Suggestion for 3D Modeling\", SIGGRAPH Asia 2017.\n\nThis is another paper that presents a fairly general framework for an autoregressive layout generator. The current paper should be compared with this as well, via experiments if possible:\n\nRitchie et al., \"Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models\", CVPR 2019\n\nAnd this paper is more recent than the ones compared to in the paper for document layout generation:\n\nGadi Patil et al., \"READ: Recursive Autoencoders for Document Layout Generation\", CVPR 2020 Workshop on Text and Documents in Deep Learning Era.\n\n(Various flavours of layout generation have seen a huge amount of research in the last few years, so I am not totally confident of my assessment especially vis-a-vis prior work.)\n\nThere are minor typos and grammatical errors -- a thorough proofread would help.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eQThgfu9B05", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis work proposes a model to generate scene layouts by treating the scene as a composition of primitives, such as instance class, coordinates or scales. The model is a Transformer architecture, that attends on all previously predicted or given instance primitives. The probability of a scene layout is defined with a joint distribution, modeled as the product of conditional distributions using the chain rule. The model predicts an end of sequence token, that allows the generated layouts to have variable size. Moreover, the model allows to either complete an existing incomplete layout or to generate one from scratch. The paper presents experiments in four datasets, spanning different data domains, including 2D and 3D data.\n\n\n### Strengths\n+ The motivation and objective of this work are clear.\n+ The general framework is simple and gives enough flexibility to apply it to different types of inputs and datasets.\n+ Providing results on different data domain datasets helps supporting the work. \n\n### Major concerns\nSome important points of the method, experimental setup and the results are not clear (and need to be clarified), as they bring some important concerns:\n- Using test set as validation set: It seems that the same split was used for both finding hyper-parameters (considering number of trained epochs due to early stopping also a hyper-parameter) and for reporting final results, at least for COCO-Stuff. It is not clear from this paper this is also the case for all the other datasets. In general, this is a bad practice and gives unfair advantage over baselines such as LayoutVAE, that uses a separate validation set to find hyper-parameters. \n- Unclear comparison with baselines: Given that some metrics/datasets were not used in the original papers, I have some concerns regarding fairness of comparisons; (1) were the methods re-trained for this paper or taken as is from the code of original authors? (2) If the answer is the former, were hyper-parameters tuned independently for all datasets, as to give the chance for all baselines to do better on each given task? If the answer is the latter, it is unfair to use the same setup and hyper-parameters designed for one task and apply it for all.  (3) how were the proposed method's parameters tuned? are these parameters tuned for each dataset? Related to these questions, Figure 5 shows really bad results for LayoutVAE. It is unclear to me if this is what one would expect from this method and it makes me suspect the baseline is not properly used, as it seems as if this baseline is generating random layouts. It could help to see the generated layout for this baseline, and not only the final image. \n\n### Other concerns\n- In page 4, it is said that at validation time, teacher forcing is used. Does this mean that for the final reported results teacher forcing is used as well?  This is important to clarify, as always having access to the ground-truth layout is an unrealistic assumption at test time.\n- The paper states in several parts that the framework allows to \"generate a new layout either from an empty set or from an initial seed set of primitives\". Which experiments show the method generating a layout from an empty set? It seems that the COCO-Stuff and 3D objects start from an initial set.  (1) Are the experiments on documents and applications from an empty set? (2) why not test the empty set start for COCO-Stuff and 3D objects? (3) when starting from an initial set, how many objects are used to start with? is it always the same number for all methods and setups? \n- I would like to see standard deviations for all reported metrics, as it is unclear for some of them whether the position with respect to other baselines is meaningful or not.\n- In Section 3 \"other details\", it is stated that nucleus sampling is used. However, throughout the paper, beam-search (Figure 2) or greedy-search (Figure 3) are mentioned. Which sampling method is actually used?\n\n\n### Additional questions/comments\n- As I understand, an EOS token is generated at some point and the layout generation stops. How is it prevented to generate a EOS token in the middle of an instance bounding box, let's say, after \"h\" and before outputting a \"w\" value. Is it only possible to generate an EOS token only when the next step generation is a class ID?\n\n- At the end of Section 2, one of the listed advantages of the proposed method is \"The autoregressive nature of the model allows us to generate layouts of arbitrary lengths as well as ...\", while in fact other existing approaches (for instance, LayoutVAE) can also generate arbitrary length layouts, and it is not something introduced in this paper.\n\n- In terms of general format:  Starting from the paragraph \"3D Primitive Auto-encoding\" in Page 5, this should go in another Section about experimental setup, not in the main method. Similarly, the conclusion should also be formatted as a separate section. \n\n- Regarding the sentence: \" The function that projects node i to latent space s_i can be learned independently or jointly with our layout generation framework.\" Where is this discussed in the paper or supported in experimental results? I could not find it and I am unsure of why is it mentioned here.\n- Given that geometry coordinates (x,y, w,h) are discretized, which primitives remain continuous? Equation 4 contemplates the case where primitives are continuous, but it is not clear which primitives remain continuous after the discretization step. \n\n- Why can't LayoutVAE and ObjGAN be applied to PoseNet? I would like to understand what is the limitations of these methods with respect to the proposed method.\n\n### Reason for score\nIt is a simple approach based on existing work, just slightly adapted for this layout generation task. Moreover, I had to infer most of the experimental and comparison details, as it is not clear in the paper. Additionally, given the important concerns already mentioned above regarding the provided results, I cannot discern whether this simple approach brings any actual improvement over existing methods. \n\n### What can be improved\n- A clearer discussion of what are the advantages of the proposed method. Although other methods did not provide results on such diverse data domains, it seems that LayoutVAE and ObjGAN were adapted as baselines for all except the 3D dataset. Therefore, the flexibility to different data domains of the proposed method, a major selling point, could in fact also be present in other methods. If baselines are allowed to tune parameters for each specific dataset, do they provide consistently worse results than the proposed method?\n- I would like the authors to extensively discuss the \"Major concerns\" and paint a better picture of the experimental setup. In this line, these details should also be included in the paper, whether it is in the main body or in the appendix. \n- If all other concerns raised are solved, this could also be clarified in the text. \n- Include results as mean and stdev over different random seeds for all experiments, providing more robust comparisons.\n\n\n### After authors response\nAlthough authors have addressed one of my main concerns and some minor ones, I still have doubts regarding the fairness of comparison with the baselines (lack of proper hyper-parameter optimization) and therefore cannot trust the results. All in all, I keep my rating. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "agKEAN2kNtN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=====Post-Rebuttal Comment=====\n\nI thank the authors for their detailed response to my concerns.\n\nWhile my opinion of this work remains largely similar, I raised by score from 5 to 6 for the following reasons:\n\n- I do buy the argument that the proposed method \"allow both developers and researchers to start from a strong method with a low barrier to entry in diverse domains\".\n- I do not share the concerns of R2 & R4 regarding the quality of results and fairness of comparison. As my primary concern (amount of technical innovation) is not shared by other reviewers, I am swayed to change my initial \"on the borderline\" rating to the positive side.\n\nI still would recommend adding the following evaluations:\n\n- more diverse initial state for Figure 10.\n- a more interpretable dataset for Figure 15: I think ShapeNet would demonstrate this point better.\n\n\n=====Summary=====\n\nThis work introduces a general framework for generating scene layouts in 2D or 3D. The key idea of this work is that one can represent an arbitrary layout as a sequence of objects, where each object contains a set of attributes (location, sizes, category, shape, etc.). By projecting these features into the same (high dimensional) embedding space, one can thus represent the entire layout as a sequence of embeddings. Consequently, it is possible to borrow architectures commonly used in natural language processing and design a generative model for such sequences. Here, the authors use a simple model with masked self attention and trained with teacher forcing. The proposed model is shown to have performance comparable or slightly better than SoTA methods in four different tasks under multiple metrics. It is also demonstrated that the learned embedded space displays some structure that respects the semantics.\n\n=====Strengths=====\n- The idea behind this paper seems to be general enough and can be applied to any scenarios where one can represent each entity in the layout share the same set of attributes.\n- I like the idea of separating different attributes a lot --- probably my favorite part of the paper. It makes sense intuitively that doing so will allow the attention module to more easily focus on the attributes that matter.\n- Strong empirical performance: comparable results for 3D shape synthesis and seems to achieve slightly better performances for the other three tasks evaluated.\n- Clear exposition and comprehensive discussion of related works in multiple areas\n\n=====Weaknesses=====\n- Would like to mention that one of the main novelty (to my understanding) in this work has already been attempted in an earlier work “PolyGen: An Autoregressive Generative Model of 3D Meshes”. The vertex model of PolyGen, in particular, uses a similar transformer decoder to generate the positions of points, a task that is very similar to predicting the position of objects in the layout. (the coordinates are also discretized in a similar way to this work, and the ordering strategy is similar, but these are much more minor points).\n- Following previous point: I think the novelty in this work is quite limited in general, using existing architectures and relying on ideas that have been explored (albeit in slightly different settings) before.\n- The empirical performance of the method is not strong enough to convince me that it can replace the more domain specific methods compared here. The reason is two-fold. First, it is unclear to me whether the proposed method is flexible enough to handle all the other tasks that those domain-specific method could, as only generation results are shown here. Second, I am not sure if the proposed method is flexible enough / has the right inductive bias for the layout tasks. For example, not taking hierarchy into account seems to be a minus for me when one want to interpret / manipulate the outputs. The sequential nature of the model also put constraints on the type of partial input acceptable e.g. it seems to be hard to handle input objects with only partial information available)\n\n=====Reasons for Score=====\n\nI’m right on the borderline for this paper and think this can go either way. \n\nOn the positive side, the paper shows a very clear message (again) that models borrowed from NLP can be surprisingly effective if we find a way to convert other structures (layout here) to a sequence. The empirical performance is also quite decent. On the negative side, the amount of novelty behind the main message of this paper is questionable: neither the architecture nor the idea of converting stuff to sequences are completely new, and the empirical performance is not strong enough for one to favor this approach over others (in their respective tasks). \n\nTo me, the question boils down to whether the main message of this paper is a very important one that deserves to be heard more by the broader community. I am not particularly convinced here, and lean slightly towards rejecting this paper. \n\n=====Additional Comments & Questions=====\n\nOther minor questions in addition to what I listed above:\n\n- I am not sure if it makes sense to map different types of features onto the same embedding space e.g. I don’t see how spatial coordinates and color can share any feature, and they would ideally just occupy distinct regions of the embedding space. Could the authors explain the design choice here? Why can’t one use different embedding spaces (and modify the attention module to handle that)?\n- Would like to see examples demonstrating that the model can generate diverse outputs, as opposed to always do the same thing for the same (partial) input. The shape completion tasks in figure 3 seems to be a good task to do this on. (The authors mentioned in conclusion that diversity is a problem, but I would still want to see how much of a problem it is, as I think some of the other works compared here don’t suffer too much from diversity problems)\n- Would also like to see a figure showing the nearest neighbor from the training set(s) to make sure that the model is not simply memorizing the input data. \n- It is mentioned in section 3.1 that discretization helps learning symmetries, are there concrete evidence towards this?\n- Figure 6: I get that there is structure here, but does the method learn better structure than other works?\n- Figure 5: I don’t get the message of this figure - all the images look equally bad to me.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Zoo7A8nzqwM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a transformer based architecture to model the probability distribution of scene layouts. The authors evaluate the model in four different application domains. The model performs competitively with state-of-the-art methods with respect to appropriate metrics.\n\n**Strengths**\n+ The paper is written clearly and the implementation details are appropriately described\n+ The idea of using transformers is interesting\n+ The evaluation of the method is thorough\n\n**Weaknesses**\nWhile the evaluation conducted by the authors is thorough, my main critique of this paper is that the results are not convincing enough to show the value of the proposed model.\n- The quantitative results do not outperform the state-of-the-art models consistently across all metric. For instance, the mdoel does not outperform PointFlow and PQ-Net across all metrics in Table 1. Same is the case for ObjGAN (IS=7.5) vs proposed (IS=7.1) in Figure 7. The authors fail to comment on why is that the case. \n- The qualitative samples are also not realistic in some cases. Some of the COCO results in Figure 4 and 5 do not look realistic -- layouts are too cluttered leading to incomprehensible scene. (row 2,col 3) in Figure 4 and (row 3,col 4) in Figure 5). From the layout samples presented in the paper, it seems the model produces cluttered layouts when the model is trying to generate higher number of objects.  The authors do not discuss this aspect in the paper. Is the quality of the layout dependent on the number of entities in the scenes? If yes, I think it is important for a scene layout generation model should be robust enough to handle flexible lengths.\n\nOverall, while the paper has interesting approach to model the relations between the elements of a scene, the results are not convincing enough to demonstrate the effectiveness of the proposed model. Therefore, my initial rating is 5.             \n                                                                                                                                                                                      \n=========================================**Post-rebuttal comments**=======================================\n\nI appreciate the revisions and additional visualization provided the authors. The revised version of the paper provides clarifications that make the paper easier to follow. While the paper presents an interesting idea, I am not convinced of the effectiveness of the proposed method based on the results provided by the authors. Therefore, my final rating as 5 (same as the initial rating).  ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper proposes an approach for scene autoregressive layout generation. Four expert reviewers evaluated the paper outlining the following pros/cons of the work. 

> Pros:
- Good performance across different domain [R1,R2,R3,R4]
- Formulation is general [R1,R2]
- Clever separation of different attributes [R1]
- The idea of using transformers is interesting [R4]

> Cons:
- Missing related works [R3]
- Unclear comparison with baselines that [R2]
- Lacks of  hyper-parameter tuning on the baselines [R2]
- The quantitative results do not outperform the state-of-the-art models consistently across all metric [R4]

Authors have addressed some of the concerns in the rebuttal and generally reviewers are more convinced after the rebuttal than before. The fairness of comparison to baselines remains an issue for two of the reviewers, and quality of results for one. AC acknowledges and agrees with these concerns. As such, given the large number of highly qualified submissions to ICLR and in comparison to those submissions, the paper fell slightly bellow the acceptance threshold. 

That said, AC believes the approach, overall, is interesting and warrants re-submission after the appropriate revisions are implemented. 
Decision: Reject