Compositional Transformers for Scene Generation
Authors: Drew Arad Hudson
C. Lawrence Zitnick
Venue: NeurIPS 2021 Poster (2021)
Keywords: GANs
transformers
compositionality
reasoning
image generation
structure
objects
iterative
layouts

Abstract: We introduce the GANformer2 model, an iterative object-oriented transformer, explored for the task of generative modeling. The network incorporates strong and explicit structural priors, to reflect the compositional nature of visual scenes, and synthesizes images through a sequential process. It operates in two stages: a fast and lightweight planning phase, where we draft a high-level scene layout, followed by an attention-based execution phase, where the layout is being refined, evolving into a rich and detailed picture. Our model moves away from conventional black-box GAN architectures that feature a flat and monolithic latent space towards a transparent design that encourages efficiency, controllability and interpretability. We demonstrate GANformer2's strengths and qualities through a careful evaluation over a range of datasets, from multi-object CLEVR scenes to the challenging COCO images, showing it successfully achieves state-of-the-art performance in terms of visual quality, diversity and consistency. Further experiments demonstrate the model's disentanglement and provide a deeper insight into its generative process, as it proceeds step-by-step from a rough initial sketch, to a detailed layout that accounts for objects' depths and dependencies, and up to the final high-resolution depiction of vibrant and intricate real-world scenes. See https://github.com/dorarad/gansformer for model implementation.

Compositional Transformers for Scene Generation
Drew A. Hudson
C. Lawrence Zitnick Facebook Al Research Facebook, Inc. zitnickefb. com
Department of Computer Science
Stanford University dorarad@cs swwnfstadddrrooo edu
Abstract
We introduce the GANformer2 model, an iterative object-oriented transformer, explored for the task of generative modeling. The network incorporates strong and explicit structural priors, to reflect the compositional nature of visual scenes, and synthesizes images through sequential process. It operates in two stages: a fast and lightweight planning phase, where we draft a high-level scene layout, followed by attention-based execution phase, where the layout is being refined, evolving into a rich and detailed picture. Our model moves away from conventional black-box GAN architectures that feature flat and monolithic latent space towards a transparent design that encourages efficiency, controllability and interpretability. We demonstrate GANformer2's s strengths and qualities through a careful evaluation over a range of datasets, from multi-object CLEVR scenes to the challenging COCO images, showing it successfully achieves state-of-the-art performance terms of visual quality, diversity and consistency. Further experiments demonstrate the model's disentanglement and provide deeper insight into its generative process, as it proceeds step-by hep-by-step from rough initial sketch, to detailed layout that accounts for objects' depths and dependencies, and up to the final high-resolution depiction of vibrant and intricate real-world scenes. See https: // thub. com/ dorarad/ aeeaarddganstiia ggggganffoooo former for model implementation.
Introduction
Drawing, the practice behind human visual and artistic expression, can essentially be defined as an iterative process. It starts from an initial outline, with just few strokes that specify the spatial arrangement and overall layout, and is then gradually refined and embellished with color, depth and richness, until a vivid picture eventually emerges. These initial schematic sketches can serve as an abstract scene representation that is very concise, yet highly expressive: several lines are enough to delineate three dimensional structures, account for perspective and proportion, convey shapes and geometry, and even imply semantic information [52, 63, 67]. A large body of research in psychology highlights the importance of sketching in stimulating creative discovery [40, 58], fostering cognitive development [27], and facilitating problem solving [26, 44). In fact, large variety of generative and RL tasks either in the visual [5], textual [22, 78] or symbolic modalities [2], can benefit from the same strategy prepare high-level plan first, and then carry out the details.
At the heart of this hierarchical strategy stands the principle of compositionality - where the meaning of the whole can be derived from those of its constituents. Indeed, the world around is highly structured. Our environment consists of varied collection of objects, tightly interconnected through dependencies of all sorts: from close-by long-range, and from physical and dynamic abstract and semantic [10, 66). As pointed out by prior literature [24, 51, 55, 57], compositional representations are pivotal to human intelligence, supporting our capabilities of reasoning [32], planning [53], learning [54] and imagination [6]. Realizing this principle, and explicitly capturing the objects and elements composing the scene, thereby desirable goal, that make the generative process more explainable, controllable, versatile and efficient.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Mierarchical
Senantis Canageries
Variables
Semantic Matching
Structure Latents
D2
Segment Fidelity
Planning Stage:
recurrently constructing compositional schemstic scerre layout
Exccution Stage: transforming the layout en celpulinagge as guided
The GANformer2 model a set of latent variables that represent objects entities within the generated scene. It proceeds through two stages, planning and execution: first itcratively drafting high-level layout of the scene's s spatial structure, and then refining and translating it into photo meato-tellistic image. Each latent corresponds to different layout's segment, controlling its structure during the planning, and its style during the execution. On the discriminator side, we introduce new structural losses for compositional scene generation.
Figure 1:
Yet, the vast majority of generative models do not directly reflect this compositionality so intrinsic to visual scenes. Rather, most networks, and GANs in particular, aim generate the entire scene all at once from a single monolithic latent space, through a largely opaque transformation. Consequently, while they excel in producing stunning, strikingly-realistic portrayals of faces, sole centralized objects, or natural scenery, they still struggle to faithfully mimic richer or densely-packe packedda scenes, as those common to everyday contexts, falling short of mastering their nuances and intricacies [5, 50, 76). Likewise, while there is some evidence for property disentanglement within their latent space to independent of variation at the global scale [28, 36, 64, 71J. most existing GANs fail provide controllability at the level of the individual object or local region. Understanding of the mechanisms by which these models give rise an output image, and the transparency of their synthesis process. thereby remain rather limited.
Motivated alleviate these shortcomings and make generative modeling more compositional, inter- pretable and controllable, we propose GANformer2, a structured object- transformer that decouples the visual synthesis task into two stages: planning and execution. We begin by sampling a set of latent variables, corresponding to the objects and entities that compose the scene. Then, at the planning stage, transform the latents into a schematic layout scene representation that depicts the object segments, reflecting their shapes, positions, semantic classes and relative ordering. Its construction occurs an iterative manner, to capture the dependencies and interactions between different objects. Next, at the execution stage, the layout is translated into the final image, with the latents cooperatively guiding the content and style of their respective segments through bipartite attention [33]. This approach marks shift towards a more natural process of image generation, in which objects and the relations among them are explicitly modeled, while the initial sketch is successively tweaked and refined to ultimately produce a rich photo- choto-realistic scene.
We study the model's behavior and performance through an extensive set of experiments, where attains state-of-the-ard results in both conditional and unconditional synthesis, as measured along metrics of fidelity, diversity and semantic consistency. GANformer2 reaches high versatility, demon- strated through multiple datasets of challenging simulated and real-world scenes. Further analysis illustrates its capacity to manipulate chosen objects and properties an independent manner, achiev- ing both spatial disentanglement and separation between structure and style. We notably observe amodal completion of occluded objects, likely driven by the sequential nature of the computation. Meanwhile, inspection of the produced layouts and their components shed more light upon the syn- thesis of each resulting scene, substantiating the model's transparency and explainability. Overall, by integrating strong compositional structure into the model, we can move in the right direction towards multiple favorable properties: increasing robustness for rich and diverse visual domains, enhancing controllability over individual objects, and improving the generative process's 's interpretability.
2 Related Work
Over the last years, the field of visual sythesis has witnessed astonishing progress, driven in particular by the emergence of Generative Adversarial Networks [23]. Tremendous strides have been made in
wide variety of tasks, from image [11] to video generation [7], to super-resolution [45], style transfer
[15] and translation [35]. Mcanwhile, growing body of literature explored object discovery and

---

Figure 2: Layout and image generation. The GANformer2 model generates cach image by first creating a layout that specifies its outline object composition, which is then translated into the final image. While doing so, the model produces depth maps of the object segments which are used to determine their relative ordering, and notably are implicitly derived by the model from the layouts in fully unsupervised manner.
sequential scene decomposition, albeit in the context of variational inference [1. 16, 19, 49]. DRAW [25] uses recurrent attention encode and decode MNIST digits, while AIR [20], MONet [12] and IODINE [24] reconstruct simple simulated scenes by iteratively factorizing them into semantically meaningful segments. We draw inspiration from this pioneering line of works and marry the merits of its strong structure with the scalability and fidelity obtained by GANs, yielding compositional model that successfully creates intricate real-world scenes, while offering a more transparent and controllable synthesis along the way, conceptually meant to better resemble the causal generative process and serve as a more natural emulation of how a human might approach the creative task.
To this end, our model leverages the GANformer architecture we introduced earlier this year [33] bipartite transformer that uses soft-attention to cooperetaviely translate set of latents into generated image [33, 70]. In GANformer2, we take this idea step further, and show how we can use the bipartite transformer as a backbone for a more structured process, that enables not only an implicit, but also explicit modeling of the objects and constituents composing the image's 's scene. Our iterative design endows the GANformer2 model with the flexibility to consider the correlations and interactions not only among latents but between synthesized scene elements directly, circumventing the need for partial independence assumptions made by carlier works [18, 56, 69], and thereby managing to scale to natural real-world scenes. Concurrently, the explicit object-oriented structure enhances the model's compositionallity so to outperform past coarse-to- cearse-to-fine cascade [80], layered [34], and hierarchical approaches [5], which, like employ two-stage layout-based procedure, but contrary to our work, do so in a non-compositional fashion, and indeed were mostly explored within limited domains, such as indoor NYUv2 [73], faces [41], pizzas [59], or bird photographs [77].
Our model links to research on conditional generative modeling [72, 85, 87], including semantic image synthesis and text-to-image generation. The seminal Pix2Pix [35] and SPADE [60] works respectively adopt either a U-Net [62] or spatially-adaptive normalization [61] to map a given input layout into output high-resolution image. However, beyond their notable weakness of being conditional and thus unable to create scenes from scratch, their reliance on both static class embeddings for feature modulation, along with perceptual or feature-matching reconstruction losses, render their sample variation rather low [68, 86). our work, we mitigate this issue by proposing new purely generative structural loss functions: Semantic Matching and Segment Fidelity, which maintain the one-to to--to-mm to-y relation that holds between layouts and images, and consequently, as demonstrated in section 3.3, significantly enhance the output diversity.
While GA GANformer2 is an unconditional model, shares some high-level ideas with conditional text-to-image [30, 31] and scene-to-image [4, 38] techniques, which use semantic layouts

---

Figure 3:
Recurrent scene generation. GANformer2 creates the layout sequentially, segment-by-segmen to capture the scene's is compositionallty. effectively allowing to add or remove objects from the resulting images.
bottleneck intermediate representation. But whereas these works focus on heavily-engineers pipelines consisting of multiple pre- and post-processing stages and different sources of supervision, including textual descriptions, scene graphs, bounding boxes, segmentation masks and images, we propose a simpler, more elegant design, with streamlined architecture and fewer supervision signals - relying on images and auto anto-predicted layouts only, to widen its applicability across domains. Finally, our approach relates to prior works on visual editing [8, 9, 46, 84] and image compositing [3, 13, 47, 79]. but while these methods modify existing source image, GANformer2 stands out being capable of creating structured manipulatable scenes from scratch.
3 The GANformer2 model
The GANformer2 model is a compositional object-oriented transformer for visual scene generation. It features set of latent variables corresponding to the objects and entities composing the scene, and proceeds through two synthesis stages: (1) sequential planning stage, where scene layout a collection of interacting object segments, is being created (section 3.2), followed by (2) a parallel execution stage, where the layout is being refined and translated into an output photo-realistic image (section 3.3). Both stages are implemented using customized versions of the GANformer model (section 3.1) a bipartite transformer that uses multiple latent variables to modulate the evolving visual features of a generated image, so to create
compositional fashion.
In this paper, we explore learning from two training sets: of images and layouts. Specifically, we panoptic segmentations [43], which specify for every segment s; its instance identity p; and semantic category m,, but other types of segmentations can likewise be used. The images and layouts either be paired or unpaired, and training scheme accommodates both options: either by pre-training each stage independently first and then fine-tune them jointly in the paired case, or training them together from scratch in the unpaired one (See section F for further details).
3.1 Generative Adversarial Transformers
The GANformer [33] is a tranformer-basee Generative Adversarial Network: it consists of generator G(Z)= = X that translates partitioned latent Z = [21, ..., #t| into image XHWe, and discriminator DXX that seeks to discern between real and fake images. The generator begins by mapping the bormally-distribung
latents into an intermediate space W [wr...., Wi] using feed-forward network. Then, it sets an initial 4x4 grid, which gradually evolves through convolution and upsampling layers to the final image. Contrary to traditional GANs, the GANformer also incorporates appartit-transforee
layers into the generator, which compute attention between the latents and the image features, support spatial and contentual region-wise modulation:
Attention X, W) softmax (( (W))
v(W)
Vd
x(X, W) 7a ( W) c Layer + (X,
(2)
Where the equations respectively express the attention and modulation between the latents and the image; q(-), k(-), v(-) are the query, key, and value mappings; and 7-),, (-)) compute multiplicative and additive styles (gain and bias) as a function of Attention(X. (X, W). The update rule lets the latents shape and impact the image regions as induced by the key-value attention image-to-latents assignment computed between them. This encourages visual compositionality, where different latents specialize to represent and control various semantic regions visual elements within the generated scene.

---

Figure 4: GANformer2 style variation. Images are produced by varying the style latents of the objects while keeping their structure latents fixed, achieving high visual diversity while maintaining structural consistency.
As will see below, for each of the two generation stages, couple a modified version of the GANformer with novel loss functions and training schemes. One key point in which we notably depart from the original GANformer regards the number of latent variables used by the model. While the GANformer employs a fixed number of k latent variables, GANformer2 supports a variable number of latents, thereby becoming more flexible in generating scenes of diverse complexity.
3.2 Planning Stage Layout Synthesis
In the first stage, we create from scratch schematic layout depicting the structure of the scene to be formed (see figure 2). Formally, we define layout S as set of l segments {s1,....s)) 8k). Each generated segment 8; is associated with the following':
Shape & position: specified by spatial distribution P. y) over the image grid H x W.
Semantic category: a soft distribution M.(c) over the semantic classes.
Unsupervised depth-ordering: per-pixel values (1, y) indicating the order relative to other segments, internally used by the model to compose the segments into a layout (details below).
Recurrent Layout Generation. To explicitly capture conditional dependencies among segments, we construct the layout in an iterative manner, recurrently applying parameter-lightweigee GANformer G1 over T steps (around 2-4). At each step t, we generate a variable number of segments, hi ~ ((,,o)) sampled from a trainable normal distribution' and gradually aggregate them into scene layout S. To generate the segments at step I, we use GANformer with ky sampled latents, each one zı is mapped through feed-forward network Fi (z:) u, into intermediate structure latent ui which then corresponds to segment 8;, guiding the synthesis of its shape, depth and position.
GU S,_,) S.
To make the GANformer recurrent, and condition the generation of new segments on ones produced in prior steps, we change the query function q(-) to consider not only the newly formed layout Se (as in the standard GANformer) but also the previously aggregated one S (concatenating them together and passing as an input), allowing the model to explicitly capture dependencies across steps.
Object Manipulation. Notably, this construction supports posteriori object- wise manipulation. Given generated layout S, we can recreate a segment 8; by modifying its corresponding latent from 2 to & while conditioning on the other segments S- mathematically by (2) .S.)). We thus reach a favorable balance between expressive and contextual modeling of relational interactions the one hand, and stronger object disentanglement and controllability on the other.
Layout Composition. To compound the segments together into layout S., we overlay them according to their predicted depths d,(x,, y) and shapes P. y) by computing
(oftmax(d,(x. + log P. v))
'To produce the segments (instead of RGB values), we modify the output layer to predict every segment its shape P,, category M, and depth d, Since the distributions are soft, model stays fully differentiable. Segments synthesized in groups for computational efficiency when generating crowded scenes.
5

---

Figure 6: Object controllability. GANformer2 supports entity-level manipulation, and separates style from structure. enabling control over the chosen objects and properties. At each row, we begin by generating scene from scratch (leftmost image) and then gradually interpolate the latent vector of single object: (1) modify its style latent w, lcading to a color change, (2) modify its structure latent u, resulting in position change, and (3) modify both the structure and style latents, yiclding respective changes in shape and material.
which yields for each image position (r,y) distribution over the segments. Intuitively, this allows resolving occlusions among overlapping segments. E.g. a cube segment should appear before a sphere segment j. then (x(x. y) > (a, y) within the overlap region {(x, y)}- Notably, we do not provide any supervision to the model about the true depths of the scene elements, and it rather creates its internal depth ordering in an unsupervised manner, only indirectly moderated by the need to produce realistic looking layouts when stitching the segments together.
Noise for Discrete Synthesis. The segmentations we train the planning stage over inherently discrete, while the generated layouts soft and continuous. To maintain stable GAN training under these conditions, and inspired by the instance noise technique [65], add simple hierarchical noise map the layouts fed into the discriminator, which helps ensuring its task remains challenging. We define the noise as Σ Upsample(n x2"), summing up
Figure 5: Hierarchi- cal Noise scheme.
noise grids n, ~ [0, o²) across multiple resolutions, making it partly correlated between adjacent positions and more structured overall. This empirically enhances the effectiveness and substantially improves the generated layouts's quality.
3.3 Execution Stage: Layout-to-Iman Translation
In the second stage, we proceed from planning to execution, and translate the generated layout into photo-realistic image X. For each segment 8,, first concatenate its mean depth d; and category distribution m, to its respective latent z;, and map them through feed-forward network F22 into an intermediate style latent w,, which will determine the content, texture and style of its segment. Then, we pass the resulting vectors into a second GANformer G2(W, = X to render the final image.
Layout Translation. As opposed to the original GANformer [33] that computes Attention (X, W) on-the-fly fhy to determine the assignment between latents and spatial regions, here we rely instead on the pre-planned layout S as a conditional guidance source to specify this association. That way, each latent w; modulates the features within the soft region its respective segment 8,, achieving a more explicit ceaaanicalll--mmnnngee since the whole image is still generated together, with all regions processed through the convolution, upsampling and modulation layers of G2, the model still ensure global visual coherence.
assignment between latents and scene elements. At the same time,
Layout Refinement. To increase the model's flexibility while rendering the output image, we optionally multiply the latent-to-segment distribution induced by the layout S with a trainable

---

Figure 7: Conditional generation. Sample images generated by GANformer2 conditioned on input layouts.
sigmoidal gate o (gg(S, X, W)), which considers the layout S, the evolving image features X. and the style latents W. This allows the model locally adjust the modulation strength sees fit, to refine the pre-computed layout during the final image synthesis.
Semantic-Matchinn Loss. To promote streutrual and semantic consistency between the layout S and image produced by each of the synthesis stages, we complement the standard fidelity loss L(D with two objectives: Semantic Matching and Segment Fidelity. We note that the relation between a given layout and the set of possible images that could correspond to it is one-to-many relation. incorporate into the discriminator U-Net [62] to predict the segmentation S' of each image, compare it using cross-entropy to the source layout, and add the resulting loss term to the discriminator (over real samples) and the generator (over fake ones).
To encourage
(SM(S,)) (S) where S' = See((X)
(5)
This stands in stark contrast to the reconstruction perceptual or feature-matching losses com- monly used in conditional layout-to-image approaches [14, 35, 60], which, unjustifiably, compare the appearance of a newly generated image X to that of natural source image X' (mathematically, even though there is no reason for them to be
through either Lx(X, X') or LPM similar in terms style and texture, beyond sharing the same layout. Consequently. those losses severely and fundamentally hinder the diversity of the generated scenes (as shown in section 4). We circumvent this deficiency by comparing instead the layout implied by the new image with the source layout it is meant to follow, thereby staying
the original objective of the translation task.
Gegment-Fidelity Loss. To further promote the semantic alignment between the layout and the image, and encourage fidelity not only of the whole scene, but also of cach segment it is composed of, introduce the Segment-Fiche Fidelity loss. Given the layout S and the generated image X, the discriminator first concatenates and processes them together through shared nonvolution-downsamge reduce their resolution. Then, it partitions the image according to the layout (like a jigsaw puzzle) and assesses the fidelity of each of the segments (s;, 22) independently, through:
stem to
1 Σ LSF (D(s,
(6)
7

---

Table 1: Unconditional approaches comparison. We compare the models on generating images from scratch, using the FID and Precision/Recal metries for evaluation, computed over 50k sample images. COCOp refers a mean score over a partitioned version of dataset, created improve samples' visual quality (section
CLEVR
Bedrooms
FFHQ
Model
FID
Precision
Recall
FID
Precision
Recall
FID
Precision
Recall
GAN
25.02
21.77
16.76
12.16
52.17
13.63
13.18
67.15
17.64
K-GAN
28.29
22.93
SAGAN
26.04
30.09
15.16
14.06
54.82
7.26
16.21
64.84
12:26
StyleGAN2
VQGAN
32,60
46.55
63.33
59.63
55.24
28,00
63.12
67.01
29.67
SBGAN
35.13
48.12
64.41
48.75
56.14
31,02
24.52
58.32
8.17
GANionmer
9.17
47.55
66.63
6.51
57.41
29.71
7.42
68.77
5.76
GANformer2
4.70
64.18
67.03
6.05
60.93
37.15
7.77
61.54
34.45
COCO
COCO,
Ciyssaaes 4
Precision
Precision
GAN
41.00
48.57
7.06
37.53
51,02
9.30
11.56
61.09
15.30
K-GAN
63.51
42.06
5.42
61.05
45.37
7.04
$1.08
18.80
1.73
SAGAN
46.09
50.00
7.47
42.05
51.71
7.84
12.81
43.48
7.97
StyleGAN2
26.79
50.92
23.30
25.24
52.93
24.48
8.35
59.35
27.82
SRGAN
108.1
42.53
17.53
104.32
44.43
19.12
54.92
52.12
25.08
GANformer
25.24
53.68
12.31
23.81
14.48
5.76
GANformer2
21.58
49.12
29.03
20.41
51,07
32,84
6.21
56.12
54.18
Using the stem both reduces the dimensiononality to improve computational efficiency, and further allows for contextual information of each segment's surroundings propagate into its representation. This improves over the stricter patchGAN [35] which blindly splits the image into a fixed tabular grid, while our new loss offers a more natural choice, dividing up into semantically meaninnll regions.
4 Experiments
We investigate GANformer2's properties through suite of quantitative and qualitative experiments. As shown in section 4.1, the model attains of- performance, successfully generating images of high quality, diversity and semantic consistency, over multiple challenging datasets of highly-structured simulated and real-world scenes. section 4.2, we compare the execution stage to leading payout-to-image approaches, demonstrating significantly larger output variation, possibly thanks to the new structural loss functions we incorporate into the network. Further analysis conducted in sections 4.3 and 4.4 reveals several favorable properties that GANformer2 possesses, specifically of enhanced interpretabilty and strong spatial disentanglement, which enables object-level manipulation. In the supplementary. we provide additional visualizations of style and structure disentanglement as well as ablation and variation studies, to shed more light upon the model behavior and operation. Taken altogether, the evaluation offers solid evidence for the effectiveness of our approach in modeling compositional scenes in a robust, controllable and interpretable manner.
4.1 State-of-the-Art Comparison
We compare our model with both baselines as well as leading approaches for visual synthesis. Unconditional models include baseline GAN [23], StyleGAN2 [42],
GAN [81], the autoregressive VQGAN [21], the layerwise k-GAN [69], the two-stage SBGAN [5] and the original GANformer. Conditional approaches include SBGAN, BicycleGAN [86], Pix2PixHD [72], and SPADE [60]. We implement the unconditional methods within our public GANformer codebase and use the authors' official implementations for the conditional ones. All models have been trained with resolution of 256 x 256 and for an equal number of training steps, roughly spanning 10 days single V000 GPU per model. See section for description of basclines and competing approaches, implementation details, hyperparameter settings, data preparations and training configuration.
As tables and show, the GANformer2 model outperforms prior work along most metrics and datasets in both the conditional and unconditional settings. It achieves substantial gains in terms of FID score, which correlates with visual quality [29], as likewise reflected through the good Precision scores. Notable also are the large improvements along Recall, in the unconditional and especially the conditional settings, which indicate wider coverage of the natural image distribution. We note that the performance gains are highest for the CLEVR [37] and the COCO [48] datasets, both focusing on varied arrangements of multiple objects and high structural variance. These results serve strong evidence for the particular aptitude of GANformer2 for compositional scenes.

---

Table 2: Disentanglement & controllability comparison over sample CLEVR images, using the DCI metrics for latent-space space disentanglement and our correlation-aase metrics for controllability (see section 4.4).
Model
Disentanglement
Modularity
Completeness
Informadiveness
Informaticenes'
Object-p
Propett-p2
GAN
0.126
0.631
0.U71
0,583
0.434
0.72
0.85
StyleGAN
0.208
0.703
0.124
0.685
0.332
0.64
0.49
GANformer
0.768
0.952
0.270
0.972
0.963
0.38
0.45
MONet
0.821
0.912
0.349
0.955
0.946
0.29
0.37
lodine
0.784
0.948
0,382
0.941
0.937
0.27
0.35
GANformer2
0.852
0.946
0.413
0.974
0.965
0.19
0.32
4.2 Scene Diversity & Semantic Consistency
Beyond visual fidelity, we study the models under the potentially compet- ing aims of content variability on the one hand and semantic consistency on the other. To assess consistency, we compare the input layouts S with those implied by the synthesized images, using standard metrics of mean loU, pixel accuracy and ARI. Following prior works [60, 86]. the inferred layouts S' are obtained by a pre-trained segmentation model that use for evaluation purposes. To measure diversity. we sample n 20 images X, for each input layout S, and compute the mean LPIPS pairwise distance [82] over image pairs X,, X,. Ideally, we expect a good model to achieve strong alignment between each input layout and the one derived from the output synthesized image, while still maintaining high variance between samples that are conditioned on the same source layout.
Consistency Losses (COCO)
Store
(Ming)
Planning Steps (CLEVR)
We see in table 3 that GANformer2 produces images of significantly better diversity, reflected both through the L.PPS and Recall scores. Other conditional approaches such as Pix2PixHD. BicycleGAN and SPADE that rely on pixel-wise perceptual or feature matching, reach high precision at the expense of considerably reduced variability, revealing the weakness of those losses. We further note that our model achieves better consistency than prior work, as is also illustrated by figures 22-24 which present qualitative comparison between synthesized samples. These results demonstrate the benefits of our new structural losses (section 3.3). as is also corroborated by the model's 's learning efficiency when evaluated with different objectives (figure 8). We compare the Semantic- Semantic-Matching and Segment-Fidee Fidelity losses with several alternatives (section D), and observe improved performance and accelerated learning when using the new objectives, possibly thanks to the local semantic alignment and finer-detail fidelity they respectively encourage.
Step (King)
Figure 8: Model's learn- ing curves.
4.3 Transparency & Interpretability
An advantage of our model compared to traditional GANs, is that we incorporate explicit structure into the visual generative process, which both makes more explainable, while providing means for controlling individual scene elements (section 4.4). In figures 3 and 11, we see how GANformer2 gradually constructs the image layout, recurrently adding new object segments into the scene. The gains of iterative over non-iterative generation are also demonstrated by figures 8 and 21. We note that different datasets benefit from a different number of recurrent synthesis steps, reflecting their respective degree of structural compositionality (section E). Figures 2 and 10 further show that the model develops an internal notion for segments' depth, which aids in determining the most effective order to compile them together. Remarkably, explicit information is given to the network about the depth of entities within the scene, nor about effective orders to create them, and rather learns these on its own in unsupervised manner, through the indirect need to identify relative depths and segment ordering that will enable creating feasible and compelling layouts.
4.4 Controllability & Disentanglement
The iterative nature of GANformer2 and its compositional latent space enhance its disentanglement along two key dimensions: spatially, enabling independent control over chosen objects, and seman- tically, separating between structure and style- the former is considered during planning while the latter at execution. Figures 6 and 13-16 feature examples where we modify an object of interest, changing its color, material, shape, position, all without negatively impacting its surroundings. In figures and 17-20, we produce images of diverse colors and textures while conforming to source layout, and conversely, vary the scene composition while maintaining consistent style. Finally,

---

Table 3: Conditional approaches comparison. We compare conditional generative models that map input layouts into output images. Multiple metrics are used for different purposes: FID, Inception (IS) and Precision scores for visual quality; class-weighted mean loU, Pixel Accuracy (pAcc) and Adjusted Rand Index (ART) for layout-image consistency; Recall for coverage of the natural image distrihution; and LPIPS for measuring the output visual diversity given an input layout. All metrics are computed over 50k samples per model.
CLEVR
Model
FID
IS
Precision
Recall
mloU
pAcc
ARI
LPIPS
PiX2PIxHD
4,88
2.19
72.49
60.40
98.18
99.06
97.90
6c-8
BicycleGAN
35.62
2.29
6.27
10.11
88.15
92.78
90.21
0.10
SRGAN
2.22
48.20
95.27
97.81
92.34
8c-7
GANorrme22
0.99
2.34
73.26
78.38
97.83
99.15
98.46
0.19
Bedrooms
Model
HD
15
Precision 1
Recall 1
mloU
pAcco
ARI 1
LPIPS
Pix2PixIID
2.27
SPADE
17.06
2.33
80.20
7.04
70.27
80.63
79.44
0.07
BicycleGAN
2.52
10.45
70.76
72.73
0.22
SRGAN
29.35
2.28
56.21
7.93
65.31
71.39
73.24
3e-8
GANorrer22
5.84
2.55
54.92
41.94
79.06
86.62
83.00
0.50
CelebA
Precision
Recall
pAcc
1
LPIPS
Pix2PixHD
54.47
2.44
57.48
0.73
68.04
80.18
63.75
4c-9
SPADE
33.42
2.35
2.72
67.99
80.11
63.80
0.07
RicycleGAN
56.56
2.53
52.93
2.36
66.95
79.41
62.50
0.15
SBGAN
50.92
2.44
39.28
3.62
67.42
80.12
61.42
4c-9
GANormer2
0.38
Cityscapes
Model
FID
IS
Precision
Recall f
mloU
pAcc
ARI 1
LPIPS
PiX2PIxHD
47.74
1.51
50.53
0.34
73.20
83.49
78.48
3e-8
SPADE
22.98
1.46
83.32
9.55
75.96
85.25
80.74
3e-3
BicycleGAN
25.02
1.56
63.01
9.49
67.24
78.43
72.20
7c-4
SBGAN
5.39
4e-8
GANormer2
5.95
1,63
84.12
37.72
76.11
85.58
80.12
0.27
COCO
Model
HD
18 1
Precision
Recall 1 16.35
mloU
pAcc
ARI 1
LPIPS
Pix2PixIID
18.91
17.97
80.91
58.75
71.94
68.20
1e:8
SPADE
22.80
16.29
79.00
10.63
55.72
69.55
67.73
0.19
BicycleGAN
SRGAN
45.94
14.31
58.93
10.27
47.82
54.58
62.29
4e-8
GANormer2
18.01
24.86
0.38
as illustrated by figures 9 and 12, we can even add or remove objects to the synthesized scene, while respecting interactions of occlusions, shadows and reflections, effectively achieving amodal completion, and potentially extrapolating beyond the training data (see sections C-B).
To quantify the disentanglement degree, refer to the DCI and modularity metrics [17, 83], which measure the correspondence between latent factors and image attributes. As in [33, 75], consider a sample set of synthesized CLEVR images, using pre- trained object detector to extract the objects and attributes. To further assess the controllability, we perform random perturbations over the latent variables used to generate the scenes, and estimate the mean correlation between the resulting semantic changes among object and property pairs. Intuitively, expect disentangled representation yield uncorrelated changes among objects and attributes. As shown in table GANformer2 attains higher disentanglement and stronger controllability than leading GAN-based and variational approaches, quantitatively confirming the efficacy of our approach.
Figure 9: Amodal completion for object removal.
Conclusion
In this paper, we have tackled the task visual synthesis, and introduced the GANformer2 model, an iterative transformer for transparent and controllable scene generation. Similar to how a person may approach the creative task, first sketches a high-level outline and only then fills the details. By equipping the network with strong explicit structure, endow it with the capacity to plan ahead its generation process, capture the dependencies and interactions between objects within the scene, and reason about them directly, moving step closer towards compositional generative modeling.
10

---

Acknowledgements
We performed the experiments for the paper on AWS cloud, thanks to Stanford HAI credit award. Drew A. Hudson (Dor) is a PhD student at Stanford University and C. Lawrence Zitnick is research scientist at Facebook FAIR. We wish to thank the anonymous reviewers for their thorough, insightful and constructive feedback, questions and comments.
Dor wishes thank Prof. Christopher D. Manning for the kind PhD support over the years that allowed this work to happen!
References
[1] Titas Anciukevicius, Christoph Lampert, and Paul Henderson. Object-centric image gen- eration with factored depths, locations, and appearances. arXiv preprint 1tfi:20004000444 2020.
[2] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference Machine Learning, pp. 166-175. PMLR, 2017.
[3] Relja Arandjelović and Andrew Zisserman. Object discovery with copy-pasting GAN. arXiv preprint arXiv: 1339,, 2019.
[4] Oron Ashual and Lior Wolf. Specifying object attributes and relations in interactive scene generation. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 4561-4569, 2019.
[5] Samaneh Azadi, Michael Tschannen, Eric Tzeng. Sylvain Gelly, Trevor Darrell, and Mario Lucic. Semantic bottleneck scene generation. arXiv preprint 1r 1911.11377, 2019.
[6] Samanch Azadi, Deepak Pathak, Sayna Ebrahimi, and Trevor Darrell. Compositional GAN: Learning image-conditional binary composition. International Journal of Computer Vision, 128 (10):2570-2585, 2020.
[7] Amir Bar, Roei Herzig, Xiaolong Wang, Anna Rohrbach, Gal Chechik, Trevor Darrell, and Amir Globerson. Compositional video synthesis with action graphs. arXiv preprint arXiv:2006.I 15327, 2020.
[8] David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Yun-Yan Zhu, and Antonio Torralba. Semantic photo manipulation with generative image prior. arXiv preprint arXiv:205..07727, 2020.
[9] David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba. Paint by word. arXiv preprint arXiv:2103. 10951, 2021.
[10] Irving Biederman. Recognition--y--ompoeee Psychological review, 94(2):115, 1987.
theory of human image understanding.
[11] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference on Learning Representations, ICLR, 2019.
[12] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and represen- tation. arXiv preprint arXiv: 1390, 2019.
[13] Lucy Chai, Jonas Wulff, and Phillip Isola. Using latent space regression to analyze and leverage compositionality in GANs, arXiv preprint arXiv:2103. 2021.
[14] Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In Proceedings of the IEEE international conference on computer vision, pp. 1511- 1520, 2017.
[15] Yunjey Choi, Min-Je Choi, Munyoung Kim, Jung-Wer Woo Ha, Sunghun Kim, and Jaegul Choo. StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In 2018 IEEE Conference
Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 pp. 8789-8797. IEFE Computer Society, 2018. doi: 10.1109/CVPP ww..H/CPRRR........ 2018.0099...

---

[16] Zhiwei Deng, Jiacheng Chen, Yifang Fu, and Greg Mori. Probabilistic neural programmed networks for scene generation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 4032-444-000 4042, 2018.
[17] Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
[18] Sébastien Ehrhardt, Oliver Groth, Aron Monszpart, Martin Engelcke, Ingmar Posner, Niloy J. Mitra, and Andrea Vedaldi. RELATE: physically plausible multi-object scene synthesis using structured latent spaces. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, De- cember 6-12, 2020, virtual, 2020.
[19] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: generative scene inference and sampling with object-centric latent representations. In 8th International Conference on Learning Representations, ICLR, 2020.
[20] S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
[21] Patrick Esser, Robin Rombach, and Björ Ommer. Taming transformers for high-resolution image synthesis. arXiv preprint Stdi:2011200884// 2020.
[22] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint
044333 2018.
[23] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint
D:Xiv:140.2661 2014.
[24] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-object rep- resentation learning with iterative variational inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2424-2433. PMLR, 2019.
[25] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. DRAW: A recurrent neural network for image generation. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1462 1471. JMLR.org. 2015.
[26] Fanny Guérin, Bernadette Ska, and Sylvie Belleville. Cognitive processing of drawing abilities.
Brain and Cognition, 40(3):464-478. 1999.
[27] Fanny Guérin, Bernadette Ska, and Sylvie Belleville. Cognitive processing of drawing abilities. Brain and Cognition, 40(3):464-478, 1999.
[28] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. GANspace: Discovering interpretable GAN controls. arXiv preprint 2XXv:20004002466, 2020.
[29] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus. S. V. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6626-6637, 2017.
[30] Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Generating multiple objects at spatially distinct locations. arXiv preprint arXiv: 1901. 111900000666, 2019.
[31] Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7986-7994,
2018. 2018.
12

---

[32] Drew A Hudson and Christopher D Manning. Learning by abstraction: The neural state machine. arXiv preprint (90.: 2019.
[33] Drew A Hudson and C. Lawrence Zitnick. Generative adversarial transformers. Proceedings of the 38th International Conference on Machine Learning, ICML, 2021.
[34]
Daniel Jiwoong Im, Chris Dongjoo Kim, Hui Jiang, and Roland Memisevic. Generating images with recurrent adversarial networks. arXiv preprinl arXiv: 1601v::602.000 002.000220011111 10, 2016.
[35] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 5967-5976. IEEE Computer Society, 2017. doi: 10. 10.110//CPR..0 201.R.017..3
[36] Ali Jahanian, Lucy Chai, and Phillip Isola. On the" steerability" of generative adversarial networks. arXiv preprint arXiv:
2019.
[37] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp.
1988-1997, 2017.
[38] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1219- 1 1228, 2018.
[39] Asako Kanezaki. Unsupervised image segmentation by backpropagation. In 2018 IEEE international conference on acousties, speech and signal processing (ICASSP), pp. 1547. IEEE, 2018
[40] Andrea Kantrowitz. The man behind the curtain: what cognitive science reveals about drawing. Journal of Aesthetic Education, 46(1):1-14, 2012.
[41] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In 6th International Conference on Learning Representations, ICLR, 2018.
[42] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of styleGAN. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 8107-8116. IEEE, 2020. doi: 10. 12110//V2222000022222222
w0..PP44260..220..... 2220.002233
[43] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár, Panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,, pp. 9404-9413, 2019.
[44] D Lane. Drawing and sketching: Understanding the complexity of paper- interactions within technology education. Handbook of technology education. Springer international publishing AG, pp. 385 402, 2018.
[45] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using generative adversarial network. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[46] Cheng-Han Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. MaskGAN: Towards diverse and interactive facial image manipulation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5549-5558. 2020.
[47] Chen-Hsuan Lin, Ersin Yumer, Oliver Wang. Eli Shechtman, and Simon Lucey. St-GAN: Spatial transformer generative adversarial networks for image compositing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9455-9464, 2018.
[48] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
[49] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. 8th International Conference on Learning Representations, ICLR, 2020.
13

---

[50] Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, and Hongsheng Li. Learning to pre- dict -iyaaee conditional convolutions for semantic image synthesis. arXiv preprint arXiv: 1910.0680, 2019.
[51] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centrie learning with slot attention. arXiv preprint arXiv:2006. /5055, 2020.
[52] Catarina Silva Martins. From scribbles to details. A Political Sociology of Educational Knowledge: Studies of Exclusions and Difference, pp. 103, 2017.
[53] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, (1287750):55995333 2015.
[54] Li Nanbo, Cian Eastwood, and Robert Fisher. Learning object-centric representations of multi- object scenes from multiple views. Advances in Neural Information Processing Systems, 33, 2020.
[55] Charlie Nash, SM Ali Eslami, Chris Burgess, Irina Higgins, Daniel Zoran, Theophane Weber, and Peter Battaglia. The multi-entity variational autoencoder. In NIPS Workshops, 2017.
[56] Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra. Block- GAN: Learning 3d object-aware scene representations from unlabelled images. arXiv preprint arXiv:2002.08988 2020.
[57] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11453-11464, 2021.
[58] Takeshi Okada and Kentaro Ishibashi. Imitation, inspiration, and creation: Cognitive process of creative drawing by copying others' artworks. Cognitive science, (1(7):1804-1888 2017.
[59] Dim P Papadopoulos, Youssef Tamaazousti, Ferda Ofli, Ingmar Weber, and Antonio Torralba. How to make a pizza: Learning a compositional layer-based GAN model. In proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8002-8011, 2019.
[60] Taesung Park, Ming- Yu Liu, Ting-Chun Chun Wang, and Jun- Zhu. Semantic image synthesis with ppatially-adaptive normalization. In IEEE Conference on Computer Vision and Putlern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2337-2346. Computer Vision Foundation IEEE, 2019. doi: 10.1 211110/CCRRR.2199.0222
[61] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pp. 3942-3951, 2018
[62] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234-241. Springer, 2015.
[63] RA Salome and BE Moore. The five stages of development in children's art, 2010.
[64] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of GANs for semantic face editing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9243-9252, 2020.
[65] Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferenc Huszár. Amortised map inference for image super-resolution. arXiv preprint arXiv:1610.000 2016.
[66] Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1): 89-96, 2007.
[67] Bob Steele. Draw me a story: An illustrated exploration of setwwing-a--languaae Portage & Main Press, 1998.
[68] Vadim Sushko, Edgar Schönfeld, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. arXiv preprint arXiv:2012.04781, 2020.
14

---

[69] Sjoerd van Steenkiste, Karol Kurach, Jürgen Schmidhuber, and Sylvain Gelly. Investigating object compositionality in generative adversarial networks. Neural Nenworks, 130:309-325, 2020.
[70] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems, pp.
2017.
[71] Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the GAN latent space. In International Conference Machine Learning, pp. 9786-97999 PMLR, 2020.
[72] Ting-Chun Wang, Ming-Yo Yu Liu, Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional GANs. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 2018.
[73] Xiaolong Wang and Abhinav Gupta. Generative image modeling using style and structure adversarial networks. In European conference on computer vision, pp. 318-335. Springer, 2016.
[74] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2.
https://e h/ggttthub.... cww watm//faceboorrsearcchie dettseerrcc//ettttoonn.. 2019.
[75] Zongze Wu, Dani Lischinski, and Eli Shechtman. StyleSpace analysis: Disentangled controls for StyleGAN image generation. arXiv preprint arXiv:2011 1. 122799, 2020.
[76] Kun Xu, Haoyu Liang, Jun Zhu, Hang Su, and Bo Zhang. Deep structured generative models. 2018.
arXiv preprint 1807.00
[77] Jianwei Yang, Anitha Kannan, Dhruv Batra, and Devi Parikh. Lr-GAN: Layered recursive generative adversarial networks for image generation. arXiv preprint arXiv: 1703.01500 2017.
[78] Denis Yarats and Mike Lewis. Hierarchical text generation and planning for strategic dialogue. In International Conference on Machine Learning, pp. 5591-5599. PMLR, 2018.
[79] Fangneng Zhan, Jiaxing Huang, and Shijian Lu. Hierarchy composition GAN for high-fidelig image synthesis.
[80] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision,
pp. 5907-5915, 2017.
[81] Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 7354-7363. PMLR, 2019.
[82] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
[83] Bo Zhao, Bo Chang, Zequn Jie, and Leonid Sigal. Modular generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV), pp. 150-165, 2018.
[84] Haitian Zheng, Zhe Lin, Jingwan Lu, Scott Cohen, Jianming Zhang, Ning Xu, and Jiebo
Luo. Semantic layout manipulation with high-resolution sparse attention. arXiv preprint arXiv:2012.07288, 2020.
[85] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE International Conference on Computer Vision. 1CCV, pp. 224112251,20 2017.
[86] Jun Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. arXiv preprint arXiv: 1156,, 2017.
[87] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. SEAN: image synthesis with semantic region-adaptive normalization. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 5103 5112-5112. IEEE, 2020.
15


Reviews:
[{"review_id": "usv3QXSByu-", "reviewer": "Reviewer_wMMf", "summary": "This paper proposes SceneGAN that uses a two-stage strategy to perform image generation. The SceneGAN is based on a previous transformer-based GAN model, it firstly uses iterative generation to generate scene layout then based on the layouts to perform image generation. The authors demonstrate their model's performance on clevr and real images and show some good results.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "originality: low\n\nThere is no new techniques introduced in the proposed model. The SceneGAN simply adopts a transformer gan as its architecture.\n\nquality:\n\n1. The authors used an iterative method to generate layouts, are there any experiments showing it's better than a single-pass generation?\n2. It seems that the authors didn't benchmark their numbers with available settings where other papers also used this two stage generation pipeline.\n\nmissing citation:\n[1] Compositional Video Synthesis with Action Graphs -> (compositional generation, CLEVR)\n[2] Probabilistic neural programmed networks for scene generation -> (compositional generation, CLEVR)\n\n\nclarity: The writing is pretty clear to me\n\nsignificance: As mentioned, there's no significantly new techniques proposed or any new insights that can be drew from this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No related concern", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.5-1 hour"}, "extra_text": {}}, {"review_id": "mtP_7zkcT63", "reviewer": "Reviewer_tVgy", "summary": "- This paper introduces a novel GANsformer-based architecture for two-staged, compositional image generation. The first stage samples a panoptic-segmentation-like instance mask of the scene which is re-painted in the second stage to yield a high-fidelity image consisting of multiple objects.\n- The explicit architectural biases w.r.t. scene layout, object instances, depth ordering and appearance modelling enable explainable and controllable image generation.\n- The paper presents a thorough quantitative analysis of its proposed approach conducted on five suitable and challenging datasets and compared to seven adequate baselines models.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Updated Review (24/08/21)\n\nAfter considering the additional experiments provided substantiating the claims w.r.t. to disentanglement and the additional baselines provided for related models (BlockGAN, GIRAFFE), my main concern about the weak corroboration of the paper's claims has been mitigated substantially. I appreciate the authors' effort of providing these additional experiments and thank them for their clarifying statements in the comments. I expect them to transfer the clarifications provided during the review process into the final manuscript - especially the disentanglement and controllability studies as well as the additional baseline results. In the light of the reviews and comments, I have increased my initial rating of 4 to 6.\n\n\n## Original Review\n\nTL;DR: The paper proposes a reasonable, two-stage image generation process using a compositional scene layout as intermediate representation and employing GANsformers for the modality translation tasks. While the paper presents a comprehensive experimental evaluation in terms of datasets and baselines, the lack of clarity in the presentation of the model's details and the weak corroboration of the claims about compositionality and controllability prevent it from meeting the bar for acceptance in its current form.\n\n### Pros and Cons\n- Pros\n  - The paper presents a reasonable coarse-to-fine drawing approach for image generation utilising a GANsformer for modality translation.\n  - The iterative layout generation of the model affords a compositional and interpretable image formation process.\n  - The authors perform a comprehensive evaluation across multiple datasets and against several suitable GAN-based baselines for image generation.\n- Cons\n  - The model architecture and new loss functions which are main contributions of the paper are only superficially described in prose and are never specified with equations. The way the model is presented makes it almost impossible, even for an expert in this domain, to faithfully re-create the network and replicate the results.\n  - The experiments section is unfortunately riddled with quality problems: The results tables (tab. 1, 2) feature unexplained metrics like P(recision), R(ecall), mIoU or pAcc which cannot be directly linked to any experiment or claim of the paper. Furthermore, a split of the COCO dataset is introduced for no apparent reason - why is this necessary?\n  - Claims about controllability and disentanglement properties of the model are only weakly corroborated by qualitative experiments in fig. 7 and in supplementary fig. 3 despite being among the main selling points for the compositional image generation model.\n  - The manuscript also lacks clarity of presentation in the captions which makes many figures very hard to interpret (e.g. fig. 4, 6, 7).\n\n### Originality\n- Framing image generation as a coarse-to-fine drawing process is reasonable and effective, although not novel; earlier methods (e.g. DRAW) have followed a similar analogy.\n- The paper presents a novel aspect in the two-stage \"translation\" process: (1) noise -> instance mask; (2) instance mask -> image. While each stage in itself is not entirely novel, the composition in this pipeline using GANsformers for the individual translation processes, is an original idea.\n\n### Quality\n- Fig. 6 is missing units on the x-axis.\n- In l. 321 fig. 5 is referenced for the visualization of a latent interpolation experiment where color, material, shape and position of an object are supposedly controlled independently. However, fig. 5 depicts the translation of segmentation maps into colored images. I suspect, fig. 7 was supposed to be referenced. However, fig. 7 is also very hard to interpret as its caption is not explaining what is truly happening in the picture strips. Fig. 3 of the supplementary material does a slightly better job in that regard.\n- incorrect statements:\n  - ll. 46-47: \"existing approaches fail to provide controllability at the level of the individual object or local region\" -> This is incorrect. For instance, [1, 16] allow to control the location and appearance of individual objects via designated elements of the latent code.\n  - ll. 95-96: \"the need for independence assumptions made by earlier works [16, 55, 68]\" -> This is also incorrect as [16, 68] explicitly model inter-object relationships in latent space to overcome independence assumptions of earlier works.\n  ll. 296-298: \"1.01 FID score when conditioned on a ground-truth layout. This result serves as an indication for the particular aptitude of the model for compositional scenes.\" -> If the ground-truth layout (i.e. instance segmentation map) is provided, the model is relieved from the hard task of composing a plausible scene layout and is only tasked with re-painting the semantic areas with suitable textures. It is well known that GANs excel at such a task (e.g. GauGAN). Consequently, this result does not corroborate the claim of the model's aptitude for compositionality.\n- typos, formatting errors:\n  - l. 56: ~~the~~ it is ready,\n  - l. 64: through **_** multiple\n  - l. 72: noise scheme **for** synthesizing\n  - ll. 142-143: the equations are missing equation numbers\n  - l. 146: ~~to~~ shape and impact\n  - l. 238: and **assess** the fidelity\n  - l. 259: that SceneGAN **possesses**\n  - l. 264: in modelling **compositional** scenes\n  - ll. 300-301: content **_** diversity\n  - ll. 322-323: the fact **that** it allocates\n- The manuscript occasionally uses embellished, slightly unsuitable choices of words for a scientific conference paper, e.g. \"celebrated StyleGAN2\" (l. 267), \"great Pix2PixHD repository\", \"the wonderful 1.01 FID score\" (ll. 296-297)\n\n### Clarity\n- Fig. 1 reads more like a system diagram than an architecture overview. The important details about model architecture, losses and training procedure are missing.\n- The manuscript would greatly benefit from a more explicit and self-contained presentation of the model using a network architecture diagram and equations for the newly introduced loss terms.\n- In tab. 1 most results seem to coincide with the numbers reported at https://github.com/dorarad/gansformer#evaluation, but some numbers differ -> Have the experiments been re-implemented or re-run or have the numbers just been copied?\n- In tab. 1, the P(recision)/R(ecall) evaluation appears out of nowhere with no justification or explanation in the text. What is the purpose of this evaluation and how does it support any of the paper's claims?\n- The paper features many very sparse figure captions to the point that the figures becomes uninterpretable, e.g. fig. 4, fig. 6, fig. 7.\n- The model in its current form is slightly limited by the need of paired training data featuring an RGB image and a segmentation mask. While it is hinted at the fact that an unpaired training setup could work as well (ll. 249-250), this is never explained in detail.\n\n### Significance\n- Compositional image generation is an important research problem in Computer Vision and a model which yields high-fidelity images while also providing scene layout masks and relative depth maps could certainly be useful for image generation and editing tools.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The authors have appropriately addressed one of the model's limitations in its training setup and the consequences of potential applications in section 5.", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6h"}, "extra_text": {}}, {"review_id": "_xYuP2ElPEc", "reviewer": "Reviewer_99ts", "summary": "This paper proposes a novel method for compositional generation for 2D segment-like elements.\nIt decomposes the task into two stages: a sketching stage for soft layout generation and a painting stage for refining the layout and generating photo-realistic images conditioned on the soft layout. The paper uses different modified versions of GANsformer in each stage.  Experiments are carried out on multiple datasets with better results compared with prior works. The proposed framework is capable of separating style from structure and decomposition of multiple individual object properties.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I recommend accepting the paper. Overall, this paper is fairly well written and organized. The proposed SceneGAN is definitely interesting and addressing a very important problem - compositional generation of 2D segment-like elements. It undoubtedly reaches the new State-of-The-Art in this area. I summarize the strengths and weaknesses as follows:\n\nStrengths:\n\n- It successfully fulfills the goal of separating structure from style with careful design of the two careful-designed different-functioning stages and the losses of each stage.\n- Soundly evidenced by ablation studies and extensive experiments in the paper, the proposed novel semantic consistency objective functions particularly hit the nail on the head, ensuring semantic consistency while not hinder diversity.\n- This paper marches a powerful step towards explainable and controllable generative scene decomposition.\n- Results are very promising in terms of diversity, fidelity and decomposition; the efficacy of the methods is clearly demonstrated through experiments carried on multiple datasets.\n\nWeakness:\n\n- A few typos\n    - line 64 (through multiple)\n    - line 72 (fir->for)\n    - line 193 ($d(x,y)_i$ -> $d_i(x,y)$)\n    - line 300 (poentially -> potentially)\n    - line 301 (content diversity)\n\nQuestions:\n- Since the transformers are already capable of reasoning contextual information, why do the transformers in the sketching stage still need to be recurrent? I'm aware that Fig.6 shows the empirical proof. Is there any theoretical or intuitive evidence?\n- How does the segmentation network in the semantic matching loss work with the blurry and meaningless generated images of the early/initial training stage? Will this not introduce misleading gradients?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors realize the potential negative social impact of fake or unauthentic content synthesis. However, this is one of the most common stumbling blocks in this area and we believe more and more anti-fake technologies will help us embrace more positive opportunities.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}, {"review_id": "9VtP151XB6o", "reviewer": "Reviewer_Rc1b", "summary": "This paper introduces an object-oriented GAN model, which generates an image of a scene in a two stage process. First, it generates a layout, where different masks show what object should be painted in which part of the image. Second, this layout is refined by a second network that converts it into a real-world-looking image. Both stages are based on modified GANformer networks. Finally, the paper introduces two novel losses. These make sure that the generated image reflects the generated layout, and that each portion of the image (as given by the layout) looks realistically. The method is fairly well evaluated, but the paper lacks comparison with two key baselines, that is BlockGAN (Nguyen-Phuoc et al) and GIRAFFE (Niemeyer & Geiger), which are not cited. Modelling decisions are evaluated in an ablation study.  The introduced method is supervised and requires paired (image, segmentation masks; this is in contrast to what the authors claim, see my comments below) inputs.\n\n\n# UPDATE\nThe authors addressed my questions about unpaired training and using CNNs instead of Transformers. I'm still waiting for details of the CNN experiment, but I am already happy to increase my score to 7.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n- The task of object-oriented scene generation is well established. This work seems to achieve new SOTA results on a number of datasets.\n- The majority of this work is a combination of well-known techniques (layout-conditioned image generation, GANformer), but it also introduces two novel and conceptually sound losses that significantly improve the results.\n- AFAIK, this is the first application of transformer to object-centric scene generation.\n- The related work seems to be adequately cited, but it misses two key approaches that should be compared against, that is BlockGAN (Nguyen-Phuoc et al) and GIRAFFE (Niemeyer & Geiger).\n- The paper makes many references to \"sketching\": I think you should cite SPIRAL (Ganin et al) to say that sketching was considered as a stand-alone task in a GAN.,\n- The author claim that their method produces unsupervised depth estimates. While it very much look like it does, it is very likely that some of the previous object-centric generative model (Monet, Iodine, Genesis+v2) should provide those estimates as well---it is just that they were never analyzed from this point of view. Moreover, BlocKGAN, GIRAFEE and Obsurf (Stelzner et. al.) do provide depth estimates. So this point is not novel, and is hardly differentiation from prior art.\n\nQuality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?\n\nGood:\n- The submission is technically sound. The overall approach makes sense, and there are ablation studies that validate modelling decisions.\n- The numerical results are SOTA (though it would be easier to interpret PR AUC instead of separate precision and recall scores), and the qualitative results look superb.\n- The authors acknowledge the limitation that segmentation masks are required.\n\nBad:\n- In L46-47 you write: \", existing approaches fail to provide controllability at the level of the individual object or local region\". This is very much not true. Models like Monet, Iodine, Genesis (+v2), Obsurf and many others, not to mention HoloGAN and GIRAFFE do provide that controllability. Please remove that claim.\n- it seems that one of the key components of the method is using GANformer. I wonder how important it is, i.e. what are the advantages, both theoretical and empirical, over using a CNN-based generator? Please provide ablations where you change the decoder to a convolutional one and discuss.\n- The authors claim that paired inputs are not necessary to train the model. I **strongly disagree**. Currently, the \"layout\" stage is pre-trained with externally-provided semantic segmentation. I believe that this is vital. The generated layouts are used as targets in an auxiliary loss for the discriminator. Therefore, if the \"layout\" stage is not pre-trained, it is almost a given that the generated layout would converged to empty images, which is a degenerate solution that is trivially matched by the discriminator.\n\nClarity: \n- It is first unclear that the paper is about supervised learning. I only realized it by noticing \"unsupervised depth ordering\" in L54. Please make it explicit very early on.\n- The term \"panoptic segmentation\" is defined only in L284-285. Please move it to the very beginning of the paper as some readers (myself included) in this community are unaware of this term.\n- \n- L1: delete \"explored\".\n- L6: unclear what is \"honed and refined\".\n- L9: I am not sure how \"efficiency\" is encouraged by this approach.\n- L26: \"imply semantic information\"---I disagree; this is dependent on prior knowledge about the world which is absent here.\n- L68: what is \"amodal completion\" please define before using.\n- L72: :s/fir/for/\n- L136 seems to suggest that both the latents and the image have the same dimensionality \"d\". Is that true? If so, this is non-standard; please make it more explicit.\n- L158: the 2nd part of (1) does not make sense given the \"spatial distribution\" assumption, please rephrase.\n- L162: The way you use depth in L193 in the softmax, and this relation, suggests that d(x, y) is negative depth. Please explain that in the paper.\n- L172-173: how do you choose the number of segments to generate per iteration?\n- L179-189 and L220-226: you only mention changing the key function, but this seems insufficient. You need to also modify the value function if you want to use the auxiliary inputs in the attention. Also, since you want to augment these auxiliary inputs as a by-product, you probably also change the query function? In total, it seems to me that you are using normal self-attention but concatenate the previous latents to the current inputs. Please clarify that.\n- Tab 1. \"over 10 random\", you meant to add \"runs\"?\n\nSignificance: \n- Object-centric image (or scene) generation is currently very fashionable. This paper introduces a major improvement into current modelling framework. I would say that the most important pieces are the auxiliary losses introduced in the paper. To know how significant the current work really is, we would need to see comparisons with GIRAFFE and BlockGAN. But as of now, I think it is a solid piece of work that will benefit the community a lot.\n\nI will increase my score if comparisons with GIRAFFE and BlockGAN are favourable and if you remove some of the claims (as discussed above).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: - The proposed method tackles an important problem, takes a reasonable and novel approach (the loss function and recurrent generation), and the experiments are well planned, executed, and show SOTA results. 
- The rebuttal addressed the major concerns from the reviewers well and thus some reviews raised the score.
- The clarity is good enough but some figures and the details of the architecture need to be clarified further. 

Decision: Accept (Poster)