Compositional Transformers for Scene Generation

1 Introduction . Drawing , the practice behind human visual and artistic expression , can essentially be defined as an iterative process . It starts from an initial outline , with just a few strokes that specify the spatial arrangement and overall layout , and is then gradually refined and embellished with color , depth and richness , until a vivid picture eventually emerges . These initial schematic sketches can serve as an abstract scene representation that is very concise , yet highly expressive : several lines are enough to delineate three dimensional structures , account for perspective and proportion , convey shapes and geometry , and even imply semantic information [ 52 , 63 , 67 ] . A large body of research in psychology highlights the importance of sketching in stimulating creative discovery [ 40 , 58 ] , fostering cognitive development [ 27 ] , and facilitating problem solving [ 26 , 44 ] . In fact , a large variety of generative and RL tasks either in the visual [ 5 ] , textual [ 22 , 78 ] or symbolic modalities [ 2 ] , can benefit from the same strategy – prepare a high-level plan first , and then carry out the details . At the heart of this hierarchical strategy stands the principle of compositionality – where the meaning of the whole can be derived from those of its constituents . Indeed , the world around us is highly structured . Our environment consists of a varied collection of objects , tightly interconnected through dependencies of all sorts : from close-by to long-range , and from physical and dynamic to abstract and semantic [ 10 , 66 ] . As pointed out by prior literature [ 24 , 51 , 55 , 57 ] , compositional representations are pivotal to human intelligence , supporting our capabilities of reasoning [ 32 ] , planning [ 53 ] , learning [ 54 ] and imagination [ 6 ] . Realizing this principle , and explicitly capturing the objects and elements composing the scene , is thereby a desirable goal , that can make the generative process more explainable , controllable , versatile and efficient . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Yet , the vast majority of generative models do not directly reflect this compositionality so intrinsic to visual scenes . Rather , most networks , and GANs in particular , aim to generate the entire scene all at once from a single monolithic latent space , through a largely opaque transformation . Consequently , while they excel in producing stunning , strikingly-realistic portrayals of faces , sole centralized objects , or natural scenery , they still struggle to faithfully mimic richer or densely-packed scenes , as those common to everyday contexts , falling short of mastering their nuances and intricacies [ 5 , 50 , 76 ] . Likewise , while there is some evidence for property disentanglement within their latent space to independent axes of variation at the global scale [ 28 , 36 , 64 , 71 ] , most existing GANs fail to provide controllability at the level of the individual object or local region . Understanding of the mechanisms by which these models give rise to an output image , and the transparency of their synthesis process , thereby remain rather limited . Motivated to alleviate these shortcomings and make generative modeling more compositional , interpretable and controllable , we propose GANformer2 , a structured object-oriented transformer that decouples the visual synthesis task into two stages : planning and execution . We begin by sampling a set of latent variables , corresponding to the objects and entities that compose the scene . Then , at the planning stage , we transform the latents into a schematic layout – a scene representation that depicts the object segments , reflecting their shapes , positions , semantic classes and relative ordering . Its construction occurs in an iterative manner , to capture the dependencies and interactions between different objects . Next , at the execution stage , the layout is translated into the final image , with the latents cooperatively guiding the content and style of their respective segments through bipartite attention [ 33 ] . This approach marks a shift towards a more natural process of image generation , in which objects and the relations among them are explicitly modeled , while the initial sketch is successively tweaked and refined to ultimately produce a rich photo-realistic scene . We study the model ’ s behavior and performance through an extensive set of experiments , where it attains state-of-the-art results in both conditional and unconditional synthesis , as measured along metrics of fidelity , diversity and semantic consistency . GANformer2 reaches high versatility , demonstrated through multiple datasets of challenging simulated and real-world scenes . Further analysis illustrates its capacity to manipulate chosen objects and properties in an independent manner , achieving both spatial disentanglement and separation between structure and style . We notably observe amodal completion of occluded objects , likely driven by the sequential nature of the computation . Meanwhile , inspection of the produced layouts and their components shed more light upon the synthesis of each resulting scene , substantiating the model ’ s transparency and explainability . Overall , by integrating strong compositional structure into the model , we can move in the right direction towards multiple favorable properties : increasing robustness for rich and diverse visual domains , enhancing controllability over individual objects , and improving the generative process ’ s interpretability . 2 Related Work . Over the last years , the field of visual sythesis has witnessed astonishing progress , driven in particular by the emergence of Generative Adversarial Networks [ 23 ] . Tremendous strides have been made in a wide variety of tasks , from image [ 11 ] to video generation [ 7 ] , to super-resolution [ 45 ] , style transfer [ 15 ] and translation [ 35 ] . Meanwhile , a growing body of literature explored object discovery and sequential scene decomposition , albeit in the context of variational inference [ 1 , 16 , 19 , 49 ] . DRAW [ 25 ] uses recurrent attention to encode and decode MNIST digits , while AIR [ 20 ] , MONet [ 12 ] and IODINE [ 24 ] reconstruct simple simulated scenes by iteratively factorizing them into semantically meaningful segments . We draw inspiration from this pioneering line of works and marry the merits of its strong structure with the scalability and fidelity obtained by GANs , yielding a compositional model that successfully creates intricate real-world scenes , while offering a more transparent and controllable synthesis along the way , conceptually meant to better resemble the causal generative process and serve as a more natural emulation of how a human might approach the creative task . To this end , our model leverages the GANformer architecture we introduced earlier this year [ 33 ] – a bipartite transformer that uses soft-attention to cooperetaviely translate a set of latents into a generated image [ 33 , 70 ] . In GANformer2 , we take this idea a step further , and show how we can use the bipartite transformer as a backbone for a more structured process , that enables not only an implicit , but also an explicit modeling of the objects and constituents composing the image ’ s scene . Our iterative design endows the GANformer2 model with the flexibility to consider the correlations and interactions not only among latents but between synthesized scene elements directly , circumventing the need for partial independence assumptions made by earlier works [ 18 , 56 , 69 ] , and thereby managing to scale to natural real-world scenes . Concurrently , the explicit object-oriented structure enhances the model ’ s compositionallity so to outperform past coarse-to-fine cascade [ 80 ] , layered [ 34 ] , and hierarchical approaches [ 5 ] , which , like us , employ a two-stage layout-based procedure , but contrary to our work , do so in a non-compositional fashion , and indeed were mostly explored within limited domains , such as indoor NYUv2 [ 73 ] , faces [ 41 ] , pizzas [ 59 ] , or bird photographs [ 77 ] . Our model links to research on conditional generative modeling [ 72 , 85 , 87 ] , including semantic image synthesis and text-to-image generation . The seminal Pix2Pix [ 35 ] and SPADE [ 60 ] works respectively adopt either a U-Net [ 62 ] or spatially-adaptive normalization [ 61 ] to map a given input layout into an output high-resolution image . However , beyond their notable weakness of being conditional and thus unable to create scenes from scratch , their reliance on both static class embeddings for feature modulation , along with perceptual or feature-matching reconstruction losses , render their sample variation rather low [ 68 , 86 ] . In our work , we mitigate this issue by proposing new purely generative structural loss functions : Semantic Matching and Segment Fidelity , which maintain the one-to-many relation that holds between layouts and images , and consequently , as demonstrated in section 3.3 , significantly enhance the output diversity . While GANformer2 is an unconditional model , it shares some high-level ideas with conditional text-to-image [ 30 , 31 ] and scene-to-image [ 4 , 38 ] techniques , which use semantic layouts as a bottleneck intermediate representation . But whereas these works focus on heavily-engineered pipelines consisting of multiple pre- and post-processing stages and different sources of supervision , including textual descriptions , scene graphs , bounding boxes , segmentation masks and images , we propose a simpler , more elegant design , with a streamlined architecture and fewer supervision signals – relying on images and auto-predicted layouts only , to widen its applicability across domains . Finally , our approach relates to prior works on visual editing [ 8 , 9 , 46 , 84 ] and image compositing [ 3 , 13 , 47 , 79 ] , but while these methods modify an existing source image , GANformer2 stands out being capable of creating structured manipulatable scenes from scratch . 3 The GANformer2 model . The GANformer2 model is a compositional object-oriented transformer for visual scene generation . It features a set of latent variables corresponding to the objects and entities composing the scene , and proceeds through two synthesis stages : ( 1 ) a sequential planning stage , where a scene layout – a collection of interacting object segments , is being created ( section 3.2 ) , followed by ( 2 ) a parallel execution stage , where the layout is being refined and translated into an output photo-realistic image ( section 3.3 ) . Both stages are implemented using customized versions of the GANformer model ( section 3.1 ) – a bipartite transformer that uses multiple latent variables to modulate the evolving visual features of a generated image , so to create it in a compositional fashion . In this paper , we explore learning from two training sets : of images and layouts . Specifically , we use panoptic segmentations [ 43 ] , which specify for every segment si its instance identity pi and semantic category mi , but other types of segmentations can likewise be used . The images and layouts can either be paired or unpaired , and our training scheme accommodates both options : either by pre-training each stage independently first and then fine-tune them jointly in the paired case , or training them together from scratch in the unpaired one ( See section F for further details ) . 3.1 Generative Adversarial Transformers . The GANformer [ 33 ] is a transformer-based Generative Adversarial Network : it consists of a generator G ( Z ) = X that translates a partitioned latent Zk×d = [ z1 , ... , zk ] into an image XHWc , and a discriminator D ( X ) that seeks to discern between real and fake images . The generator begins by mapping the normally-distributed latents into an intermediate space W = [ w1 , ... , wk ] using a feed-forward network . Then , it sets an initial 4×4 grid , which gradually evolves through convolution and upsampling layers to the final image . Contrary to traditional GANs , the GANformer also incorporates bipartite-transformer layers into the generator , which compute attention between the k latents and the image features , to support spatial and contentual region-wise modulation : Attention ( X , W ) = softmax ( q ( X ) k ( W ) T√ d ) v ( W ) ( 1 ) u ( X , W ) = γa ( X , W ) LayerNorm ( X ) + βa ( X , W ) ( 2 ) Where the equations respectively express the attention and modulation between the latents and the image ; q ( · ) , k ( · ) , v ( · ) are the query , key , and value mappings ; and γ ( · ) , β ( · ) compute multiplicative and additive styles ( gain and bias ) as a function of Attention ( X , W ) . The update rule lets the latents shape and impact the image regions as induced by the key-value attention image-to-latents assignment computed between them . This encourages visual compositionality , where different latents specialize to represent and control various semantic regions or visual elements within the generated scene . As we will see below , for each of the two generation stages , we couple a modified version of the GANformer with novel loss functions and training schemes . One key point in which we notably depart from the original GANformer regards to the number of latent variables used by the model . While the GANformer employs a fixed number of k latent variables , GANformer2 supports a variable number of latents , thereby becoming more flexible in generating scenes of diverse complexity .

Summary: - This paper introduces a novel GANsformer-based architecture for two-staged, compositional image generation. The first stage samples a panoptic-segmentation-like instance mask of the scene which is re-painted in the second stage to yield a high-fidelity image consisting of multiple objects. - The explicit architectural biases w.r.t. scene layout, object instances, depth ordering and appearance modelling enable explainable and controllable image generation. - The paper presents a thorough quantitative analysis of its proposed approach conducted on five suitable and challenging datasets and compared to seven adequate baselines models.