Compositional Transformers for Scene Generation

1 Introduction . Drawing , the practice behind human visual and artistic expression , can essentially be defined as an iterative process . It starts from an initial outline , with just a few strokes that specify the spatial arrangement and overall layout , and is then gradually refined and embellished with color , depth and richness , until a vivid picture eventually emerges . These initial schematic sketches can serve as an abstract scene representation that is very concise , yet highly expressive : several lines are enough to delineate three dimensional structures , account for perspective and proportion , convey shapes and geometry , and even imply semantic information [ 52 , 63 , 67 ] . A large body of research in psychology highlights the importance of sketching in stimulating creative discovery [ 40 , 58 ] , fostering cognitive development [ 27 ] , and facilitating problem solving [ 26 , 44 ] . In fact , a large variety of generative and RL tasks either in the visual [ 5 ] , textual [ 22 , 78 ] or symbolic modalities [ 2 ] , can benefit from the same strategy – prepare a high-level plan first , and then carry out the details . At the heart of this hierarchical strategy stands the principle of compositionality – where the meaning of the whole can be derived from those of its constituents . Indeed , the world around us is highly structured . Our environment consists of a varied collection of objects , tightly interconnected through dependencies of all sorts : from close-by to long-range , and from physical and dynamic to abstract and semantic [ 10 , 66 ] . As pointed out by prior literature [ 24 , 51 , 55 , 57 ] , compositional representations are pivotal to human intelligence , supporting our capabilities of reasoning [ 32 ] , planning [ 53 ] , learning [ 54 ] and imagination [ 6 ] . Realizing this principle , and explicitly capturing the objects and elements composing the scene , is thereby a desirable goal , that can make the generative process more explainable , controllable , versatile and efficient . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Yet , the vast majority of generative models do not directly reflect this compositionality so intrinsic to visual scenes . Rather , most networks , and GANs in particular , aim to generate the entire scene all at once from a single monolithic latent space , through a largely opaque transformation . Consequently , while they excel in producing stunning , strikingly-realistic portrayals of faces , sole centralized objects , or natural scenery , they still struggle to faithfully mimic richer or densely-packed scenes , as those common to everyday contexts , falling short of mastering their nuances and intricacies [ 5 , 50 , 76 ] . Likewise , while there is some evidence for property disentanglement within their latent space to independent axes of variation at the global scale [ 28 , 36 , 64 , 71 ] , most existing GANs fail to provide controllability at the level of the individual object or local region . Understanding of the mechanisms by which these models give rise to an output image , and the transparency of their synthesis process , thereby remain rather limited . Motivated to alleviate these shortcomings and make generative modeling more compositional , interpretable and controllable , we propose GANformer2 , a structured object-oriented transformer that decouples the visual synthesis task into two stages : planning and execution . We begin by sampling a set of latent variables , corresponding to the objects and entities that compose the scene . Then , at the planning stage , we transform the latents into a schematic layout – a scene representation that depicts the object segments , reflecting their shapes , positions , semantic classes and relative ordering . Its construction occurs in an iterative manner , to capture the dependencies and interactions between different objects . Next , at the execution stage , the layout is translated into the final image , with the latents cooperatively guiding the content and style of their respective segments through bipartite attention [ 33 ] . This approach marks a shift towards a more natural process of image generation , in which objects and the relations among them are explicitly modeled , while the initial sketch is successively tweaked and refined to ultimately produce a rich photo-realistic scene . We study the model ’ s behavior and performance through an extensive set of experiments , where it attains state-of-the-art results in both conditional and unconditional synthesis , as measured along metrics of fidelity , diversity and semantic consistency . GANformer2 reaches high versatility , demonstrated through multiple datasets of challenging simulated and real-world scenes . Further analysis illustrates its capacity to manipulate chosen objects and properties in an independent manner , achieving both spatial disentanglement and separation between structure and style . We notably observe amodal completion of occluded objects , likely driven by the sequential nature of the computation . Meanwhile , inspection of the produced layouts and their components shed more light upon the synthesis of each resulting scene , substantiating the model ’ s transparency and explainability . Overall , by integrating strong compositional structure into the model , we can move in the right direction towards multiple favorable properties : increasing robustness for rich and diverse visual domains , enhancing controllability over individual objects , and improving the generative process ’ s interpretability . 2 Related Work . Over the last years , the field of visual sythesis has witnessed astonishing progress , driven in particular by the emergence of Generative Adversarial Networks [ 23 ] . Tremendous strides have been made in a wide variety of tasks , from image [ 11 ] to video generation [ 7 ] , to super-resolution [ 45 ] , style transfer [ 15 ] and translation [ 35 ] . Meanwhile , a growing body of literature explored object discovery and sequential scene decomposition , albeit in the context of variational inference [ 1 , 16 , 19 , 49 ] . DRAW [ 25 ] uses recurrent attention to encode and decode MNIST digits , while AIR [ 20 ] , MONet [ 12 ] and IODINE [ 24 ] reconstruct simple simulated scenes by iteratively factorizing them into semantically meaningful segments . We draw inspiration from this pioneering line of works and marry the merits of its strong structure with the scalability and fidelity obtained by GANs , yielding a compositional model that successfully creates intricate real-world scenes , while offering a more transparent and controllable synthesis along the way , conceptually meant to better resemble the causal generative process and serve as a more natural emulation of how a human might approach the creative task . To this end , our model leverages the GANformer architecture we introduced earlier this year [ 33 ] – a bipartite transformer that uses soft-attention to cooperetaviely translate a set of latents into a generated image [ 33 , 70 ] . In GANformer2 , we take this idea a step further , and show how we can use the bipartite transformer as a backbone for a more structured process , that enables not only an implicit , but also an explicit modeling of the objects and constituents composing the image ’ s scene . Our iterative design endows the GANformer2 model with the flexibility to consider the correlations and interactions not only among latents but between synthesized scene elements directly , circumventing the need for partial independence assumptions made by earlier works [ 18 , 56 , 69 ] , and thereby managing to scale to natural real-world scenes . Concurrently , the explicit object-oriented structure enhances the model ’ s compositionallity so to outperform past coarse-to-fine cascade [ 80 ] , layered [ 34 ] , and hierarchical approaches [ 5 ] , which , like us , employ a two-stage layout-based procedure , but contrary to our work , do so in a non-compositional fashion , and indeed were mostly explored within limited domains , such as indoor NYUv2 [ 73 ] , faces [ 41 ] , pizzas [ 59 ] , or bird photographs [ 77 ] . Our model links to research on conditional generative modeling [ 72 , 85 , 87 ] , including semantic image synthesis and text-to-image generation . The seminal Pix2Pix [ 35 ] and SPADE [ 60 ] works respectively adopt either a U-Net [ 62 ] or spatially-adaptive normalization [ 61 ] to map a given input layout into an output high-resolution image . However , beyond their notable weakness of being conditional and thus unable to create scenes from scratch , their reliance on both static class embeddings for feature modulation , along with perceptual or feature-matching reconstruction losses , render their sample variation rather low [ 68 , 86 ] . In our work , we mitigate this issue by proposing new purely generative structural loss functions : Semantic Matching and Segment Fidelity , which maintain the one-to-many relation that holds between layouts and images , and consequently , as demonstrated in section 3.3 , significantly enhance the output diversity . While GANformer2 is an unconditional model , it shares some high-level ideas with conditional text-to-image [ 30 , 31 ] and scene-to-image [ 4 , 38 ] techniques , which use semantic layouts as a bottleneck intermediate representation . But whereas these works focus on heavily-engineered pipelines consisting of multiple pre- and post-processing stages and different sources of supervision , including textual descriptions , scene graphs , bounding boxes , segmentation masks and images , we propose a simpler , more elegant design , with a streamlined architecture and fewer supervision signals – relying on images and auto-predicted layouts only , to widen its applicability across domains . Finally , our approach relates to prior works on visual editing [ 8 , 9 , 46 , 84 ] and image compositing [ 3 , 13 , 47 , 79 ] , but while these methods modify an existing source image , GANformer2 stands out being capable of creating structured manipulatable scenes from scratch . 3 The GANformer2 model . The GANformer2 model is a compositional object-oriented transformer for visual scene generation . It features a set of latent variables corresponding to the objects and entities composing the scene , and proceeds through two synthesis stages : ( 1 ) a sequential planning stage , where a scene layout – a collection of interacting object segments , is being created ( section 3.2 ) , followed by ( 2 ) a parallel execution stage , where the layout is being refined and translated into an output photo-realistic image ( section 3.3 ) . Both stages are implemented using customized versions of the GANformer model ( section 3.1 ) – a bipartite transformer that uses multiple latent variables to modulate the evolving visual features of a generated image , so to create it in a compositional fashion . In this paper , we explore learning from two training sets : of images and layouts . Specifically , we use panoptic segmentations [ 43 ] , which specify for every segment si its instance identity pi and semantic category mi , but other types of segmentations can likewise be used . The images and layouts can either be paired or unpaired , and our training scheme accommodates both options : either by pre-training each stage independently first and then fine-tune them jointly in the paired case , or training them together from scratch in the unpaired one ( See section F for further details ) . 3.1 Generative Adversarial Transformers . The GANformer [ 33 ] is a transformer-based Generative Adversarial Network : it consists of a generator G ( Z ) = X that translates a partitioned latent Zk×d = [ z1 , ... , zk ] into an image XHWc , and a discriminator D ( X ) that seeks to discern between real and fake images . The generator begins by mapping the normally-distributed latents into an intermediate space W = [ w1 , ... , wk ] using a feed-forward network . Then , it sets an initial 4×4 grid , which gradually evolves through convolution and upsampling layers to the final image . Contrary to traditional GANs , the GANformer also incorporates bipartite-transformer layers into the generator , which compute attention between the k latents and the image features , to support spatial and contentual region-wise modulation : Attention ( X , W ) = softmax ( q ( X ) k ( W ) T√ d ) v ( W ) ( 1 ) u ( X , W ) = γa ( X , W ) LayerNorm ( X ) + βa ( X , W ) ( 2 ) Where the equations respectively express the attention and modulation between the latents and the image ; q ( · ) , k ( · ) , v ( · ) are the query , key , and value mappings ; and γ ( · ) , β ( · ) compute multiplicative and additive styles ( gain and bias ) as a function of Attention ( X , W ) . The update rule lets the latents shape and impact the image regions as induced by the key-value attention image-to-latents assignment computed between them . This encourages visual compositionality , where different latents specialize to represent and control various semantic regions or visual elements within the generated scene . As we will see below , for each of the two generation stages , we couple a modified version of the GANformer with novel loss functions and training schemes . One key point in which we notably depart from the original GANformer regards to the number of latent variables used by the model . While the GANformer employs a fixed number of k latent variables , GANformer2 supports a variable number of latents , thereby becoming more flexible in generating scenes of diverse complexity .

Summary: This paper introduces an object-oriented GAN model, which generates an image of a scene in a two stage process. First, it generates a layout, where different masks show what object should be painted in which part of the image. Second, this layout is refined by a second network that converts it into a real-world-looking image. Both stages are based on modified GANformer networks. Finally, the paper introduces two novel losses. These make sure that the generated image reflects the generated layout, and that each portion of the image (as given by the layout) looks realistically. The method is fairly well evaluated, but the paper lacks comparison with two key baselines, that is BlockGAN (Nguyen-Phuoc et al) and GIRAFFE (Niemeyer & Geiger), which are not cited. Modelling decisions are evaluated in an ablation study.  The introduced method is supervised and requires paired (image, segmentation masks; this is in contrast to what the authors claim, see my comments below) inputs.   # UPDATE The authors addressed my questions about unpaired training and using CNNs instead of Transformers. I'm still waiting for details of the CNN experiment, but I am already happy to increase my score to 7.