LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators

1 INTRODUCTION . Graphic design is an important visual communication tool in our modern world , encompassing everything from book covers to magazine layouts to web design . Whereas methods for generating realistic natural-looking images have made significant progress lately , particularly with Generative Adversarial Networks ( GANs ) ( Karras et al. , 2018 ) , methods for creating designs are far more primitive . This is , in part , due to the difficulty of finding data representations suitable for learning . Graphic designs are normally composed of vector representations of primitive objects , such as polygons , curves and ellipses , instead of pixels laid on a regular lattice . The quality and content of a design depends on the presence of elements , their attributes , and their relations to other elements . The visual perception of design depends on the arrangement of these elements ; misalignment of two elements of just a few millimeters can ruin the design . Training from images of designs using conventional GANs synthesizes the layouts in pixel space , and thus mixes up layout and its rendering , and thus would be unlikely to capture layout styles well . Modeling such highly-structured data using neural networks is of great interest as they usually represent human abstract knowledge about the visual world ( Zitnick & Parikh , 2013 ; Song et al. , 2017 ) and how this knowledge are expressed via documents and designs ( Deka et al. , 2017 ; Yang et al. , 2017 ) . This paper introduces LayoutGAN , a novel GAN which directly synthesizes a set of graphical elements in a design . In a given design problem , a fixed set of element classes ( e.g. , “ title , ” “ figure ” ) is specified in advance . In our network , each element is represented by its class probabilities and its geometric parameters , i.e. , bounding-box keypoints . The generator takes as input graphic elements with randomly-sampled class probabilities and geometric parameters , and arranges them in a design ; the output is the refined class probabilities and geometric parameters of the design elements . The generator has the desirable property of being permutation-invariant : it will generate the same layout if we re-order the input elements . We propose two kinds of discriminator networks for this structured data . The first is similar in structure to the generator : it operates directly on the class probabilities and geometric parameters of the elements . Though effective , it is not sensitive enough to misalignment and occlusion between elements . The second discriminator operates in the visual domain . Like a human viewer who judges ∗Work done during Jianan Li 's internship at Adobe Research . a design by looking at its rasterized image , the relations among different elements can be evaluated well by mapping them to 2D layouts . Then Convolutional Neural Networks ( CNNs ) can be used for layout optimization as they are specialized in distinguishing visual patterns including but not limited to misalignment and occlusion . However , the key challenge is how to map the geometric parameters to pixel-level layouts differentiably . One approach would be to render the graphic elements into bitmap masks using Spatial Transformer Networks ( Jaderberg et al. , 2015 ) . But we found that filled pixels within the design elements cause occlusions and are ineffective for back-propagation , for example , when a small polygon hides behind a larger one . We experimented with bitmap mask rendering but it was not successful . In this paper , we propose a novel differentiable wireframe rendering layer that rasterizes both synthesized and real structured data of graphic elements into wireframe images , upon which a standard CNN can be used to optimize the layout across the visual and the graphic domain . The wireframe rendering discriminator has several advantages . First , convolution layers are very good at extracting spatial patterns of images so that they are more sensitive to alignment . Second , the rendered wireframes make elements visible even when they overlap and thus the network is alleviated from inferring the occlusions that may occur in other renderings such as masks . We evaluate the LayoutGAN for several different tasks , including a sanity test on MNIST digits , generating page layouts from labeled bounding boxes , generating clipart abstract scenes , tangram graphic design , and mobile app design layouts . In each case , our method successfully generates layouts respecting the types of elements and their relationships for the problem domain . In summary , the LayoutGAN comprises the following contributions : 1 . A Generator that directly synthesizes structured data , represented as a resolution-independent set of labeled graphic elements in a design . 2 . A differentiable wireframe rendering layer which allows the Discriminator to judge alignment from discrete element arrangements . 2 RELATED WORK . Structured data generation . Convolutional networks have been shown successful for generating data in regular lattice , such as images ( Radford et al. , 2015 ) , videos ( Vondrick et al. , 2016 ) and 3D volumes ( Yan et al. , 2016 ; Wu et al. , 2016 ) . When generating highly-structured data , such as text ( Donahue et al. , 2015 ) and programs ( Reed & De Freitas , 2015 ) , recurrent networks are often the first choice ( Sutskever et al. , 2014 ) , especially equipped with attention ( Bahdanau et al. , 2014 ) and memory modules ( Graves et al. , 2014 ) . Recently , researchers show that convolutional networks can be also used to synthesize sequences ( Oord et al. , 2016 ; van den Oord et al. , 2016 ) using autoregressive models . However , in many cases , an object has no sequential order ( Vinyals et al. , 2015 ) , but is a set of elements , e.g . point clouds . Fan et al . ( 2017 ) propose a point set generation network for synthesizing 3D point clouds of the object shape from a single images . It is further paired with a point set classification network ( Charles et al. , 2017 ) for auto-encoding 3D point clouds ( Achlioptas et al. , 2017 ) . Our work extends the set representation to more general primitive objects , i.e. , labeled polygons . Meanwhile , researchers also model the structured data of connected elements using graph convolutions ( Kipf & Welling , 2017 ) . Data-driven graphic design . Automating layout is a classic problem in graphic design ( Hurst et al. , 2009 ) . O ’ Donovan et al . ( 2014 ) formulate an energy function by assembling various heuristic visual cues and design principles to optimize single-page layouts , and extend this to an interactive tool ( O ’ Donovan et al. , 2015 ) . The model parameters are learned from a small number of example designs . Pang et al . ( 2016 ) optimize layout for desired gaze direction . Deka et al . ( 2017 ) collect a mobile app design database for harnessing data-driven applications and present preliminary results of learning similarities of pixel-level textural/non-textual masks for design search , but do not learn models from this data . Swearngin et al . ( 2018 ) propose an interactive system that converts example design screenshots to vector graphics for designers to re-use and edit . Bylinskii et al . ( 2017 ) analyze the visual importance of graphic designs and use saliency map as the driving force to assist retargeting and thumbnailing . Previous methods have learned models for other graphic design elements , such as fonts ( O ’ Donovan et al. , 2014 ) and colors ( O ’ Donovan et al. , 2011 ) . These are orthogonal to the layout problem , and could be combined in future work . No previous method has learned to create design or layout from large datasets , and no previous work has applied GANs to layout . 3D scene synthesis . Interior scene synthesis and furniture layout generation draws great interest in graphics community . Early approaches focus on optimization of hand-crafted design princi- Generator Encoder Stacked Relation Module Real / Fake Prediction fc fc fc fc fc fc fc Decoder CNN Differentiable Wireframe Rendering Wireframe Layout f ( p1 , x1 ) f ( p2 , x2 ) … . f ( pi , xi ) … . f ( pn , xn ) f ( p1 , x1 ) f ( p2 , x2 ) … . f ( pi , xi ) … . f ( pn , xn ) f ’ ( p1 , x1 ) f ' ( p2 , x2 ) … . f ' ( pi , xi ) … . f ' ( pn , xn ) Self-attention ➕ Wireframe Rendering Discriminator ( Graphic - > Visual Domain ) Class Probabilities Geometric Parameters { ( p1 , θ1 ) , … , ( pN , θN ) } p1 θ1 p2 θ2 ● ● ●pN θN Random Input Class Probabilities Geometric Parameters { ( p1 ’ , θ1 ’ ) , … , ( pN ’ , θN ’ ) } p1 ’ θ1 ’ p2 ’ θ2 ’ ● ● ●pN ’ θN ’ Class Probabilities Geometric Parameters { ( p1 ’ , θ1 ’ ) , … , ( pN ’ , θN ’ ) } p1 ’ θ1 ’ p2 ’ θ2 ’ ● ● ●pN ’ θN ’ Figure 1 : Overall architecture of LayoutGAN . The generator takes as input graphic elements with randomly sampled class probabilities and geometric parameters from Uniform and Gaussian distribution respectively . An encoder embeds the input and feeds them into the stacked relation module , which refines the embedded features of each element in a coordinative manner by considering its semantic and spatial relations with all the other elements . Finally , a decoder decodes the refined features back to class probabilities and geometric parameters . The wireframe rendering discriminator feeds the generated results to a differentiable wireframe rendering layer which raterizes the input graphic elements into 2D wireframe images , upon which a CNN is applied for layout optimization . ples ( Merrell et al. , 2011 ) and learning statistical priors of pairwise object relationships ( Fisher et al. , 2012 ) due to limited data . Wang et al . ( 2018a ) recently proposed a sequential decision making approach to indoor scene synthesis . In each step , a CNN is trained to predict either location or category of one object by looking at the rendered top-down views . This resembles our wireframe rendering discriminator , in the sense of using convolutions to capture spatial patterns of layouts . 3 LAYOUTGAN . This section describes our data and model representations . 3.1 DESIGN REPRESENTATION . In our model , a graphic design is comprised of a set of N primitive design elements { ( p1 , θ1 ) , · · · , ( pN , θN ) } . Each element has a set of geometric parameters θ , and a vector of class probabilities p. The entries of these variables are problem-dependent . For example , document layouts include 6 classes , e.g . “ title ” and “ picture ” while clipart layouts include 6 classes , e.g . “ boy ” and “ hat ” . For 2D point set generation ( in MNIST digits ) , θ ≡ [ x , y ] , representing the coordinates of each point ; for bounding-box generation in document layout , θ ≡ [ xL , yT , xR , yB ] , representing the top-left and bottom-right coordinates of each bounding box ; for layouts with scale and flip ( clipart abstract scenes ) , θ ≡ [ x , y , s , l ] , representing the center coordinates , scale and flip of each element .

Summary: This paper presents a novel GAN framework for generating graphic layouts which consists of a set of graphic elements which are geometrically and semantically related. The generator learns a function that maps input layout ( a random set of graphic elements denoted by their classes probabilities and and geometric parameters) and outputs the new contextually refined layout. The paper also explores two choices of discriminators: (1) relation based discriminator which directly extracts the relations among different graphic elements in the parameter space, and (2) wireframe rendering discriminator which maps graphic elements to 2D wireframe images using a differentiable layer followed by a CNN for learning the discriminator. The novel GAN framework is evaluated on several datasets such as MNIST, document layout comparison and clipart abstract scene generation