CANVASEMB: Learning Layout Representation with Large-scale Pre-training for Graphic Design

Layout representation , which models visual elements in a canvas and their interrelations , plays a crucial role in graphic design intelligence . With a large variety of layout designs and the unique characteristic of layouts that visual elements are defined as a list of categorical ( e.g . shape type ) and numerical ( e.g . position and size ) properties , it is challenging to learn a general and compact representation with limited data . Inspired by the recent success of self-supervised pre-training techniques in various natural language processing tasks , in this paper , we propose CanvasEmb ( Canvas Embedding ) , which pre-trains deep representation from unlabeled graphic designs by jointly conditioning on all the context elements in the same canvas , with a multi-dimensional feature encoder and a multi-task learning objective . The pre-trained CanvasEmb model can be fine-tuned with just one additional output layer and with a small size of training data to create models for a wide range of downstream tasks . We verify our approach with presentation slides data . We construct a large-scale dataset with more than one million slides , and propose two novel layout understanding tasks with human labeling sets , namely element role labeling and image captioning . Evaluation results on these two tasks show that our model with fine-tuning achieves state-of-the-art performances . Furthermore , we conduct a deep analysis aiming to understand the modeling mechanism of CanvasEmb , and demonstrate its great potential use on more applications such as layout auto completion and layout retrieval . 1 INTRODUCTION . Graphic design leverages layout to set up and arrange visual elements in a canvas for conveying message in different types of documents , while layout representation is the reversed process to understand visual elements and their inter-relations in a canvas , which is the key for the analysis ( Stoffel et al. , 2010 ) , retrieval ( Beusekom et al. , 2006 ) and generation ( Li et al. , 2020b ; Lee et al. , 2020 ) of graphic designs . However , elements in a layout are complex , which are defined with multi-dimensional properties such as type ( e.g. , text box , image or button ) , position and color . For example , the web page and presentation slide shown in Figure 1 is defined by a lot of settings , as each example is constructed by several elements and each element is defined by several proprieties . Due to the complex and sparse features of elements , as well as the rich diversity of layouts , learning a general and compact layout representation is challenging with limited amount of data . Previous works related to layout representations ( Li et al. , 2019 ; Tabata et al. , 2019 ; Lee et al. , 2020 ) are mostly task-oriented . They simplify the layout only as the positions of elements , and directly optimize task-specific labels with less than a few thousands instances . Recently a majority of self-supervised pre-trained models such as ELMO ( Peters et al. , 2018 ) , GPT ( Radford , 2018 ) and BERT ( Devlin et al. , 2019 ) have shown promising results in improving a variety of natural language processing ( NLP ) tasks . The success of pre-trained models in NLP has inspired us to learn contextual layout representations from large-scale unlabeled graphic designs , which can facilitate various downstream tasks for design intelligence . As one highly related work , LayoutLM ( Xu et al. , 2019 ) is a document pre-trained model incorporating both text content and layout information for scanned documents . However , it is difficult to generalize to other document types , since its input is word-level and it defines layout only as the word position , which is insufficient to describe a layout in graphic design . In this paper , we present CanvasEmb , a large-scale pre-trained model for learning contextual layout representation . It is designed to pre-train deep representation from unlabeled graphic designs by jointly conditioning on all the context elements in the same canvas , and the pre-trained CanvasEmb model can be fine-tuned with just one additional output layer and with a small size of training data to create models for a wide range of downstream tasks . Specifically , we define a generic and high-coverage vocabulary to describe element properties in the canvas . A feature encoder is designed to jointly incorporate multi-dimensional properties , and it is developed with the multi-layer Transformer ( Devlin et al. , 2019 ) for modeling element contexts . To ensure the representation conditioning on all dimensions of element contexts , we adopt the masked language modeling strategy with a multi-task objective , where we randomly mask some properties of elements for prediction in the pre-training . To verify our approach , we construct a large-scale dataset with more than one million presentation slides containing rich layout meta-information for pre-training . We then propose two novel downstream tasks for layout understanding with human labeling sets to evaluate the performance of our pre-trained CanvasEmb model . The first task is element role labeling . Only given the information of layout , the goal is to classify the semantic role of each element ( e.g. , title , subtitle ) . The second task is image captioning , which detects if a text box and an image in a slide belongs to the image captioning relation . Experimental results on the two tasks show that fine-tuning the CanvasEmb model achieves state-of-the-art performance . Furthermore , we conduct deep analysis to understand the modeling mechanismCanvasEmb . Also , we demonstrate the great potential use of our pre-trained CanvasEmb with two extended applications , including layout auto completion ( Li et al. , 2020b ) and layout retrieval . The contributions of this work are as follows : • We propose CanvasEmb , which to the best of our knowledge is the first pre-trained model for layouts in graphic design . It can be fine-tuned with a small size of training data for a wide range of downstream tasks . • We construct a large-scale dataset of presentation slides with rich layout information , as well as two novel tasks for layout understanding ( i.e. , element role labeling and image captioning ) with human labeling sets . • We demonstrate that our model achieves state-of-the art performances on the two downstream tasks , and show the potential for more applications such as layout auto-completion and layout retrieval . 2 RELATED WORK . Layout representation is the focal point of design in rich media , including presentation slides , magazines , comics , posters and web pages . High-quality representations can be conductive to multiple practical design tasks . Early works on design layout or document layout mainly rely on templates ( Hurst et al. , 2009 ; Damera-Venkata et al. , 2011 ) or heuristic rules ( O ’ Donovan et al. , 2014 ; Tabata et al. , 2019 ) and require professional knowledge and manual efforts . To efficiently facili- tate the problem-solving aspects of sketching in the graphic designs , Todi et al . ( 2016 ) propose an interactive layout design tool which uses a real-time layout optimiser without requiring extensive input . However , these methods are restricted and usually fail to model the rich varieties of media information . Xinru Zheng & Lau ( 2019 ) make use of the content information to model the graphic design layouts in the purely data-driven scenario to adapt to the contents to be laid out . Recently , there is a trend to adopt neural networks and deep learning methods to promote automating layout to be more efficiently . For example , to be more user-friendly , Pang et al . ( 2016 ) adopt attention mechanisms to trace the user ’ s attention , and Lee et al . ( 2020 ) improve the conventional GANbased methods ( Li et al. , 2019 ; Xinru Zheng & Lau , 2019 ) to explicitly model relationships among components and user-specified constraints . BERT4Rec ( Sun et al. , 2019 ) employs the deep bidirectional self-attention to model user behavior sequences . And Li et al . ( 2020b ) develop Transformerbased tree decoders on the task of auto completion of user interface layout design , which can ease the efforts of UI designers and developers . However , previous works typically deal with limited kinds of design elements and fail to give general and scalable solutions to layout representation learning . Enlightened by the significant impact of large-scale pre-trained models in the area of NLP ( Peters et al. , 2018 ; Radford , 2018 ; Devlin et al. , 2019 ; Yang et al. , 2019 ) and multi-modal learning ( Sun et al. , 2019 ; Lu et al. , 2019 ; Li et al. , 2020a ) , our work implements the attention-based Transformer framework enhanced with pretraining to propose a data-driven and scalable method that captures contextual information for layout representation , which can be applied well on downstream tasks in graphic design . 3 LAYOUT IN GRAPHIC DESIGN . Layout in graphic design refers to the way in which we arrange the visual elements on a canvas . Though some settings might vary specific to different document types , there exists basic characteristics of elements that make up the content of layouts ( example shown in Figure 1 ) : • Type Properties . Elements can be text boxes , pictures or lines . According to the semantic roles , elements can be divided into title , subtitle , button or other placeholders . • Geometry Properties . Position and size indicate the elements ’ placement in the layout . Besides , z-order is the ordering of overlapping two-dimensional elements , and rotation describes the an element ’ s circular movement . • Color Properties . Color is one of the most straightforward visual feature , including the RGBA channels and extra features such as color gradient . • Content-related Properties . Though user contents are separated from the layout , some content-related properties ( e.g. , text font size and font type ) can affect the layout arrangement . Elements are complex and sparse , composed with the above properties of either categorical ( e.g. , shape type , color ) or numerical ( e.g. , position , font size , word count ) values . Hence , layouts are diverse and complicated for modeling . In the next section , we will introduce our approach for layout representation learning . 4 MODELING . We present our model CanvasEmb , which inputs elements with multi-dimensional properties and outputs representation for the layout . To train our model , we adopt the two-stage learning framework , namely pre-training and fine-tuning . 4.1 MODEL ARCHITECTURE . We formulate the input as a sequence of visual elements { x0 , x1 , x2 , ... , xn } in the layout , where each element xi is defined with m properties { p1i , p2i , ... , pmi } . Here x0 is the sequence representation which is randomly initialized . Figure 2 shows the overview architecture of our model , which is similar to BERT ( Devlin et al. , 2019 ) . The feature embedding encodes high dimensional properties for elements , and is concatenated with the transformer encoder to model the global context of elements . The output representation can be further used to make prediction of element-level and layout-level labels , as well as the relations between the elements , with an extra task-specific prediction layer . Here , we introduce the details of model components . Feature Embedding . For the i-th element xi = [ p1i ; p2i ; ... ; pmi ] , the embedding ei is obtained by concatenating m property embeddings : ei = Θ ( 1i ⊕ 2i ⊕ ... ⊕ mi ) , ( 1 ) where ⊕ is the concatenation operator and Θ is a non-linear transform function . For each channel j , the corresponding property pji contains multi-dimensional values . For example , given a 2-dim numerical property pji = [ p j,1 i ; p j,2 i ] ( e.g. , element size with height and width ) , the embedding in this channel can be calculated as : ji = ξ j ( pj,1i ) ⊕ ξ j ( pj,2i ) , ( 2 ) where pj , ki represents the k-th value in p j i and ξ j is the embedding function . There are two types of embedding functions . For properties with categorical values such as type and color , we use the embedding matrix as the learning parameter . For properties with numerical values such as position and size , the positional encoding ( Vaswani et al. , 2017 ) is adopted : PE ( p j , k i , 2h ) = sin ( p j , k i /10000 2h/dj , k ) ( 3 ) PE ( p j , k i , 2h+ 1 ) = cos ( p j , k i /10000 2h/dj , k ) ( 4 ) where dj , k means the embedding dimension assigned to pj , ki . Transformer Encoder . On top of the feature embeddings , we use a transformer encoder ( Vaswani et al. , 2017 ) to encode the element contexts . Similar to BERT ( Devlin et al. , 2019 ) , the multi-layer transformer with the multi-head self-attention mechanism enables to capture correlations between different elements and property fields . Finally , we can get the low-dimensional representations { h ( L ) 0 ; h ( L ) 1 ; ... , h ( L ) n } for all elements from the last , i.e . the L-th encoding layer .

Summary: This paper applies state-of-the-art transformer-based neural networks to layout representation learning of slides. The most notable contribution of this paper is the construction of large-scale parsed slide layout dataset. This paper proposes to pre-train the network on this large-scale dataset without masked reconstruction strategy and verifies it with several subtasks including element role labeling, image captioning, auto-completion and layout retrieval, with a comparison to a decision-tree based method as baseline.