CANVASEMB: Learning Layout Representation with Large-scale Pre-training for Graphic Design
Authors: Yuxi Xie
Danqing Huang
Jinpeng Wang
Chin-Yew Lin
Venue:  (2021)
Keywords: Layout Representation
Pre-training

Abstract: Layout representation, which models visual elements in a canvas and their inter-relations, plays a crucial role in graphic design intelligence.
With a large variety of layout designs and the unique characteristic of layouts that visual elements are defined as a list of categorical (e.g. shape type) and numerical (e.g. position and size) properties, it is challenging to learn a general and compact representation with limited data. Inspired by the recent success of self-supervised pre-training techniques in various natural language processing tasks, in this paper, we propose CanvasEmb (Canvas Embedding), which pre-trains deep representation from unlabeled graphic designs by jointly conditioning on all the context elements in the same canvas, with a multi-dimensional feature encoder and a multi-task learning objective. The pre-trained CanvasEmb model can be fine-tuned with just one additional output layer and with a small size of training data to create models for a wide range of downstream tasks. We verify our approach with presentation slides data. We construct a large-scale dataset with more than one million slides, and propose two novel layout understanding tasks with human labeling sets, namely element role labeling and image captioning. Evaluation results on these two tasks show that our model with fine-tuning achieves state-of-the-art performances. Furthermore, we conduct a deep analysis aiming to understand the modeling mechanism of CanvasEmb, and demonstrate its great potential use on more applications such as layout auto completion and layout retrieval.

Under review as a conference paper at ICLR 2021
CANVASEMB: LEARNING LAYOUT REPRESENTATION WITH LARGE-SCALE PRE-TRAINING FOR GRAPHIC DESIGN
Anonymous authors Paper under double-blind review
ABSTRACT
Layout representation, which models visual elements in canvas and their inter- relations, plays crucial role in graphic design intelligence. With a large variety of layout designs and the unique characteristic of layouts that visual elements are defined as a list categorical (e.g. shape type) and numerical (c.g. position and size) properties, it is challenging to learn general and compact representation with limited data. Inspired by the recent success of self-supervised pre-training techniques in various natural language processing tasks, in this paper, we propose CanvasEmb (Canvas Embedding). which pre-trains deep representation from un- labeled graphic designs by jointly conditioning all the context elements in the same canvas, with a multi-dimensional feature encoder and a multi-task learning objective. The pre-trained CanvasEmb model can be fine-tuned with just one addi- tional output layer and with small size of training data to create models for a wide range of downstream tasks. We verify our approach with presentation slides data. We construct large-scale dataset with more than one million slides, and propose two novel layout understanding tasks with human labeling sets, namely element role labeling and image captioning. Evaluation results these two tasks show that our model with fine-tuning achieves state-of-the-arr performances. Further- more, conduct deep analysis aiming understand the modeling mechanism Can, and demonstrate its great potential use on more applications such layout auto completion and layout retrieval.
INTRODUCTION
Graphic design leverages layout to set up and arrange visual elements in canvas for conveying message in different types of documents, while layout representation the reversed process to understand visual elements and their inter-relations in canvas, which is the key for the analy- sis (Stoffel et al. 2010), retrieval (Beusekom et al., 2006) and generation (Li et al.] 2020bb Lee al. (2000) of graphic designs. However, elements layout complex, which dimti-dimeniionll properties such as type (e.g., text box, image or button), position and color. For example, the web page and presentation slide shown Figure is defined by lot of settings, as each example is constructed by several elements and each element is defined by several propricties. Due to the complex and sparse features of elements, as well as the rich diversity of layouts, learning general and compact layout representation is challenging with limited amount of data.
detined with
Previous works related to layout representations (Li et al., 2019; Tabata et al. 2019 Lee et al., 2020) are mostly task-oriented. They simplify the layout only as the positions of clements, and directly optimize task task-specific labels with less than a few thousands instances. Recently a majority of self-supervised pre-trained models such as ELMO (Peters et al.) (018). GPT (Radford) (018) and BERT (Devlin et al. 2019) have shown promising results in improving a variety of natural language processing (NLP) tasks. The success of pre-trained models in NLP has inspired us to learn contextual layout representations from large-scale unlabeled graphic designs, which can facilitate various downstream tasks for design intelligence. As one highly related work, LayoutLM (Xu et al.) 2019) is a document pre-trained model incorporating both text content and layout information for scanned documents. However, it is difficult generalize to other document types, since its input is

---

Under review as a conference paper at ICLR 2021
Properties
Text
Title
Top Left: (0.08, 0.46)
INTRODUCTION VANDO MO OEAL SUE
Height: 0.12
Z-order:
Rotation:
Color: AFFFFFF
Font Type: Corbel
Word Count:
(a)
(b)
Figure 1: Example layouts for different document types: (a) Web Page (b) Slide. As shown the right, elements have multiple properties, which make layout complex and diverse.
word-level and defines layout only as the word position, which is insufficient to describe layout in graphic design.
In this paper, we present CanvasEmb, large-scale pre-trained model for learning contextual layout representation.
designed to pre-train deep representation from unlabeled graphic designs by jointly conditioning all the context elements in the same canvas, and the pre-trained CanvasEmb model can be fine-tuned with just one additional output layer and with a small size of training data to create models for wide range of downstream tasks. Specifically, we define generic and high-coverage vocabulary to describe element properties in the canvas. A feature encoder is de- signed to jointly incorporate nulti-dimensional properties, and it is developed with the multi-layer Transformer (Devlin et al. (019)) for modeling element contexts. To ensure the representation con- ditioning all dimensions of element contexts, adopt the masked language modeling strategy with a multi-task objective, where we randomly mask some properties of elements for prediction in the pre-training.
To verify our approach, we construct a large-scale dataset with more than one million presentation slides containing rich layout meta-information for pre-training. We then propose two novel down- stream tasks for layout understanding with human labeling sets to evaluate the performance of our pre- pre-trained CanvasEmb model. The first task element role labeling. Only given the information of layout, the goal is to classify the semantic role of each element (e.g., title, subtitle). The second task image captioning, which detects a text box and an image in a slide belongs to the image cap- tioning relation. Experimental results on the two tasks show that fine-tuning the CanvasEmb model achieves state-of-the-art performance. Furthermore, we conduct deep analysis to understand the modeling mechanismCanvvashn CanvasEmb with two extended applications, including layout auto completion (Li et al. 2020b) and layout retrieval.
Also, demonstrate the great potential use of our pre-trained
The contributions of this work are as follows:
We propose CanvasEmb, which to the best our knowledge the first pre-trained model for layouts in graphic design. It can be fine-tuned with small size of training data for wide range of downstream tasks.
We construct a large-scale dataset of presentation slides with rich layout information, as well as two novel tasks for layout understanding (i.e., element role labeling and image captioning) with human labeling sets.
We demonstrate that our model achieves state-of-the art performances on the two down- stream tasks, and show the potential for more applications such as layout auto-completion and layout retrieval.
2 RELATED WORK
Layout representation is the focal point of design in rich media, including presentation slides, mag- azines, comics, posters and web pages. High-quality representations can be conductive to multi- ple practical design tasks. Early works design layout or document layout mainly rely tem- plates (Hurst et al. 2009; Damera- Venkata et al. 201T) or heuristic rules (O (o'Donvvan et al., 2014] Tabata et al. 2019) and require professional knowledge and manual efforts. To efficiently facili-
2

---

Under review as a conference paper at ICLR 2021
tate the croblem--olviig aspects of sketching in the graphic designs, Todi al. (2016) propose interactive layout design tool which uses a real-time layout optimiser without requiring extensive input. However, these methods information. Xinru Zheng & Lau 2019) make use of the content information to model the graphic design layouts in the purely data-driven scenario to adapt to the contents be laid out.
restricted and usually fail to model the rich varieties of media
Recently, there is a trend to adopt neural networks and deep learning methods to promote automating layout be more efficiently. For example, be more user-friendly, Pang al. (2016) adopt atten- tion mechanisms to trace the user's attention, and Lee et al. (2020) improve the conventional GAN- based methods (Li al. 2019; Xinru Zheng & Lau 2019) to explicitly model relationships among components and user-specified constraints. BERT4Rec (Sun et al. [2019) employs the deep bidirec- tional self-attention to model user behavior sequences. And Li et al. (2020b) develop Transformer- based tree decoders on the task of auto completion of user interface layout design, which can ease the efforts of UI designers and developers.
However, previous works typically deal with limited kinds of design elements and fail to give gen- eral and scalable solutions to layout representation learning. Enlightened by the significant impact of large-scale pre-trained models in the area of NLP (Peters et al.] [0188] Radford 20188 Devlin ct al.
2019 Yang et al., 2019) and multi-modal learning (Sun et al., 2019 Lu et al. 2019 Li 2020a). our work implements the attention based Transformer framework enhanced with pre- training to propose a data-driven and scalable method that captures contextual information for layout representation, which can be applied well downstream tasks in graphic design.
LAYOUT IN GRAPHIC DESIGN
Layout in graphic design refers to the way in which we arrange the visual elements on canvas. Though some settings might vary specific different document types, there exists basic character- istics of elements that make up the content of layouts (example shown in Figure
Type Properties. Elements can be text boxes, pictures or lines. According to the semantic roles, elements can be divided into title, subtitle, button other placeholders.
Geometry Properties. Position and size indicate the elements' placement in the layout. Besides, z-order is the ordering of overlapping two-dimensional elements, and rotation describes the an element's circular movement.
Color Properties. Color is one of the most straightforward visual feature, including the RGBA channels and extra features such as color gradient.
Content-related Properties. Though user contents are separated from the layout, some content-related properties (e.g., text font size and font type) can affect the layout arrange- ment.
Elements complex and sparse, composed with the above properties of either categorical (c.g., shape type, color) or numerical (e.g., position, font size, word count) values. Hence, layouts are diverse and complicated for modeling. In the next section, will introduce approach for layout representation learning.
4 MODELING
We present our model CanvasEmb, which inputs elements with multi-dimensional properties and outputs representation for the layout. To train our model, we adopt the two-stage learning frame- work, namely pre-training and fine-tuning.
4.1 MODEL ARCHITECTURE
We formulate the input sequence of visual elements (Xo, X1, ..., } } in the layout, where each element x; is defined with m properties pp..p?.... "?.....p'"' p."h Here Xy is the sequence representa- tion which is randomly initialized. Figure shows the overview architecture of our model, which similar to BERT (Devlin et al. 2019). The feature embedding encodes high dimensional prop- erties for elements, and is concatenated with the transformer encoder model the global context
Xx.,
3

---

Under review as a conference paper at ICLR 2021
Sequence Features
Element Features
Attention
T
Element
Element States
AA:
CAI
(A)
Embedding
Transformer Encoder
Property Embeddings
Element Embeddings
Ey
teature
Property
Vectors
Feature Embedding
Input Elements
X1
Xjr1
Xy-1
XW
Figure 2: Overall Architecture of CanvasEmb.
of elements. The output representation can be further used make prediction element- and layout-level labels, as well as the relations between the elements, with an extra task-specific prediction layer. Here, we introduce the details of model components.
Feature Embedding. For the i-th element concatenating m property embeddings:
[p!: """ the embedding is obtained by
e; O of e eT"),
(1)
where + is the concatenation operator and θ is non-linear transform function.
For each channel j. the corresponding property p contains multi-dimensional values. For example, given 2-dim numerical property p :p! (e.g.. element size with height and width), the embedding in this channel can be calculated as:
of = ( ( (x)
(2)
where of represents the k-th value in P and ę is the embedding function.
There are two types of embedding functions. For propertics with categorical values such as type and color, the embedding matrix the learning parameter. For properties with numerical values such as position and size, the positional encoding (Vaswani et al. (017)) is adopted:
PE 2h) sin(p) /100002h/440
(3)
PE k 2h 1) cos(p)* /100002hh//4 et al. 2017) to encode the element contexts. Similar to BERT (Devlin et al.) 2019), the multi-layer transformer with the multi-head self-attention mechanism enables to capture correlations between different elements and property fields. Finally, we can get the low-dimensional representations (hg (b) h (L). :..., (())) for all elements from the last, iee. the I--th encoding layer.
(4)
where disk means the embedding dimension assigned to p..
Transformer Encoder. On top of the feature embeddings, we use a transformer encoder (Vaswani
4.2 PRE-TRAINING CANVASEMB
We adopt the masked language modeling (MLM) the objective of CanvasEmb pre-training, which has been proven effective in several domains (Devlin et al. 2019] Sun et al. 2019 Lu et al.) 2019). To enable the model learn correlations from different properties, of properties to mask for an element during training. CanvasEmb trained to approximately maxi- mize the pseudo likelihood Ex-D Σn ΣØ= P(pllp." (),6 (Sun et al. 2019), where D is the real data distribution, 0 represents the model parameters, p, represents the properties masking yi of
randomly select any type

---

Under review as a conference paper at ICLR 2021
the i-th element, and a means all the elements in the input excluding for the i-th one. Particularly, we adopt cross- cross-entropy loss and MSE loss for classification and regression tasks respectively:
Σ
Σ - log Pppipp (x,
L(0)
JEMo
Σ
Ip' P.l2
JEMu
Here Me and M, represent the index sets of categorical and numerical properties, Pf is the model prediction.
4.3 FINE-TUNING CANVASEMB
For different downstream tasks, feed the same input as in the pre-training phase, and fine-tune the model with extra task-specific layers:
Element-Level tasks aimed to predict specific features of the elements, e.g. properties which are different from but correlated to the input ones.
Element-te to- no-Element tasks predict the relations between pair of elements. In CanvasEmb, build query- attention-based mechanism for the relation prediction (Wang et al. 2017):
R(x,, x,) = =.i.j + Cga.
exp W (L). W W
(1)
0i Ej exp W (D)
Wy
where W and W3s represent trainable transforming weight matrix. Sequence-Level tasks utilize the Xo representation token make prediction.
EXPERIMENT
5.1 DATASET AND TASK DEFINITION
Pre-training Dataset. CanvasEmb designed to represent various types of graphic designs. this paper, we adopt presentation slides an example, huge amount of slides are publicly available and the understanding of slides would enable many applications. We first crawl presentation slides from the web to conduct a pre-training dataset with one million pages of slides" For each element (e.g., a textbox or a table) in page of the slide, we further extract their properties as defined in Section 3
Element Role Labeling, This task is to detect the semantic role of each element on a canvas, which one of the most important problems in Robotic Process Automation. As shown in Figure 3 each element is assigned with one of the following labels: ritle, subtitle, footer, decorator, heading, caption, and other. We create a dataset containing 48, 309 slides and split it into 38, 819 for training and 9, 490 for testing. Each element on the slides is annotated by five labelers and we use majority voting to assign the final label.
Image Captioning. Given pair of an image and text box in slide, the goal is detect if the pair belongs to image captioning or not. In the example shown in Figure33bb, there are four caption pairs on the right of the slide. With similar annotation pipeline, finally get 2, 996 slides, divided as 1, 496 for training and 1, 500 for testing.
5.2
SET UP
Model and Training Parameters: Following Devlin (2019), we testify the effect of model size by comparing two model sizes: CanvasEmb (Base) (L=3, H=64) and CanvasEmb (Large) (L==6,
We plan to open source the script for crawling slides.
5

---

Under review as a conference paper at ICLR 2021
Budget Development Fall Financial Review
Presidents of the Time
Auurpptoon
Collage Initaal
Heading
sua 2011
Puloo
Lo contretteen en 1to Lag
BILL
TTe bnie wosteg orpie vvey
undsup 2001 anm0
Other
sould On Cann plar (2.AM))
Wiltion (DN (1S1s 2920/4
ailk rilaelly Viain
Caption
Figure 3: Examples for two layout understanding tasks. For (a) element role labeling, element in blue color indicates ritle, purple for heading, green for caption and red for other. For (b) image captioning, four images each with text box in orange color are image caption pairs.
H=128), where L and H denote the number of encoder layers and the hidden size, respectively. And the sequence length during training is 16 for both models. During pre-training, we randomly select one type of property to mask for each sample by replacing with [MASK] (90% of the time) or random values (10% of the time). For multi-task learning objectives, the loss of each prediction task weighted according to the current performance on the validation set, where the weight for task E MM is log max( accuracy, 1) and for task E _M is log min( (equered.eror,, 100). For fine-tuning, we adopt the a-balaaced focal loss (Lin et al., 2017) to address the label unbalance. The trade-off parameters
10
focal loss is 2. All models are trained using Adam (Kingma Bal (015) with mini-batch size 64. The learning rate is initially set 10 and adaptive learning rate decay applied. We also adopt early stopping and use gradient clipping (Pascanu al. 2012).
Since there are few strong neural methods that are applicable for the graphic layout design, we compare our proposed model CanvasEmb against the traditional method, i.e. Decision Tree implemented as GBDT (Ke al. 2017). The input is the same CanvasEmb and the same focal loss settings.
Baselines:
5.3 RESULT ANALYSIS
Table Results for element role labeling.
100% labeled data
titlc
subtitle
footer
heading
dccorator
caption
other
macro
micro
Decision Tree
96.62
82.28
93.24
53.79
58.45
50.34
73.59
76.79
89.34
CanvasEmb (Base)
95.65
91.76
95.94
34.55
71.15
71.48
82.41
91.77
w/o pretrained
95.88
90.62
95.76
39.79
97.44
64.91
72.73
81.95
91.99
CanvasEmb (Large)
98.74
88. 88.28
94.18
55.15
98.66
78.25
78.73
85.45
95.97
w/o pretrained
95.18
88.07
92.37
30.93
97.29
70.06
65.59
79.93
90.87
30% labeled data
Decision Tree Can (Base)
97.06
71.44
66.67
0.00
63.03
19.75
36.14
54.54
86.62
97.69
89.78
88.20
60.56
95,40
75.80
29.18
80.51
93.16
w/o pretrained
93.67
86.61
82.83
48.44
95.71
58,03
19.16
74.77
86.88
CanvasEmb (Large)
98.98
87.46
88.20
61.59
95.40
75.80
29.18
82.52
95.42
w/o pretrained
94.08
88.18
92.51
51.73
95.89
73.08
15.89
78.55
88.18
Element Role Labeling. We report the Fl score for each role and the overall macro/micro Fl metrics in Table On the one hand, all CanvasEmb methods outperform traditional Decision Tree model with large margin on most of the metrics. On the other hand, CanvasEmb (Large) achieves better results than CanvasEmb (Base), and pre-training further boosts the performances.
Image Captioning. We report F1 and AUC scores in Table Similarly, CanvasEmb (Large) with pre-trained achieves the best results. Since this task focuses on the relation between elements, we also train enhanced baseline (+2D features) where we explicitly incorporate additional hand- crafted 2-dimensional features, such as the distance between two elements. We observe that Can- vasEmb (Large) outperforms the enhanced baseline with large margin. This indicates the ability of Emb to capture the relations between elements, which is much more efficient than the heuristic way of manual feature engineering.

---

Under review as a conference paper at ICLR 2021
100% % labeled data
F1
AUC
Table 2: Results for image captioning.
Decision Tree
84,30
98.32
2D features
84.50
98.21
CanvasEmb (Base)
88.99
98.23
w/o pretrained
89.28
98.33
CanvasEmb (Large)
90,95
98.51
-wo pretrained
89.68
98.32
Table Results for auto completion. We
show accuracy for Shape Type (ST.) and Fill Color (FC.), loU for Position (P.).
30% labeled data
Decision Tree
76.39
96.42
+ 2D features
81.34
97.36
Model
ST.
FC.
CanvasEmb (Base)
78.37
94.26
w/o pretrained
75.69
93.69
CanvasEmb (Base)
87.38
69.49
44.01
CanvasEmb (Large)
80.64
95.27
CanvasEmb (Large)
89.37
75.63
46.40
w/o pretrained
79.43
94.85
Let's contac
Agency's Mission
REACH
comsnreers of yea lestate ssance, on diaaed derelippment sary beterced
onniemmen heaithy nec estece market
(a)
(c)
Figure 4: The element-to co-etttto-leeettt relations. The first row shows the original slides with element index, and the second shows the corresponding headmaps which X- and Y-axis the element indexes. (a) Tre-Trained Model (b) Elemeet-Rolee-.abelingg Model (c) Image-Captioning Model.
5.4 EFFECT OF PRE-TRAINING
Correlation between pre-training and model size. In general, CanvasEmb models with pre- trained outperform those without pre-trained on both tasks. And the increase of model size in the large model (L=66, H=128) further boosts the performance compared the base model (L=3, H=64). For example, in the task of element role labeling, the pre-trained model brings 5.52% macro Fl gain for CanvasEmb (Large) while only 0.46% gain for CanvasEmb (Base). This is accordant with our intuition, í.e. larger model can capture and embody more knowledge from the pre-training.
Correlation between pre-training and labeled fine-tuning data size. We also conduct ablation study with only 30% labeled training data for both downstream tasks, shown in Table [ and respectively. We observe that pre-training contributes more when labeled data size is small. In ele- ment role labeling task, there is A 5.47% macro F1 gain from pre-training for CanvasEmb (Base) under the setting of 30% training data, compared to only 0.46% gain using full data. Similarly, in the task of image captioning, there is A 2.68% Fl increase for CanvasEmb (Base) from pre-training using 30% data while pre-training only works a little bit for the base model when using full data. This shows that our pre-traine pre-trained models are robust and well aligned to the downstream tasks, which alleviates the data scarcity issue.
5.5 PROBING INTO SHAPE CONTEXT
To investigate how Emb captures element-to-ellmmet relations, we use the impact func- tion (Wu et al. 2020) fimpact (X; x,) dist(h; (x;)), hX) (x,, x;))) show the importance

---

Under review as a conference paper at ICLR 2021
Course Couctivv
seppling
Paaill
Find Doextioon tearch Pestires
(a)
Figure 5: (a) A query layout and (b)(c) top 2 similar candidates retrieved by CanvasEmb (Large).
of the elements by calculating the distance between the hidden states of the layout when removing elements (x:) and (x, x The impact matrix heatmaps displayed in Figure 4
Pre- Tre-Training. From Figure 4a), we observe that elements in close region tend to have stronger influences each other (shown by darker-colored pixels). For example, element and 4 on the bot- tom right of the slide have significantly strong impacts on each other, but weaker impacts on element 3 the left. This indicates that the pre-trained model can typically capture location information in layout representation.
Element Role Labeling. Figure 4 bb) visualizes the model fine-tuned the task of element role labeling. We can see elements 1, 3. 5 generally have the most significant impacts on all other elements in the slide. Specifically, element 3 and 5, which are symmetric and have the same semantic roles, strongly affect each other. This shows that the model learns representations mainly based on the content-related features and location of elements in the slide.
Image Captioning. For the image captioning task in Figure the fine-tuned model pays more attention to images and the corresponding caption-like elements. Element pairs (1,3} and {2, 4) have very strong impacts each other, while element O has the weakest influences in the slide. This
is an interesting observation to see how caption pairs in a slide interact with each other, as the model in this case obviously separates the element 0 from others with captioning relation.
5.6 EXTENDED APPLICATIONS OF PRE-TRAINED CANVASEMB
To further demonstrate the knowledge embodied in our pre-trained CanvasEmb, we show two ex- tended applications.
Layout Auto Completion. Given a partial layout, the model needs to complete the layout by pre- dicting the remaining elements with their properties (Li et al.) (2020b). We constrain the setting to one remaining element, which aligned to our pre-training objective. From the result shown in Table 3 we see our pre-trained model without any fine-tuning achieves promising results with respect to the properties of shape type, fill color and position. This shows the accurate modeling of elements and contextual representations of layouts in the pre-trained CanvasEmb.
Layout Retrieval. Given query layout, the scenario is to retrieve similar layouts from the database, which can be used for layout recommendation. Figure shows an example retrieved by CanvasEmb (Large) with layout embedding cosine similarity. As we can see, both the query and the two candi- dates contain similarly four images with captions. For a quick evaluation, we construct 20 queries and manually score model's 's retrieval quality (rate similarity score from 1 to 5). The average score of top 3 candidates are 3.86 (> above average), which demonstrates CanvasEmb effective layout representations.
CONCLUSION
In this paper, we present CanvasEmb, a large-scaled pre-trained model for layout representation in graphic design. It encodes elements in layout with the multi-dimennional properties and models the element context with a Transformer-based architecture. To pre-train our model CanvasEmb, we adopt the masked language modeling with multi-task learning objective. the two proposed tasks related to layout understanding, fine-tuned Cmb achieves state-of-the-art performances. Fur- thermore, we conduct a deep analysis to prob into the embodied knowledge learned by CanvasEmb, and show its great potential for more applications in graphic design. As future work, going to apply our model to more downstream tasks, and verify our model on more document types such web pages word documents.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joost Beusekom, Daniel Keysers, Faisal Shafait, and Thomas Breuel. Distance measures for layout- based document image retrieval. In Proceedings of the Second International Conference on Doc- ument Image Analysis for Libraries, 2006.
Niranjan Damera- Venkkta, José Bento, and Eamonn O'Brien-Strain Probabilistic document model for automated document composition. In Proceedings of the I Ith ACM Symposium on Document Engineering, DocEng '11, pp. 3-12, New York, NY, USA, 2011. Association for Computing Machinery.
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171-4188 Association for Computational Linguistics, 2019.
Nathan Hurst, Wilmot Li, and Kim Marriott. Review of automatic document formatting. In Pro- ceedings of the 9th ACM Symposium on Document Engineering, DocEng '09, pp. 99-108, New York, NY. USA, 2009. Association for Computing Machinery.
Guolin Ke, Qi Meng, Thomas Finely, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems 30 (NIP 2017), 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA. May 7-9, 2015, Conference Track Proceedings, 2015.
Hsin-Ying Lee, Weilong Yang, Lu Jiang, Madison Le, Irfan Essa, Haifeng Gong, and Ming-Hsuan Yang. Neural design network: Graphic layout generation with constraints. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In The Phirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Shirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. AAAI Press, 2020a.
Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. Layoutgan: Generating graphic layouts with wireframe discriminators, 2019.
Yang Li, J. Amelot, X. Zhou, S. Bengio, and S. Si. Auto completion of user interface layout design using transformer- based tree decoders. ArXiv, 0iix/2001.05300 2020b.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. 2017 IEEE International Conference on Computer Vision (ICCV), 2017.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and- lan-auuugge tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'lchéBBus Buc, Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NewrIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 13-23, 2019.
Peter O' O'Donovan, Aseem Agarwala, and Aaron Hertzmann. Learning Layouts for Single-Page Graphic Designs. IEEE Transactions on Visualization and Computer Graphics, pp. 1200-1213, 2014.
Xufang Pang. Ying Cao, Rynson W. H. Lau, and Antoni B. Chan. Directing user attention via visual flow web designs. ACM Trans. Graph., 2016.
9

---

Under review as a conference paper at ICLR 2021
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks, 2012.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018.
A. Radford. Improving language understanding by generative pre-training. 2018.
A. Stoffel, David Spretke, Henrik Kinnemann, and D. Keim. Enhancing document structure analysis using visual analytics. In Proceedings of the 2010 ACM Symposium Applied Computing, 2010.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
S. Tabata, Hiroki Yoshihara, H. Maeda, and Kei Yokoyama. Automatic layout generation for graph- ical design magazines. ACM SIGGRAPH 2019 Posters, 2019.
Kashyap Todi, Daryl Weir, and Antti Oulasvirta. Sketchplore: Sketch and explore with layout optimiser. In DIS 2016 Proceedings of the 2016 ACM Conference on Designing Interactive Systems: Fuse, pp. 543-555 ACM, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
Wenhui Wang. Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 189-198, Vancouver, Canada, 2017. Association for Computational Linguistics.
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. Perturbed masking: Parameter-fre free probing for analyzing and interpreting bert. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Ying Cao Xinru Zheng, Xiaotian Qiao and Rynson W.H. Lau. Content-awan generative modeling of graphic design layouts. ACM Transactions on Graphics (Proc. of SIGGRAPH 2019), 2019.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre- training of text and layout for document image understanding. 2019.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 5754-5764, 2019.
10


Reviews:
[{"review_id": "ooxIeD-GI5o", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applies state-of-the-art transformer-based neural networks to layout representation learning of slides. The most notable contribution of this paper is the construction of large-scale parsed slide layout dataset. This paper proposes to pre-train the network on this large-scale dataset without masked reconstruction strategy and verifies it with several subtasks including element role labeling, image captioning, auto-completion and layout retrieval, with a comparison to a decision-tree based method as baseline. \n\n+Most of previous layout learning works only show experimental results on small labeled datasets (a few thousands), partially due to the scarcity nature of layout data. This paper looks at slide layout data and constructs a large-scale (>1m) dataset with parsed element properties. \n+The chosen network design and training strategies all make sense. \n\n-It is pity that this paper didn’t disclose sufficient details of how the large-scale dataset was constructed and of data statistics, e.g. how many elements in each slide, templates, completeness of properties, etc. How are the properties parsed, fully automatic? Is the role labeling dataset part of the pretraining dataset?\n-Pretraining. The proposed evaluation tasks all seem to be sub-tasks of pre-training and it doesn’t look falling into the classic scheme of unsupervised pretraining + supervised fine-tuning. Dataset differentiation is another issue. For example, in the role labeling experiment, is this targeted dataset a subset of the large-scale one? And is the only difference that the training loss? \n-Evaluation. Evaluating graphic layout can be a hard problem and this paper tried to propose several small tasks as probes into the learned network. However, it will be more convincing to have a systematic design of experiments. First of all, in addition to type properties, how about geometric property and color property prediction? Second, any experiments would benefit from both quantitative and qualitative results. Especially for layout design, visualization is very important. \n-Layout retrieval is an interesting experiment, but manual scoring seems to be arbitrary. \n-Baselines. Neural design network by Lee et al. in ECCV2020 and LayoutGAN by Li et al. in ICLR 2019 seem to be good baseline network architectures to compare, although they are trained in different ways. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8pHhf6e214", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to learn representations for graphic design layouts unsupervised using a Masked Language Model objective as in BERT with additional domain specific input embeddings. It also presents a large dataset of slides along with manual annotations used for their evaluation. \n\nThe idea of using BERT like training to pre-train features for graphic design layouts is sound and the proposed approach modifies the input token representation to represent different layout elements -- using positional embeddings for continuous attributes (position, rotation etc.) and learned embeddings for categorical attributes (element type etc.). Since there is no positional embedding for position in the sequence, the ordering in the sequence does not matter for training, which is a huge benefit over training sequential models in this context, where ordering can be arbitrary. \n\nI believe the evaluation of the method presented in the paper is lacking and could be improved significantly. Moreover, it would be useful to make it clear if the authors plan to release the collected slides dataset since that would count as a positive contribution in my books. More training details (how many/what GPUs, how much time/epochs, any learning rate tricks, optimizer etc.) would also help with the reproducibility aspect of the paper. \n\nW.r.t evaluation, the authors propose to do the tasks of element type classification (called role labeling in the paper) and image-caption pair detection (i.e. a binary classifier). On both of these tasks, the only baseline presented is a gradient boosted decision tree, which performs quite well on both tasks already. This begs the question, are all these parameters used to learn CanvasEmb needed? In my head there are the following ways to answer this question:\n1. Are there other higher complexity tasks (where the decision tree baseline would fail a lot) that are important for graphic design layout intelligence? \n2. Are there other lower complexity (parameter wise) methods that also take 2D layout into consideration that you could train for these two tasks? For example, how would a simple convnet trained to classify UI elements perform? You could imagine a variety of ways of showing the convnet which element to classify, one example would be passing its bounding box mask as a separate input channel. How does CanvasEmb perform as compared to them?\n\nThe other evaluation presented is on layout retrieval and layout auto completion. Layout auto completion in this case is a misnomer (when related to previous work), since the authors only mask 1 attribute of 1 element at a time and compute the accuracy of predicting that attribute on test set examples. This differs strongly from layout auto completion in other work where complete elements along with their properties are added. Similarly for layout retrieval, a quick ad-hoc rating study is presented. \n\nIt is hard to position this work w.r.t. previous work given how the evaluation is performed. It would be great to evaluate this method in the context of existing work to really understand the importance of doing this sort of pre-training. Doing higher complexity tasks as mentioned above would also be very useful. For example, can you use CanvasEmb to offer layout suggestions (as in DesignScape, CHI 2015 as an example)? You can imagine a Gibbs Sampling like procedure where you repeatedly randomly erase a part of the parameters of the layout elements (i.e. their position, rotation, color etc.) and repredict them for some steps. Would this be able to generate plausible layout suggestions? \n\nOverall, I believe this is an exciting direction and I'm looking forward to hear how the authors think their paper could be evaluated on harder tasks, against stronger baselines and against existing work. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "4Ppf_MqM7OK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the work: \nThis work presents a Transformer-based framework to learn Graphical Layout representation/embedding, taking inspiration from transformer-based models in the area of NLP.  To train their model, named CANVASEMB, the paper contributes a dataset of Powerpoint Slides, with more than 1 million slides (the paper promises to make the dataset public). When performing new tasks on graphical layouts, CANVASEMB can be used as a pre-trained model. The paper also demonstrates that CANVASEMB achieves SOTA performance on two downstream tasks, viz., element role labeling, and the so-called \"image captioning\" in the context of a layout image and its text tag.\n\nOriginality:\nThe task being attempted (Learning Layout embeddings) is not new. However, the paper makes use of a Transformer-based model for Layout embedding, which is different and was not explored/employed before. However, the idea of employing Transformers is not well motivated;  the method section is technically sparse, in the sense that not much of the Transformer model is presented (at least using a Figure, if not via Mathematical formulation). It would help the reader if details on the Transformer model and the Attention scheme were presented to some extent.\n\nSignificance:\nLayout Representation and Learning is an import problem and is gaining significance. That said, any method should evaluate thoroughly to demonstrate its merits and advantages over prior works, for potential impact. This paper lacks a thorough evaluation. More on this later. \n\nReferences: \nPenultimate line in the first paragraph of RW section: It should be \"Cao et. al 2019\", not \"Xinru Zheng and Lau (2019)\". The first author naming convention cannot be superseded at the writer's discretion. Only include the second and last author in the reference? Has been cited again after this, in the same way. I will assume that this was an oversight. Needs to be corrected.\nAlso, the paper is missing the following reference, which, similar to Li 2020b, uses a Tree-based representation of Document Layouts.\n1) READ: Recursive Autoencoders for Document Layout Generation, CVPR 2020\n\n\nQuality:\n1) The paper claims that CanvasEmb is the first work to provide pre-trained models for layouts in graphic design. I think the \"Content-Aware Generative Modeling of Graphic Design Layouts\" from SIGGRAPH 2019 is the first work to do so, although on a much smaller scale (~4K magazine designs as against 1M used in this paper).\n2) The paper does not contrast how it is different from the above work (Cao et al., SIGGRAPH 2019). What are the key differences? What is special about CANVASEMB, that can not be done by the 2019 work when given such a large dataset? Is there a key technical bottleneck that is overcome in this work? I would first like to know the maximum number of layout elements in a Slide in the entire dataset.\n3) The \"Related Work\" section could be better. Currently, there seems to be a lack of some sort of explanation on what differentiates this work from the prior works. In other words, there is a frequent and incoherent jump from reference-to-reference in the context of this paper. The desire for a reader is to understand the difference in contributions of this work as compared to the prior works in the related work section.\n4) \"Content-Related Properties\" are already taken into account when the semantic box geometries are considered. So, the text font-size and font-type will not have an additional effect on the layout, but only on the perceptual quality of a content-filled layout. So, I don't think this counts as one of the layout properties when you have the box geometries already accounted for.\n5) The so-called \"image captioning\" task is nothing but a binary classification of an image-text tag pair. \n6) Comparisons against the SIGGRAPH 2019 paper (Cao et al, 2019), Li 2020b, and READ (CVPR 2020) should be presented.  Doing such an evaluation on the same datasets as CANVASEMB will demonstrate the strength and weaknesses of the Transformer-based architecture.\n7) In addition to these evaluations, I would like to see why the SOTA Graph Neural Networks can not be employed on this task when they can capture rich structural properties of a layout and require less computation compared to Transformers. This is a very important and interesting experiment, which should be performed.  Moreover, training a Graph Neural Network does not require humungous data. This weakens the motivation of the approach presented in this paper.\n\nClarity:\n1) The writing is good. As I said earlier, the RW section could (and should) be improved.\n2) I would suggest presenting an image that contains the components of the Transformer. The reader should not have to refer to the cited papers to know/learn what the Transformer is made up of.\n3) I think the word \"pre-trains\" in the paper means to say that \"CanvasEmb\" can be used as a pre-trained model for other tasks. CanvasEmb, in the due course of its development, is not \"pre-trained\" to begin with. So, I think the sentence in the abstract, the introduction, and parts of the evaluation involving the word \"pre-trains\" should be rephrased to remove confusion and for an easier understanding of the paper. Ex: A Transformer is first trained using the Slides crawled from the internet, which we term as CanvasEmb, and can be used as a pre-trained model for other downstream tasks.\n4) \"mechanism\" typo in the last paragraph of introduction (it appears twice continuously)\n5) In Equation 5, the tasks for each loss formulation should be written (in the equation itself)\n6) I would like to know the maximum number of layout elements in a Slide in the entire dataset. For ex: if there are 3 Text Boxes and 2 Figures in a Slide, then the total number of layout elements is 5. \nThere has been no mention of this in the entire paper, but this is crucial to understand if there is really a need for Transformer-based architecture.\n\nPros:\n1) Different, interesting approach to Layout Embedding\n2) Contributes a huge dataset of Powerpoint Slides (1M) \n3) Simple and easy writing\n \n\nCons:\n1) Some important dataset-related stats are missing (as pointed above). They are essential for understanding the richness of the dataset and to make an informed decision on what kind of deep-learning models should be employed when using such a dataset for different tasks.\n2) Weak evaluation\n3) Comparison to Cao et al., SIGGRAPH 2019, Li et al 2020b, and READ (CVPRW 2020) are missing. The last two are tree-based generative models of Document layouts, and should be tested for performance on the same dataset used to train CANVASEMB.\n4) Comparison to SOTA Graph Neural Networks: Missing explanation for a motivation to use Transformer-based architecture, when GNNs can capture rich structural properties of a layout and require less computation compared to Transformers. This is a very important and interesting experiment, which should be performed.  Moreover, training a Graph Neural Network does not require humungous data. This weakens the motivation of the approach presented in this paper.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to learn layout representations for graphic design using transformers with a masking approaching inspired by BERT.  The proposed model is pretrained on a large-collection of ~1M slides (the script for crawling the slides will be open-sourced) and evaluated in several downstream tasks. 

Review Summary:
The submission received slightly negative with scores of 4 (R3) and 5 (R2,R4).  
Reviewers found the paper to be well-written and clear, and the problem of layout embedding to be interesting.  Reviewers agree that the use of transformers for layout embedding has not been explored in prior work.  However, the paper did not have proper citation and comparisons against prior work for layout embedding, and lacked systematic evaluation.  Reviewers also would like to know more details about the dataset that was used for pre-training.  

Pros:
- Novel use of transformers for layout embedding (not yet explored in prior work)
- Use of large dataset of slides

Cons:
- Lacked proper citation and comparisons against prior work for layout embedding
- Lacked systematic evaluation
- Missing details about the dataset

Reviewer Discussion:
During the author response period, the authors responded to the reviews indicating that they will improve the draft based on the feedback, but did not submit a revised draft. As there was no revision to the submission, there was limited discussion with the reviewers keeping with their original scores.  All reviewers agrees that the direction is interesting but the current submission should not be accepted.

Recommendation:
The AC agrees with the reviewers that the current version is not ready for acceptance at ICLR, and it would be exciting to see the improved version.  We hope the authors will continue to improve their work based on the reviewer feedback and that they will submit an improved version to an appropriate venue. 

Decision: Reject