Auto Completion of User Interface Layout Design Using Transformer-Based Tree Decoders
Authors: Yang Li
Julien Amelot
Xin Zhou
Samy Bengio
Si Si
Venue:  (2020)
Keywords: Transformer
decoder
user interface
layout design

Abstract: It has been of increasing interest in the field to develop automatic machineries to facilitate the design process. In this paper, we focus on assisting graphical user interface (UI) layout design, a crucial task in app development. Given a partial layout, which a designer has entered, our model learns to complete the layout by predicting the remaining UI elements with a correct position and dimension as well as the hierarchical structures. Such automation will significantly ease the effort of UI designers and developers. While we focus on interface layout prediction, our model can be generally applicable for other layout prediction problems that involve tree structures and 2-dimensional placements. Particularly, we design two versions of Transformer-based tree decoders: Pointer and Recursive Transformer, and experiment with these models on a public dataset. We also propose several metrics for measuring the accuracy of tree prediction and ground these metrics in the domain of user experience. These contribute a new task and methods to deep learning research.

For auto-completion of words, typical choice of solution is using a language model that is trained to predict next word W++1 given previous words w/+1 = arg max P(wx+1 Wop ...; wy). As generative model, language model can essentially complete sentence by predicting the rest words in an Under review as a conference paper at ICLR 2020
AUTO COMPLETION OF USER INTERFACE LAYOUT DE- SIGN USING TRANSFORMER-BASED TREE DECODERS
Anonymous authors Paper under double-blind review
ABSTRACT
It has been of increasing interest in the field to develop automatic machineries to facilitate the design process. In this paper, we focus assisting graphical user interface (UI) layout design, a crucial task in app development. Given a partial layout, which designer has entered, our model learns to complete the layout by predicting the remaining UI elements with a correct position and dimension as well as the hierarchical structures. Such automation will significantly ease the effort of UI designers and developers. While we focus on interface layout prediction, our model can be generally applicable for other layout prediction problems that involve tree structures and 2-dimensional placements. Particularly, we design two tree decoders: Pointer and Recursive Transformer,
versions of Transformer-bast and experiment with these models on a public dataset. We also propose several metries for measuring the accuracy tree prediction and ground these metries the domain of user experience. These contribute new task and methods to deep learning research.
INTRODUCTION
Layout design is a universal task in many domains, ranging from mechanical design to graphical layouts. There has been a long tradition in developing computer-aided design (CAD) tools both academia and industry Hurst et al. 2009). Recently, there has been increasing interest in developing deep learning models for design generation (Zheng et al./ 2019 Ha and Eck 2017: Li et al., 2019 Isola al. 2016). In this paper, we focus on models that can automate graphical user interface (GUI) layout design, a central task in modern software development such as smartphone apps.
As designer or app developer creates an interface design, model recommends UI elements based on the partial design that has been entered. The automation can significantly case the effort of designers and developers for creating interfaces, because it not only reduces the input required but also brings design knowledge to the process. This is analogous to auto completion that is widely available in text editing tools, although layout completion
much more complicated problem. The task is appealing to the deep learning field for number of reasons. First, it is non-linear and involves 2D placement of elements, instead of the sequential next word prediction for text completion. Second, the problem is highly structural that takes structural input partial tree and generates structural output- completed tree.
auto-regressive fashion. Intuitively, we need a layout model-a layout decoder-thal predicts the rest elements and structures needed to complete a given partial tree.
Previously, models for tree structure generation have been proposed for a number of problems such as language syntax trees (Vinyals al. 2015b) and program generation (Chen et al. 2018 Dai et al. 2018). In these problems, the model first encodes the source input via an encoder then generates tree structures in target domain via decoding. For layout auto completion, involving the decoding process, although we can potentially include an encoder to bring in additional information. Additionally, our problem involves unique aspects of 2D placements.
focus on solely
hhtps://alssamig.com https//wwwwssketch.or

---

Under review as a conference paper at ICLR 2020
We design our layout decoder model based on Transformer, model that has gained popularity on a number of tasks (Vaswani et al., (017). The attention- based nature of Transformer allows us to easily model structures, similar to Pointer Networks (Vinyals et al., 2013a), and represent 2D spatial relationships. In particular, we examine two versions of Transformer-based tree decoder: Pointer Transformer and Recursive Transformer for this task.
Layout completion, as a structural prediction problem, lacks evaluation metrics. We design three sets of metrics to measure the quality of layout prediction based on the literature and the domain specifies of user interface interaction. We experimented with these models on a public dataset of over 50K Android user interface layouts. The experiments indicate that these models can bring values to a user interface layout design process.
2 RELATED WORK
Recently, there have been increasing efforts in using deep learning to enhance design practice with Generative Adversarial Networks (GANs) (Goodfellow al. 2014), Variational Auto Encoders (Kingma and Welling 2014), and Autoregressive models (Reed et al. (017) as three major underlying approaches. There is rich body of work based Generative Adversarial Networks (GANs) (Goodfellow et al. 2014 Isola et al. 2016 Zhu et al. (017) where a discriminator is often introduced during training for distinguishing synthesized and real designs, which in the end the model learns to generate realistic designs. In particular, LayoutGAN (Li et al., 2019) is the most related to our work in the literature, which takes a collection of randomly parameterized (e.g., positioned and sized) elements via an encoder and generates the well-arranged elements via a generator. Although we also generate 2D layouts, our problem is substantially different because takes in a tree-structured 2D partial layout and outputs tree-structured 2D full layout. Our model does not see all the elements as LayoutGAN does and our input and output involves a tree structure.
Our approach is related to SketchRNN (Ha and Eck 2017), model that can generate sketch drawings. SketchRNN takes a latent representation that is learned via Variational Inference (Kingmal and Welling 2014) and generate or complete drawing using an autoregressive decoder (Reed et al. 2017p. Similar to SketchRNN, our decoder is also an autoregressive model. The unique challenge in our case is the structure representation and generation. We can easily extend our models take varitional input as SketchRNN, although we focus on tree 2D decoding in this paper and leave the varitional input to future work. pix2code is another work that is related our effort, which generates UI specifications from a screenshot pixels (Beltramelli) 2017). It uses CNN to encode UI screen and generate UI specification with an LSTM autoregressive decoder. In this work, a tree structure is handled as flattened sequence such that the decoding can be handled in the same way as language sentences, which similar treatment is conducted for syntax parse tree prediction (Vinyals et al.. 2015b). Zhu et al. took one step further to propose attention-basel hierarchical decoder (Zhu et 2018) where two LSTMs are used: one for deciding the blocks that the program needs and the other generating tokens within block. Our recursive Transformer is applied hierarchically well although we use the same model repetitively.
While our target domain is user interface layouts, our problem is fundamentally related to structure prediction. Previously, much efforts have been devoted to program generation (Chen et al.) 2018 Dai et al. 2018 Si et al. 2019). Jenatton et al. (Jenatton et al. 2017) proposed approach to predict tree structures by using Bayesian optimization to combine independent Gaussian Processes with a linear model that encodes tree-based structure. An important strategy that was explored previously is to extend inherently sequential recurrent models such as LSTM to the hierarchical situation (Tai cet al., 2015), which can handle a range of tree structures. Particularly. Chen et al.'s decoder generates a binary tree by applying LSTM recursively (Chen et al., 2018). with one LSTM for generating the left child and the other for generating the right child. Dong and Lapata applied LSTT a recursive fashion to generate logic forms from language input (Dong and Lapata| (016). Based on the previous work, we design our recursive tree decoder based on Transformer (Vaswani et al.) 2017) to handle arbitrary trees. Rather than carrying the hidden state and cell values as recurrent nets, our model can access all the nodes on the ancestry path via attention. Our positional encoding of 2D coordinates is similar to Image Transformer (Parmar et al.) 2018). It is possible to employ tree positional encoding as proposed in (Shiv and Quirk] [uir9]2019) when we keep the hidden states of all the previously generated nodes. Our current design for recursive Transformer decoder is to make the ancestry hidden states
2

---

Under review as a conference paper at ICLR 2020
accessible for decoding child nodes. Another version of the tree decoder experiment with in the paper is designed based on Pointer Networks (Vinyals et al.) 2013a) where each node points to its parent based on the dot product similarity of their hidden states, which is simple extension based on Transformer.
Finally, the problem focus is originated from the domain of human computer interaction (HCI) and graphical layout design. A rich body of works have been produced previously (Hurst et al.) 2009).
The idea of UI auto completion has been previously envisioned (Li and Chang 2011). DesignScape is a full-fledged tool that provides both refinement suggestions and brainstorming suggestions during graphical layout design. Recently, Liu et al. mined rich set of Android apps that produced over 66K UI screens with semantically labeled elements (Liu et al. 2018), including their types, bounding boxes and the hierarchical tree structures. This dataset lays the foundation for investigating the problem of structured graphical layout prediction, which constitutes the dataset for our work.
Our work also nourished by the rich literature in the HCI domain on human performance modeling. One important aspect that is significantly underexplored for tree prediction is the accuracy metrics. We design a tree-edit distance metric based on keystroke level GOMS models (Card et al. 1980) that estimates how much effort it needs for a designer to transform the predicted layout tree into the ground ground-truth tree, which indicates the usefulness a predicted layout tree. Along few other metrics, these define
problem that deep learning models can further tackle.
UI LAYOUT COMPLETION PROBLEM
Here, we consider the problem of completing a partial tree, which represents the graphical layout that has been entered by the designer so far (see Figure 1). A design process starts with an empty design canvas, which corresponds to a partial tree with only the root node. Each node has several properties, including its type, e.g., a button or list, whether the node is terminal, and the rectangular bounds of the node. As the designer adds more elements to the canvas, the tree grows. Node a parent of node B if A is non-terminal node and A contains B. The problem is extremely challenging due to the large space of possibilities.
Figure 1: A schematic illustration of UI layout completion problems. The elements with solid bounding boxes are already added by the designer, and those with dashed bounding boxes are predicted to complete the layout given the solid elements-the partial layout. The tree on the right shows the UI structure with the line colors and styles corresponding the UI elements on the left. The figure is best viewed in color.
3.1 PROBLEM DEFINITION
A graphical layout tree, G. contains a collection of |G] nodes where its i-th node carries both spatial and semantic properties: (c,, dj. Pt, 1 < < |G| and |&] is the number of nodes in the tree. e; is a categorical value that indicates the node type, G types. d, is integer that represents the depth of the node in the tree. t; is binary value that represents whether the node is terminal node. p; is the index position the node i's parent node, 1 < P; < |G]. by is a tuple that represents the bounding box of the node, including its top-left and right-bottom corners: by (#t, Vi. Li, 9:).
where C' is the set possible element
3

---

Under review as a conference paper at ICLR 2020
A partial tree, P, contains a collection of k nodes (k < |G)). It is required that the root node must be in P. In addition, the parent of each node in P, except the root node, should also be in P, i.e., Py E P PPifj j P. such that the parent index, Py. is always valid and addresses an existing node in the partial tree. This requirement matches how design tool is typically implemented where each UI element (a node) added either the design canvas (the root node) or a container node that is already attached to the layout tree.
Given the partial tree P. the task here is to predict the full tree G' based on the partial tree: G' = F(P). We elaborate on how the function F can realized with variety of Transformer decoders in the next section.
4 TRANSFORMER-BASEE TREE DECODERS
Previous work for tree prediction is primarily based on LSTM, recurrent neural network that is amenable to handle data with an arbitrary length. In this work, we design our tree decoder models based on Transformer (Vaswani et al. 2017), an architecture that has shown advantages on number of tasks. Particularly, the positional embedding that Transformer's attentional mechanism is based on be easily extended to represent 2D spatial relationships. We here discuss three versions of the 2D layout tree decoder model: Vanilla, Pointer and Recursive Transformer.
4.1 REPRESENTING LAYOUT NODES & DECODING STEPS
Before diving into each decoder model, we first discuss the representation of each node in the layout tree, which is shared by all three decoder models. We use the general concept of embedding to represent each node (Bengio al. 2003). More specifically, for node i, we first embed each property of that node as the following. For the type property, ci, and the terminal property, 4, we embed these categorical values as Equation] and
& = L(c))
(1)
e L(4)
(2)
where ef and et are embedding for e; and I; respectively; 1-)) is one-hot vector and E E R and F E RI [E are the embedding matrix, where [V.| and [V] are the vocabulary size of each property, and [E] is the embedding dimension. For each coordinate value in b, we treat them as discrete values and represent them through similar embedding process. See Equation for embedding a coordinate æ. A similar treatment is done previously for pixel coordinates an image (Parmar et al. 2018).
e: 1x;)
where E VV and [Val the number of possible positions. Similarly, we embed the s. e, We concatenate these coordinate embeddings
four coordinates in b,, F4, Mr, L.. y)s as e , to form the bounding box embedding (see Equation 4) which form the positional embedding for using Transformer:
=
(4)
We then combine these embeddings of node properties to form the final embedding for i-th node using Equation 5
c + ef + ef.
With each node represented as an embedding vector, we now discuss how the Transformer decoder model (Vaswani al. 2017) is used to represent each decoding step. The Transformer decoder model
4

---

Under review as a conference paper at ICLR 2020
is multi-layer, multihead attention- based architecture. It computes, hi, the hidden state of step i layer by attending to the hidden states of all the steps so far the previous layer (See Equation 9 For an I.-layer Transformer, S I,, and he = ez- h R! where H is the dimension of the hidden state.
i Atteetioon(((M) K V
(6)
We use Attention to represent the multihead attention operation Transformer. Q(-) K()). and (-)) feedforward nets to compute queries, keys and values for attention computation (Vaswani et al. 2017
4.2 VANILLA TRANSFORMER DECODER
For the vanilla version of Transformer Tree Decoder, similar to previous work for predicting language parsing trees (Vinyals et al. 2015b), we linearize tree, based on the preorder depth-firse first traversal, as a sequence of tokens. The process adds the opening "(" and closing ")" tokens the children of each parent except the case when the parent has single child, as illustrated in Figure in (Vinyals) al., 2013b). Note that this representation requires the partial tree to prefix of the depth-first traversal. This data representation transforms tree prediction sequence prediction problem.
We can then define the distribution over a sequence of m decoded tree nodes, n2, k +++1 S S + m, given sequence of k nodes from a given partial tree P: n2. 1 S S k. as the following (see Equation
k++m
P( a Tk+2 u. Bi Mi f22 nx)
II i=++1
P(n (1, Bon Uky Rk+ 1.... 7i-1)
k+m
II
P(c, ti, 24, 24, Lr. M. 12, The Mk+1: 11-1)
=
4 k++1
II
II
softmax (W2mxx
k=++1
1 ZE (1..t.PP..
where n4 (ci, li Ti, Vi. Li, ú.). W. RIV-Ix is the output embedding weights for each property of node where | is the vocabulary size for the property and H is the embedding dimension. The model does not directly predict pę. Instead, p: is acquired by reconstructing the tree from brackets. Note that the opening and closing bracket are considered two special nodes that need to be embedded and predicted well. For these two nodes, only c; matters while other node properties irrelevant.
4.3 POINTER TRANSFORMER DECODER
Instead of introducing the beginning and closing tokens to represent tree hierarchies and being limited to only the depth-fire traversal order, Pointer Networks (Vinyals et al. (015a) provide natural way to represent child child-parent relationship. We define n. = (c,, t,, 2is Yir &., di P.) that includes the index position of the node's parent, Pt. We then extend Equation 7 to predict p: the following (see Equation
P( ".. 7k mi,,... n
II 2-++1
Plc., 24, Mi, E4, Di. Pu[ni,... ny, ng+1s ... T4-1)
(8)
k+m
II
[softmax (H (1
II
softmax (W.h((
1=
5

---

Under review as a conference paper at ICLR 2020
where H': R( is the embedding weights with jth row be h S i-1. softmax softmax(H,gh1) h+ computes softmax over the dot product alignments between the hidden state of the ith node and that of each previous node.
Layer
Layer
Layer 3
Figure 2: A schematic illustration of Recursive Transformer for decoding the tree on the left. The same Transformer model is applied decoding the children for each parent, top-down from the root. The hidden states of ancestry nodes are used for computing attention in downstream layers. The gray boxes represent Transformer layers and the dashed lines denote attention dependencies. The dashed nodes are predicted.
4.4 RECURSIVE TRANSFORMER DECODER
The third version of decoder models that we investigate here applies Transformer decoder model in recursive manner by using the same model to decode the children of each parent node (see Figure 2). To do so, we first compute the decoder self attention in a similar way to Equation 9 except that we only use the parent and sibling nodes for computing attention keys and values. it represents the hidden state after taking the sth sibling node.
T (itenntio(Q(h))))) ((( ))
(9)
To leverage the information beyond the parent node and siblings, we involve the ancestry nodes in the attention computation by attending their hidden states.
h (itemtion(Q(i)) v(H)))
(10)
where A donates the set of ancestry nodes, and H are the set of hidden states of the ancestry nodes, top-down manner starting
which have been previously computed as the decoding is performed from the root node. The first node fed into the decoder is the parent node. The hidden states h capture the information beyond the ancestry nodes as these nodes were able to access other nodes in the tree during attention computation. We can then define the distribution over sequence of sibling nodes given the partial tree P in similar fashion to Equation
The process not only decodes siblings but also produces hidden states that will be accessed in the downstream layers of the tree ancestry states. The decoding process starts from decoding the the children of the root node, because the root node that corresponds the blank design canvas always given. For each non-terminal nodes, I 0, apply the same transformer model to decode its

---

Under review as a conference paper at ICLR 2020
children. The size of ancestry state grows linearly with respect to the depth of the tree (see the schema illustration in Figure 2). Note that in this model, we do not flatten the tree representation and we do not compute parent pointer either. This design allows to easily batch the training and decoding for multiple trees at the same time, where multiple trees can be treated as forest.
EVALUATION
In this section, we evaluate the tree decoders propose for UI layout completion. We start by describing the dataset we use and elaborate on data processing we conducted. We then define the metries for this new problem. Lastly, we report on the results of our experiments.
5.1 DATASETS
We use a public dataset that includes over 60K UT layouts where each element in a layout is labeled with set of properties including its type and bounding boxes (Liu al.) (2018). These screens and hierarchies capture 25 UI component categories, such as text, buttons lists and icons. For each layout, a corresponding UI hierarchy represents the tree structure of the UI.
We preprocess the data to filter out ill-formed layouts. When a node is out of screen or outside its parent, we remove the entire layout out of the dataset. We also filter long-tail examples when layout has more than 100 nodes in the tree or parent has more than 30 children. These long-tail examples are a small portion of the entire dataset. The resulted dataset has more than 55K examples. There are 25 type categories, the average number of nodes layout is 16 (max=49 and min=2), and the average depth of each layout tree is (max=5, min=2). We also scale down the coordinate values to the range of 72x128 for the horizontal and vertical dimensions. At this scale, layout is still readable for human eyes.
5.2 METRICS
As far as we know, there are no established metrics for the problem of layout design completion. To evaluate the quality of each decoder model, we propose three metrics, which address different user scenarios.
Layout Tree Edit Distance The purpose of this metric is to estimate how much effort is required for a designer to transform a predicted layout into the target one. We implement this metric based on optimal tree edit distance calculation (Zhang and Shasha 1989). and ground each tree manipulation cost into time effort based on Keystroke-level GOMS model (Card et al. 1980 Kieras 2019). There are three edit operators in the edit distance caleulation: Insertion. Deletion and Change. For Keystroke-level GOMS analysis, we assume a typical layout editing environment with all the UI elements presented in a palette. For Insertion, the effort involves the designer selecting a target element from the palette, dragging and dropping it onto the design canvas and then resizing it to target dimension. and then select a target type. If the location or size is different, the designer needs to correct them by dragging and dropping as well. Lastly, for Deletion, since the designer can ignore the predictions, the absolute measure of
Change, the type is incorrect, the designer needs to right click on the element
cost is set to minimal. Note that these cost analyses should not be treated effort. Rather, this metric should be considered as a relative measure to compare models.
Parent-Child Pair Retrieval Accuracy The other measure we propose is to treat the task as an information retrieval problem (IR) in which we measure how well the model can predict parent-child pairs against the set of pairs in the ground-truth tree. This metric is less sensitive to the overall tree structure and is only concerned with local layout structures. parent-child node pair, (p. s), is defined as a concatenation of their type and spatial properties, (Cp. Ty. Vp: Ep. 3p. Cs Te. Us P.. ).). pair is successfully retrieved only all the values in the predicted tuple match ground-truth tuple. With this formation, we can compute scores such as precision, recall and their combination Fl scores.
Text-Element Prediction Accuracy Because it is challenging predict the entire tree, we look into the metrics that can capture how well a model predicts next element the designer needs. This analogous next word prediction in language models. For this metric, given prefix of the
7

---

Under review as a conference paper at ICLR 2020
preorder or breadth-fine first traversal sequence of the ground-truth layout tree, we test if the next element, immediately following the prefix, is predicted by the model.
5.3 EXPERIMENTS
Model Configuration & Training We implemented these models TensorFlow based on the Transformer implementation in Tensor2Tensof We split the dataset for training (80%), valida- tion (10%) and test (10%). Based on the training and the validation datasets, we determined an optimal model architecture and hyperparameters, including finding the hidden size in the range of 212,,256, 512] and the number of layers [2, 4, 6) as well as tuning on the learning rates and dropout ratios. We trained each of the models on single machine with 8 Tesla P100 V100 GPU cores, using& batch size of 128 the number layout trees. The training strategy is similar to the one introduced by Transformer (Vaswani et al. 2017). We trained these models until they converge.
Partial Tree Setups Our models do not require a design process to be carried out in a certain order. However, to examine how these models would aid a design, we need to assume a variety of design flows that designer might follow in a realistic design process, which would lead to different layout trees.
One factor is the order that the designer wants to progress design structurally: the depth-first versus the breadth-first strategies. For the depth-first first strategy, the designer starts by adding the container elements (parent nodes) and then finishes the child elements in the container before moving onto the next group of elements. For the breadth-first strategy, the designer will layout top-level elements before adding more detailed designs to each container. In reality, designer might alternate between the two, which we leave for future evaluation. All these strategies are assumed for examining the models although the models can be used for completing a design of arbitrary order.
For each of these flows, we vary the size of the partial layout given to the model, including 10%, 50% and 80% of the full tree. These correspond to the portion of layout design that has been already entered by the designer. Given a partial tree, there can be more than one way to complete the layout. Given a 10%, 50% and 80% BFS partial layout, the mean number of completions of the layout is 2.97, 1.23 and 1.17 respectively. Given 10%, 50% and 80% DFS partial layout, the mean number of completions 3.63, 1.24, and 1.17 respectively.
Results As we can see, Recursive Transformer Decoder outperforms the Pointer Decoder for most cases when the design flow follows the breadth-first traversal (see Table (3), even though Pointer Decoder has the direct access all the partial tree and previously decoded nodes and Recusive Decoder does not. Vanilla Decoder is not tested for the BFS case because it is only structured for DFS as discussed earlier. For the depth-first first traversal case (see Table 2 both Pointer and Recursive outputform the Vanilla version with large margin. However, the trend between Pointer and Recursive decoders is less obvious than the BFS case. This is understandable because the Recursive Transformer decodes in a top-down manner, which cannot leverage the given nodes in the partial tree in deeper layers. The depth-firs first traversal tree put Recursive decoder in a disadvantagcous position. In contrast, Pointer decoder has direct access to all the nodes in the given partial tree. Nevertheless, Recursive Transformer still consistently outperforms Pointer Transformer on Next Item prediction with large margin.
Table 1: The prediction accuracy for the breadth-first layout design flow. The F1 score (%) for retrieving parent-child pairs, next item accuracy (%), and Edit distances are calculated given 10%, 50% and 80% of the ground-truth tree the partial tree, For Fl and Next Item Accuracy, the larger is the better, while for Edit Distance, the smaller the better.
BFS
10%
BFS 50%
BFS 80%
Models
F1
Next
Edit
F1
Next
Edit
F1
Next
Edit
Pointer
11.19
12.25
87.40
61.31
16.76
35.78
87.24
27.56
13.55
Recursive
15.23
18.4
71.26
64.07
26.84
46.47
86.41
36.58
27.22
8

---

Under review as a conference paper at ICLR 2020
Table 2: The prediction accuracy for the depth-first layout design flow. The Fl score (%), next item accuracy (%), and Edit distances are calculated given 10%, 50% and 80% of the ground-truth tree.
DFS
10%
DFS
50%
DFS 80%
Models
F1
Next
Edit
F1
Next
Edit
F1
Next
Edit
Vanilla
5.35
0.74
234.17
40.4
5.35
138.88
68.82
17.03
84.32
Pointer
13.12
20.21
76.44
65.49
12.03
21.41
88.96
22.38
9.13
Recursive
15.05
29.02
62.49
60.79
22.93
52.26
84.65
33.16
34.24
Table 3: The prediction accuracy for the breadth-first layout design flow with random spatial ordering. The Fl score (%) for retrieving parent-child pairs, next item accuracy (%), and Edit distances are calculated given 10%, 50% and 80% of the ground-truth tree the partial tree. For Fl and Next Item Accuracy, the larger is the better, while for Edit Distance, the smaller the better.
BFS 10%
BFS 50%
BFS 80%
Models
F1
Next
Edit
F1
Next
Edit
F1
Next
Edit
Pointer
11.19
12.25
87.40
61.31
16.76
35.78
87.24
27.56
13.55
Recursive
15.23
18.4
71.26
70.0
17.0
22.46
91.8
30.47
11.79
DISCUSSIONS
In this paper, we introduce new problem of auto completion for UI layout design. We formulate the problem as partial tree completion, and investigate range of variations of layout decoders based on Transformer. The two models proposed, Pointer and Recursive Transformer, gave reasonable predictions (see Figure 3). also define task setups and evaluation metrics for examining the quality of prediction and report the results based public dataset.
While both Pointer and Recursive Transformer clearly outperformed the baseline model, we found that layout completion remains challenging task. Our model is often able to predict layout structures and semantic properties well, but less accurate on bounding boxes. There are many UIs with the same layout structure but different spatial details, while our current eval is using hard metrics. When relaxing the metrics by only considering structures and semantic properties, all the models got much better accuracy, e.g., Next item for 80% BFS partial (Recursive: 95.3%, Pointer: 92.9%) and for 80% DFS partial (Recursive: 93.4%, Pointer: 88.4%, Vanilla: 76.6%). Recursive still shows significant advantages over other models. We experimented with continuous coordinate output using squared errors for loss. The results showed that the model with continuous coordinates performs substantially worse than treating coordinates as one-hot categorical values, which deserves further investigation.
REFERENCES
T. Beltramelli. pix2code: Generating code from a graphical user interface screenshot. CoRR, fas/1705.07966 2017. hRtphhhhp://arxiv.orrr abs 1705. 07962]
Y. Bengio, R. Ducharme, Vincent, and Janvin. neural probabilistic language model. Mach. Learn. Res., 3:1 1311131115, Mar. 2003. ISSN 1532-4435. URL http:/ttp///l acm. org/
ci citat ion cfm? 944919 944966
S. K. Card, T. P. Moran, and A. Newell. The keystroke-level model for user performance time with interactive systems. Commun. ACM, 23(7):396-110, July 1980. ISSN 0001-0782. doi: 10.1 15114538888635589955 URLhttp::/ // acm /..1 358895
X. Chen, C. Liu, and D. Song. Tree-to-tree neural networks for program transla- tion. In S. Bengio, Н. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi. and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 2547-2551 Curran Associates, Inc., 2018. URL 7521-tree-to-treee neural orr.

---

Under review as a conference paper at ICLR 2020
partial
prediction
ground-truth
Figure 3: Examples of predicted layouts given partial layout, with respect to the corresponding groundtruth complete layout.
H. Dai, Y. Tian, B. Dai, S. Skiena, and L. Song. Sepnaa-ddreecteddvaraiiiiion variational autoocooeer for structured data. CoRR, tbss/180..0876, 2018. URL /ttp:/ttc//aaaarr..o//////////22
L. Dong and M. Lapata. Language to logical form with neural attention. CoRR, abs/160) 1000//000.,,280, 2016. bttp::t:://axi wwtp//a/////////o/////t//../aaaa 11180.10180
Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2672-26800 Curran Associates, Inc., 2014. 1423-generat.ive htt.:ggenerttt.eeeaaerrrraaaaaea -ee.sn.et.ppp... .
D. Ha and D. Eck. A neural representation of sketch drawings. CoRR, abs/1704.03477, URL
http:://rxiz owt.//gg///////e./ 2ttg/abs/1704.03777
N. Hurst, W. Li, and K. Marriott Review of automatic document formatting. In Proceedings of the 9th ACM Symposium on Document Engineering. DocEng '09, pages 99-108. New York. NY, USA, 2009. ACM. ISBN 978-1-60558 575-8.
doi: 10. 10.11 00.1145/1600193.0 16000221. URL
http://dot aacm. wwt.aaa..............m........ /00g 1ee1111451110 1ete//..1111/16611
1600217
Isola, J. Zhu, T. Zhou, and A. A. Efros. Image-to-iman translation with conditional adversarial networks. CoRR, 1tt///611.070000 2016. URL
abwpg/abb/ii www.b////.ll....... 1611.07004
R. Jenatton, C. Archambeau, J. González, and M. Seeger. Bayesian optimization with tree-structured dependencies. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Con- ference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 1655-1664, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http::/proceediin/
mlr ptwppes////////eetttoooo////
D. Kieras. Goms models for task analysis. 05 2019.
10

---

Under review as a conference paper at ICLR 2020
D. P. Kingma and M. Welling. Auto-encoding variational bayes. In 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16. 2014, Conference Track Proceedings, 2014. BRLhttt://
org/ abs/ 1312. 6114
J. Li, J. Yang, A. Hertzmann, J. Zhang, and T. Xu. Layoutgan: Generating graphic layouts with wireframe discriminators. CoRR, his/1901.06767, 2019. URL htttp://arx.v.org/////
Y. and T.-H. Chang. Auto-completion for user interface design. US Patent, 2011. URL https
7/patents. goog e. com/patents /0s2e 501 69140
T. F. Liu, M. Craft, J. Situ, E. Yumer, R. Mech, and R. Kumar. Learning design semantics for mobile apps. Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, UIST 18, pages 569-579, New York, NY, USA, 2018. ACM. ISBN 978-1- 4503-5948-1, 1. doi: 10. 10.1144/3242587..3 32242655..
URL http:://dd: acm. org/10.1145/
B242587 3242650
N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran. Image transformer. In J. Dy and A. Krause, editors, Proceedings of the 35th International Con- ference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4055 1055-4064, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://proceed// ngs. m press/ / v80/ pa rmar r 8a html
S. E. Reed, A. van den Oord, N. Kalchbrenner, S. G. Colmenarejo, Z. Wang, D. Belov, and N. de Fre- itas. Parallel multiscale autoregressiive density estimation. CoRR, abs/1703.03664 2017. URL
Shiv and C. Quirk. Novel positional encodings to enable tree-structured transformers, 2019.
URL
net forum? SJerrndd---SS-SSSS-
X. Si, Y. Yang. H. Dai, M. Naik, and L. Song. Learning meta-solver for syntax-guided program synthesis. In International Conference on Learning Representations, 2019. URL https:// openreview net Forum? SeepuuniddSyyySnneee
K. S. Tai, R. Socher, and C. D. Manning. Improved semantic representations from tree-structured long short-term memory networks. CoRR, abs/1503.00075, 2015. URL http:: org/ 0.s/15003.0007
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. CoRR, abs/1770.03762,, 2017. URL
1706. 03762]
O. Vinyals, M. Fortunato, and N. Jaitly. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances Neural Information Processing Systems 28, pages 2692-2700. Curran Associates, Inc., 2015a. URL http:/ L http:://ppppr....e www.niicppppp..eeeeee ccppaper//ppppcccpcccccccpe..epp
5866 11166-pointen networks. pdd
O. Vinyals, L. Kaiser, T. Koo, Petrov, Sutskever, and G. Hinton. Grammar as a foreign language. In Proceedings of the 28th International Conference Neural Information Processing Systems Volume 2, NIPS' 15, pages 2773-2781, Cambridge, MA, USA, 2015b. MIT Press. URL http: acm. org. cttp:iitatioooo wtw.ooon......... cfm:1d=29964422.cmmmm 211.44.....99999 2969550
K. Zhang and D. Shasha. Simple fast algorithms for the editing distance between trees and related problems. SIAM COMPUT, 18(6), 1989.
X. Zheng, X. Qiao, Y. Cao, and R. W. H. Lau. Content-aware generative modeling of graphic design layouts. ACM Trans. Graph 13(4):133:1-133:11 July 2019. ISSN 0730-0301. doi: 10.1 N0.114/3306346.332200
TRL/actp://dod acm. org wtt...og/...111...3.33.111...... 0101.01111113333331 3322971
J. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image tramslation usin cyle-oossstll adversarial networks. CORR, 0bs//103..1059) 2017. URL http://:://araw.or/raaa///////// 10593
Z. Zhu, Z. Xue, and Z. Yuan. Automatic graphics program generation using attention-basee hicrar- chical decoder. CoRR, abs/1810 11.11110.1115 11336, 2018.
11


Reviews:
[{"review_id": "rygHqoLsKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an auto completion for UI layout design. The authors formulate the problem as partial tree completion, and investigate a range of variations of layout decoders based on Transformer.\n\nThe paper proposes two models: Pointer and Recursive Transformer. The paper designs three sets of metrics to measure the quality of layout prediction based on the literature and the domain specifics of user interface interaction.\n\nThe writing quality is readable. The presentation is nice. The task of auto completion for UI layout design is relatively new.\n\nThe paper misses the key baseline in Bayesian optimisation using tree structure [1] which can perform the prediction under the tree-structure dependencies.\n\n[1] Jenatton, Rodolphe, et al. \"Bayesian optimization with tree-structured dependencies.\" Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.\n\nNB: the reviewer has low confidence in evaluating this paper.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "r1edRi1-9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper introduces the task of using deep learning for auto-completion in UI design. The basic idea is that given a partially completed tree (representing the design state of the UI), the goal is to predict or \"autocomplete\" the final tree. The authors propose a transformer-based solution to the task, considering three variants: a vanilla approach where the tree is flattened to a sequence, a pointer-network style approach, and a recursive transformer. Preliminary experiments indicate that the recursive model performs best and that the task is reasonable difficulty.\n\nAssessment: Overall, this is a borderline paper, as the task is interesting and novel, but the presentation is lacking in technical detail and there is a lack of novelty on the modeling side.\n\nIn particular, the authors spend a bulk of the paper describing the three different baselines they implement. However, despite the fact that most of the paper is dedicated to the explanation of these baselines. There is not sufficient detail to reproduce the models based on the paper alone. Indeed, without referencing the original Pointer Network and (and especially the) Transformer papers, it would not be possible to understand this paper at all. Further technical background and detail would drastically improve the paper. Moreover, it seems strange that significant space was used to give equations describing simple embedding lookups (i.e., matrix multiplications with one-hot vectors), but the basic technical foundations of Transformers were not adequately explained.  In addition, only the transformer baselines were considered, and it would seem natural to consider LSTM-based baselines, or some other related techniques.  In general, the space that was used to explain the Transformer baselines---which are essentially straightforward ways to adapt transformers to this task---could have been used to give more detail on the dataset. For example, one question is how often a single partial tree has multiple possible completions in the data. \n\nA major issue---mainly due to the lack of technical details and the lack of promise to provide code/data (unless I missed this)---is that the paper does not appear to be reproducible. Given the intent to have this be a new benchmark, ensuring reproducibility seems critical.\n\nReasons to accept:\n- Interesting new application of GNNs\n\nReasons to reject:\n- Incremental modeling contribution\n- Lack of sufficient technical detail on models and dataset\n- Does not appear to be reproducible \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1g05tby5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an autocompletion model for UI layout based on adaptations of Transformers for tree structures and evaluates the models based on a few metrics on a public UI dataset.\n\nI like the area of research the authors are looking into and I think it's an important application. However, the paper doesn't answer key questions about both the application and the models:\n\n1) There is no clear rationale on why we need a new model based on Transformers for this task. What was wrong with LSTMs/GRUs as they've been used extensively for recursive problems including operations on trees? Similarly, I'd have expected baselines that included those models in the evaluation section showing the differences in performance between the newly proposed Transformer model for trees and previously used methods.\n\n2) The evaluation metrics used while borrowed from the language or IR fields doesn't seem to translate to UI design. UI layout is about visual and functional representation of an application so if one is seeking to evaluate different models, they need to relate to those.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an interesting application of GNNs, but the reviewers find that the contribution is too limited and the motivation is too weak.
Decision: Reject