Auto Completion of User Interface Layout Design Using Transformer-Based Tree Decoders

1 INTRODUCTION . Layout design is a universal task in many domains , ranging from mechanical design to graphical layouts . There has been a long tradition in developing computer-aided design ( CAD ) tools in both academia and industry 1 ( Hurst et al. , 2009 ) . Recently , there has been increasing interest in developing deep learning models for design generation ( Zheng et al. , 2019 ; Ha and Eck , 2017 ; Li et al. , 2019 ; Isola et al. , 2016 ) . In this paper , we focus on models that can automate graphical user interface ( GUI ) layout design , a central task in modern software development such as smartphone apps . As a designer or app developer creates an interface design , a model recommends UI elements based on the partial design that has been entered . The automation can significantly ease the effort of designers and developers for creating interfaces , because it not only reduces the input required but also brings design knowledge to the process . This is analogous to auto completion that is widely available in text editing tools , although layout completion is a much more complicated problem . The task is appealing to the deep learning field for a number of reasons . First , it is non-linear and involves 2D placement of elements , instead of the sequential next word prediction for text completion . Second , the problem is highly structural that takes structural input—a partial tree—and generates structural output—a completed tree . For auto-completion of words , a typical choice of solution is using a language model that is trained to predict next word wt+1 given previous words w∗t+1 = argmaxP ( wt+1|w0 , ... , wt ) . As a generative model , a language model can essentially complete a sentence by predicting the rest words in an auto-regressive fashion . Intuitively , we need a layout model—a layout decoder—that predicts the rest elements and structures needed to complete a given partial tree . Previously , models for tree structure generation have been proposed for a number of problems such as language syntax trees ( Vinyals et al. , 2015b ) and program generation ( Chen et al. , 2018 ; Dai et al. , 2018 ) . In these problems , the model first encodes the source input via an encoder then generates tree structures in a target domain via decoding . For layout auto completion , we focus on solely involving the decoding process , although we can potentially include an encoder to bring in additional information . Additionally , our problem involves unique aspects of 2D placements . 1https : //balsamiq.com , https : //www.sketch.com We design our layout decoder model based on Transformer , an model that has gained popularity on a number of tasks ( Vaswani et al. , 2017 ) . The attention-based nature of Transformer allows us to easily model structures , similar to Pointer Networks ( Vinyals et al. , 2015a ) , and represent 2D spatial relationships . In particular , we examine two versions of Transformer-based tree decoder : Pointer Transformer and Recursive Transformer for this task . Layout completion , as a structural prediction problem , lacks evaluation metrics . We design three sets of metrics to measure the quality of layout prediction based on the literature and the domain specifics of user interface interaction . We experimented with these models on a public dataset of over 50K Android user interface layouts . The experiments indicate that these models can bring values to a user interface layout design process . 2 RELATED WORK . Recently , there have been increasing efforts in using deep learning to enhance design practice with Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) , Variational Auto Encoders ( Kingma and Welling , 2014 ) , and Autoregressive models ( Reed et al. , 2017 ) as three major underlying approaches . There is a rich body of work based on Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ; Isola et al. , 2016 ; Zhu et al. , 2017 ) where a discriminator is often introduced during training for distinguishing synthesized and real designs , which in the end the model learns to generate realistic designs . In particular , LayoutGAN ( Li et al. , 2019 ) is the most related to our work in the literature , which takes a collection of randomly parameterized ( e.g. , positioned and sized ) elements via an encoder and generates the well-arranged elements via a generator . Although we also generate 2D layouts , our problem is substantially different because it takes in a tree-structured 2D partial layout and outputs a tree-structured 2D full layout . Our model does not see all the elements as LayoutGAN does and our input and output involves a tree structure . Our approach is related to SketchRNN ( Ha and Eck , 2017 ) , a model that can generate sketch drawings . SketchRNN takes a latent representation that is learned via Variational Inference ( Kingma and Welling , 2014 ) and generate or complete a drawing using an autoregressive decoder ( Reed et al. , 2017 ) . Similar to SketchRNN , our decoder is also an autoregressive model . The unique challenge in our case is the structure representation and generation . We can easily extend our models to take a varitional input as SketchRNN , although we focus on tree 2D decoding in this paper and leave the varitional input to future work . pix2code is another work that is related to our effort , which generates UI specifications from a screenshot pixels ( Beltramelli , 2017 ) . It uses a CNN to encode a UI screen and generate UI specification with an LSTM autoregressive decoder . In this work , a tree structure is handled as a flattened sequence such that the decoding can be handled in the same way as language sentences , which a similar treatment is conducted for syntax parse tree prediction ( Vinyals et al. , 2015b ) . Zhu et al . took one step further to propose an attention-based hierarchical decoder ( Zhu et al. , 2018 ) where two LSTMs are used : one for deciding the blocks that the program needs and the other generating tokens within a block . Our recursive Transformer is applied hierarchically as well although we use the same model repetitively . While our target domain is user interface layouts , our problem is fundamentally related to structure prediction . Previously , much efforts have been devoted to program generation ( Chen et al. , 2018 ; Dai et al. , 2018 ; Si et al. , 2019 ) . Jenatton et al . ( Jenatton et al. , 2017 ) proposed an approach to predict tree structures by using Bayesian optimization to combine independent Gaussian Processes with a linear model that encodes a tree-based structure . An important strategy that was explored previously is to extend inherently sequential recurrent models such as LSTM to the hierarchical situation ( Tai et al. , 2015 ) , which can handle a range of tree structures . Particularly , Chen et al. ’ s decoder generates a binary tree by applying LSTM recursively ( Chen et al. , 2018 ) , with one LSTM for generating the left child and the other for generating the right child . Dong and Lapata applied LSTM in a recursive fashion to generate logic forms from language input ( Dong and Lapata , 2016 ) . Based on the previous work , we design our recursive tree decoder based on Transformer ( Vaswani et al. , 2017 ) to handle arbitrary trees . Rather than carrying the hidden state and cell values as recurrent nets , our model can access all the nodes on the ancestry path via attention . Our positional encoding of 2D coordinates is similar to Image Transformer ( Parmar et al. , 2018 ) . It is possible to employ tree positional encoding as proposed in ( Shiv and Quirk , 2019 ) when we keep the hidden states of all the previously generated nodes . Our current design for recursive Transformer decoder is to make the ancestry hidden states accessible for decoding child nodes . Another version of the tree decoder we experiment with in the paper is designed based on Pointer Networks ( Vinyals et al. , 2015a ) where each node points to its parent based on the dot product similarity of their hidden states , which is an simple extension based on Transformer . Finally , the problem we focus on is originated from the domain of human computer interaction ( HCI ) and graphical layout design . A rich body of works have been produced previously ( Hurst et al. , 2009 ) . The idea of UI auto completion has been previously envisioned ( Li and Chang , 2011 ) . DesignScape is a full-fledged tool that provides both refinement suggestions and brainstorming suggestions during graphical layout design . Recently , Liu et al . mined a rich set of Android apps that produced over 66K UI screens with semantically labeled elements ( Liu et al. , 2018 ) , including their types , bounding boxes and the hierarchical tree structures . This dataset lays the foundation for investigating the problem of structured graphical layout prediction , which constitutes the dataset for our work . Our work is also nourished by the rich literature in the HCI domain on human performance modeling . One important aspect that is significantly underexplored for tree prediction is the accuracy metrics . We design a tree-edit distance metric based on keystroke level GOMS models ( Card et al. , 1980 ) that estimates how much effort it needs for a designer to transform the predicted layout tree into the ground-truth tree , which indicates the usefulness of a predicted layout tree . Along a few other metrics , these define a new problem that deep learning models can further tackle . 3 UI LAYOUT COMPLETION PROBLEM . Here , we consider the problem of completing a partial tree , which represents the graphical layout that has been entered by the designer so far ( see Figure 1 ) . A design process starts with an empty design canvas , which corresponds to a partial tree with only the root node . Each node has several properties , including its type , e.g. , a button or a list , whether the node is terminal , and the rectangular bounds of the node . As the designer adds more elements to the canvas , the tree grows . Node A is a parent of node B if A is a non-terminal node and A contains B . The problem is extremely challenging due to the large space of possibilities . 3.1 PROBLEM DEFINITION . A graphical layout tree , Ĝ , contains a collection of |Ĝ| nodes where its i-th node carries both spatial and semantic properties : ( ci , di , ti , pi , bi ) , 1 ≤ i ≤ |Ĝ| and |Ĝ| is the number of nodes in the tree . ci is a categorical value that indicates the node type , ci ∈ C where C is the set of possible element types . di is an integer that represents the depth of the node in the tree . ti is a binary value that represents whether the node is a terminal node . pi is the index position of the node i ’ s parent node , 1 ≤ pi ≤ |Ĝ| . bi is a tuple that represents the bounding box of the node , including its top-left and right-bottom corners : bi = ( xi , yi , x̂i , ŷi ) . A partial tree , P̂ , contains a collection of k nodes ( k < |Ĝ| ) . It is required that the root node must be in P̂ . In addition , the parent of each node in P̂ , except the root node , should also be in P̂ , i.e. , pj ∈ P̂ if j ∈ P̂ , such that the parent index , pj , is always valid and addresses an existing node in the partial tree . This requirement matches how a design tool is typically implemented where each UI element ( a node ) is added to either the design canvas ( the root node ) or a container node that is already attached to the layout tree . Given the partial tree P̂ , the task here is to predict the full tree Ĝ ′ based on the partial tree : Ĝ ′ = F ( P̂ ) . We elaborate on how the function F can be realized with a variety of Transformer decoders in the next section .

Summary: This paper introduces the task of using deep learning for auto-completion in UI design. The basic idea is that given a partially completed tree (representing the design state of the UI), the goal is to predict or "autocomplete" the final tree. The authors propose a transformer-based solution to the task, considering three variants: a vanilla approach where the tree is flattened to a sequence, a pointer-network style approach, and a recursive transformer. Preliminary experiments indicate that the recursive model performs best and that the task is reasonable difficulty.