VUT: Versatile UI Transformer for Multimodal Multi-Task User Interface Modeling 

1 INTRODUCTION . Modern graphical user interfaces specifically touchscreen mobile UIs enable a rich problem space for modeling where the input is inherently multimodal , which consists of several distinct types of data . A user interface screen exists in both a visual form , i.e. , a screenshot , and a structural representation , i.e. , a tree-like view hierarchy . Based on graphical user interfaces , there is a wide spectrum of modeling tasks that either directly enhance user experiences or advance the development of intelligent user interfaces . For example , previous work developed models and datasets for grounding a language command into an executable UI action ( Li et al. , 2020a ) , generating language description for accessibility on mobile devices ( Li et al. , 2020b ; Wang et al. , 2021 ) , and understanding the usability of user interfaces ( Swearngin & Li , 2019 ) or identifying the objects on the screen ( Zhang et al. , 2021 ) . Previous work has also started learning effective representation of user interface screens ( He et al. , 2020 ; Li et al. , 2021a ) , which can potentially benefit downstream tasks . Although these previous works have made progress in addressing individual problems , it is important to investigate the feasibility of learning all these tasks with a single model . In addition to achieving a scientific understanding of how these UI tasks are related , it is extremely valuable to obtain such a multi-task model , which can potentially reduce the number of models that need to be developed and deployed . This is crucial for mobile devices that have limited computing resources . In this work , we propose VUT—Versatile UI Transformer , which handles three types of data : images , structures ( view hierarchies ) and language , and simultaneously performs five unique tasks that are representative in the UI modeling literature , including UI object detection , natural language command grounding , widget captioning , screen summarization and UI tappability prediction . A major challenge we need to address is how to unify these distinct tasks as well as their heterogeneous datasets such that they can be learned by a single model . To this end , we devise a general formulation for UI modeling tasks based on five inherent types of information that define a task . We also aim to design a compact model architecture such that it remains stable for addressing a diverse and potentially growing set of tasks , for which we make each model component multi-purpose . Specifically , VUT comprises two Transformer architectures ( Figure 1 ) : the Image-Structure model , and the Question-Answer model . The Image-Structure model encodes the entire screenshot of a UI along its view hierarchy tree , with early fusion of the two modalities , which is guided by a focus map when a given object is inquired . In addition to being the UI encoder , the Image-Structure model predicts UI objects when the view hierarchy is absent on the input , which achieves the UI object detection task . The Question-Answer model encodes a question while attending to the UI encodings from the Image-Structure model . It decodes a text answer when the task response is language , e.g. , widget captioning ( Li et al. , 2020c ) or screen summarization ( Wang et al. , 2021 ) . For the grounding task whose output is an object reference , the Question-Answer model serves as the question encoder and its hidden state is used to locate UI elements to be acted upon . We highlight the relation of VUT with previous works in Table 1 , and discuss their differences further in the following section . We experiment with our model on 5 datasets , and compare the accuracy of VUT when it is trained alone for each task and jointly with multiple tasks . Our experiments show that VUT is able to perform all the five tasks simultaneously and achieve the performance on par with or surpass that when each task is learned alone . The main contributions of our work are as follows . • We formulate multi-modal multi-task learning for a new domain—graphical user interfaces— with one model to accomplish a wide range of tasks for enhancing mobile user experiences . • We design VUT based on a two-tower Transformer architecture , one for handling imagestructure and the other for language data , where each Transformer is multi-purpose by both encoding and decoding its own modality , with cross-tower attention . • We experiment with VUT on 5 distinct UI tasks , and thoroughly investigated the effect of these tasks when learned alone or jointly with ablations and analysis , which show the feasibility for achieving diverse UI tasks using a single model , which offers the value for reducing the number of models and storage footprints needed for deployment ( Appendix E ) . 2 RELATED WORK . Extensive work has been conducted in multi-modal modeling with vision and languages ( Li et al. , 2019 ; Lu et al. , 2019 ; Hu & Singh , 2021 ; Lu et al. , 2020 ; Tan & Bansal , 2019 ; Kim et al. , 2021 ; Zhou et al. , 2020 ; Gupta et al. , 2021 ) . Existing works differ in the form of input they consume and the objectives of modeling . One category of work focuses on pretraining to learn an effective cross-modality representation for downstream tasks and the other directly learns multiple tasks end-to-end ( Table 1 ) . VUT belongs to the latter . In terms of the forms of multimodal data these models consume , most models handle image and text input . However , a unique form of data in UI modeling is the structure input of view hierarchies , which only VUT , UIBert and ActionBert use . Many existing works feed object regions , instead of the entire image to the model , which requires a pretrained object detection model ( Lu et al. , 2019 ; Bai et al. , 2021 ; He et al. , 2020 ; Zhou et al. , 2020 ; Tan & Bansal , 2019 ; Lu et al. , 2020 ) or address tasks only regarding the entire image ( Kim et al. , 2021 ; Hu & Singh , 2021 ) . Although ActionBert ( He et al. , 2020 ) and UIBert ( Bai et al. , 2021 ) also address the UI domain , they are targeted for representation learning , and do not support multiple tasks simultaneously . As a result , they do not deal with language input of task descriptions . Their text input is those scraped from the UI screen , e.g. , using OCR . In addition , these models require predetermined object regions similar to many BERT-based multi-modal models . In contrast , object detection is one of the tasks that VUT addresses . In terms of modeling techniques , we designed a novel Transformer architecture for multi-task modeling of the UI domain , based on building blocks previously proposed for natural images and language , e.g. , ( Hu et al. , 2020 ; Lu et al. , 2020 ) . the work that is closely related to ours is GPVI ( Gupta et al. , 2021 ) , which uses DETR ( Carion et al. , 2020 ) for object detection , and ViBERT ( Lu et al. , 2019 ) for multimodal modeling . In addition to the obvious deviation our work , e.g. , VUT uses structure input but GPV-I does not , there are several important architecture differences . While GPV-I directly embeds DETR , an encoder-decoder model , into its architecture , VUT uses a single tower design where both the image and object queries are fed to the same Transformer encoder . This design choice is motivated by our goal to achieve a compact architecture , which the Image-Structure model serves both image-structure encoding and object detection ( when the structure input is absent in the input ) . As shown in our experiment , the single tower architecture of VUT ’ s Image-Structure model showed clear advantage over the encoder-decoder architecture in DETR for the UI object detection task . To address the unique domin of UI tasks , we also introduce focus map to guide the model towards the object being inquired . VUT ’ s question-answer Transformer is designed based on existing auto-regressive multi-task language models ( Raffel et al. , 2019 ; Brown et al. , 2020 ) where a question or a command is fed to the model as a prefix , and the responses are decoded token by token . One difference is that for the language command grounding task , instead of generating a language response , the last hidden state of the language model is used , as a question encoding , to retrieve a UI object on the screen . 3 PROBLEM FORMULATION . A graphical user interface contains a collection of UI elements for fulfilling a coherent set of tasks . There are often five types of data involved to formulate a UI task : < S , V , T , Q , A > ( Figure 1 ) . S is the screenshot image that captures the visual appearance of the UI screen . V is the view hierarchy tree that represents the underlying structure of the screen . T is the target object on the screen to be inquired . Q is the natural language description of the task , which can be an open-ended question such as `` What is the caption of the element ? `` , a yes-or-no question such as `` Does the object look clickable ? '' or a command such as `` Click on the Next button. '' . See the full list of Q used in our experiments in Appendix A . Finally , A is the natural language answer to the question Q when the form of the response for the task is supposed to be natural language . Depending on each task setup , these data types appear as either input or output . We elaborate on the formation of each task here , and use F to denote the function for achieving each task . 3.1 UI OBJECT DETECTION . Given the screenshot image , S , the task is to detect each UI element on the screen , such as Text Field , Toggle Button , or Image View . This task is similar to the typical object detection task in natural images ( Carion et al. , 2020 ) or recent UI object detection work ( Zhang et al. , 2021 ) . However , our task is more challenging in that it needs to detect different types of container objects , which determine how UI objects are visually structured of the screen , such as Linear Layout , Frame Layout or List . In total there are 21 types of leaf or non-leaf objects in a view hierarchy . See Appendix D for the full list of objects we detect . UI object detection is important for improving accessibility and enabling other intelligent features such as UI adaptation when view hierarchy is not available . As a screen understanding task , UI object detection is beneficial to other UI modeling tasks as we will show in our experiments . The task is formulated as the follow ( Equation 1 ) . V = F ( S , V∅ , T∅ , Q∅ ) ( 1 ) Note that this task is achieved solely based on the single-tower Image-Structure Transformer ( Figure 1 ) and does not rely on the question-answer model . V∅ , T∅ and Q∅ represent each type of data masked out or missing in the input .

Summary: A multi-modal multi-task model is proposed for UI understanding. It has an Image-Structure Transformer and Question-Answer Transformer to encode/decode image, structure and language information. The method is evaluated thoroughly on 5 different data sets and achieves good results.