VUT: Versatile UI Transformer for Multimodal Multi-Task User Interface Modeling 

1 INTRODUCTION . Modern graphical user interfaces specifically touchscreen mobile UIs enable a rich problem space for modeling where the input is inherently multimodal , which consists of several distinct types of data . A user interface screen exists in both a visual form , i.e. , a screenshot , and a structural representation , i.e. , a tree-like view hierarchy . Based on graphical user interfaces , there is a wide spectrum of modeling tasks that either directly enhance user experiences or advance the development of intelligent user interfaces . For example , previous work developed models and datasets for grounding a language command into an executable UI action ( Li et al. , 2020a ) , generating language description for accessibility on mobile devices ( Li et al. , 2020b ; Wang et al. , 2021 ) , and understanding the usability of user interfaces ( Swearngin & Li , 2019 ) or identifying the objects on the screen ( Zhang et al. , 2021 ) . Previous work has also started learning effective representation of user interface screens ( He et al. , 2020 ; Li et al. , 2021a ) , which can potentially benefit downstream tasks . Although these previous works have made progress in addressing individual problems , it is important to investigate the feasibility of learning all these tasks with a single model . In addition to achieving a scientific understanding of how these UI tasks are related , it is extremely valuable to obtain such a multi-task model , which can potentially reduce the number of models that need to be developed and deployed . This is crucial for mobile devices that have limited computing resources . In this work , we propose VUT—Versatile UI Transformer , which handles three types of data : images , structures ( view hierarchies ) and language , and simultaneously performs five unique tasks that are representative in the UI modeling literature , including UI object detection , natural language command grounding , widget captioning , screen summarization and UI tappability prediction . A major challenge we need to address is how to unify these distinct tasks as well as their heterogeneous datasets such that they can be learned by a single model . To this end , we devise a general formulation for UI modeling tasks based on five inherent types of information that define a task . We also aim to design a compact model architecture such that it remains stable for addressing a diverse and potentially growing set of tasks , for which we make each model component multi-purpose . Specifically , VUT comprises two Transformer architectures ( Figure 1 ) : the Image-Structure model , and the Question-Answer model . The Image-Structure model encodes the entire screenshot of a UI along its view hierarchy tree , with early fusion of the two modalities , which is guided by a focus map when a given object is inquired . In addition to being the UI encoder , the Image-Structure model predicts UI objects when the view hierarchy is absent on the input , which achieves the UI object detection task . The Question-Answer model encodes a question while attending to the UI encodings from the Image-Structure model . It decodes a text answer when the task response is language , e.g. , widget captioning ( Li et al. , 2020c ) or screen summarization ( Wang et al. , 2021 ) . For the grounding task whose output is an object reference , the Question-Answer model serves as the question encoder and its hidden state is used to locate UI elements to be acted upon . We highlight the relation of VUT with previous works in Table 1 , and discuss their differences further in the following section . We experiment with our model on 5 datasets , and compare the accuracy of VUT when it is trained alone for each task and jointly with multiple tasks . Our experiments show that VUT is able to perform all the five tasks simultaneously and achieve the performance on par with or surpass that when each task is learned alone . The main contributions of our work are as follows . • We formulate multi-modal multi-task learning for a new domain—graphical user interfaces— with one model to accomplish a wide range of tasks for enhancing mobile user experiences . • We design VUT based on a two-tower Transformer architecture , one for handling imagestructure and the other for language data , where each Transformer is multi-purpose by both encoding and decoding its own modality , with cross-tower attention . • We experiment with VUT on 5 distinct UI tasks , and thoroughly investigated the effect of these tasks when learned alone or jointly with ablations and analysis , which show the feasibility for achieving diverse UI tasks using a single model , which offers the value for reducing the number of models and storage footprints needed for deployment ( Appendix E ) . 2 RELATED WORK . Extensive work has been conducted in multi-modal modeling with vision and languages ( Li et al. , 2019 ; Lu et al. , 2019 ; Hu & Singh , 2021 ; Lu et al. , 2020 ; Tan & Bansal , 2019 ; Kim et al. , 2021 ; Zhou et al. , 2020 ; Gupta et al. , 2021 ) . Existing works differ in the form of input they consume and the objectives of modeling . One category of work focuses on pretraining to learn an effective cross-modality representation for downstream tasks and the other directly learns multiple tasks end-to-end ( Table 1 ) . VUT belongs to the latter . In terms of the forms of multimodal data these models consume , most models handle image and text input . However , a unique form of data in UI modeling is the structure input of view hierarchies , which only VUT , UIBert and ActionBert use . Many existing works feed object regions , instead of the entire image to the model , which requires a pretrained object detection model ( Lu et al. , 2019 ; Bai et al. , 2021 ; He et al. , 2020 ; Zhou et al. , 2020 ; Tan & Bansal , 2019 ; Lu et al. , 2020 ) or address tasks only regarding the entire image ( Kim et al. , 2021 ; Hu & Singh , 2021 ) . Although ActionBert ( He et al. , 2020 ) and UIBert ( Bai et al. , 2021 ) also address the UI domain , they are targeted for representation learning , and do not support multiple tasks simultaneously . As a result , they do not deal with language input of task descriptions . Their text input is those scraped from the UI screen , e.g. , using OCR . In addition , these models require predetermined object regions similar to many BERT-based multi-modal models . In contrast , object detection is one of the tasks that VUT addresses . In terms of modeling techniques , we designed a novel Transformer architecture for multi-task modeling of the UI domain , based on building blocks previously proposed for natural images and language , e.g. , ( Hu et al. , 2020 ; Lu et al. , 2020 ) . the work that is closely related to ours is GPVI ( Gupta et al. , 2021 ) , which uses DETR ( Carion et al. , 2020 ) for object detection , and ViBERT ( Lu et al. , 2019 ) for multimodal modeling . In addition to the obvious deviation our work , e.g. , VUT uses structure input but GPV-I does not , there are several important architecture differences . While GPV-I directly embeds DETR , an encoder-decoder model , into its architecture , VUT uses a single tower design where both the image and object queries are fed to the same Transformer encoder . This design choice is motivated by our goal to achieve a compact architecture , which the Image-Structure model serves both image-structure encoding and object detection ( when the structure input is absent in the input ) . As shown in our experiment , the single tower architecture of VUT ’ s Image-Structure model showed clear advantage over the encoder-decoder architecture in DETR for the UI object detection task . To address the unique domin of UI tasks , we also introduce focus map to guide the model towards the object being inquired . VUT ’ s question-answer Transformer is designed based on existing auto-regressive multi-task language models ( Raffel et al. , 2019 ; Brown et al. , 2020 ) where a question or a command is fed to the model as a prefix , and the responses are decoded token by token . One difference is that for the language command grounding task , instead of generating a language response , the last hidden state of the language model is used , as a question encoding , to retrieve a UI object on the screen . 3 PROBLEM FORMULATION . A graphical user interface contains a collection of UI elements for fulfilling a coherent set of tasks . There are often five types of data involved to formulate a UI task : < S , V , T , Q , A > ( Figure 1 ) . S is the screenshot image that captures the visual appearance of the UI screen . V is the view hierarchy tree that represents the underlying structure of the screen . T is the target object on the screen to be inquired . Q is the natural language description of the task , which can be an open-ended question such as `` What is the caption of the element ? `` , a yes-or-no question such as `` Does the object look clickable ? '' or a command such as `` Click on the Next button. '' . See the full list of Q used in our experiments in Appendix A . Finally , A is the natural language answer to the question Q when the form of the response for the task is supposed to be natural language . Depending on each task setup , these data types appear as either input or output . We elaborate on the formation of each task here , and use F to denote the function for achieving each task . 3.1 UI OBJECT DETECTION . Given the screenshot image , S , the task is to detect each UI element on the screen , such as Text Field , Toggle Button , or Image View . This task is similar to the typical object detection task in natural images ( Carion et al. , 2020 ) or recent UI object detection work ( Zhang et al. , 2021 ) . However , our task is more challenging in that it needs to detect different types of container objects , which determine how UI objects are visually structured of the screen , such as Linear Layout , Frame Layout or List . In total there are 21 types of leaf or non-leaf objects in a view hierarchy . See Appendix D for the full list of objects we detect . UI object detection is important for improving accessibility and enabling other intelligent features such as UI adaptation when view hierarchy is not available . As a screen understanding task , UI object detection is beneficial to other UI modeling tasks as we will show in our experiments . The task is formulated as the follow ( Equation 1 ) . V = F ( S , V∅ , T∅ , Q∅ ) ( 1 ) Note that this task is achieved solely based on the single-tower Image-Structure Transformer ( Figure 1 ) and does not rely on the question-answer model . V∅ , T∅ and Q∅ represent each type of data masked out or missing in the input .

Summary: The paper proposes an architecture for graphical user interfaces which involve multi-modal inputs (UI screenshots, Hierarchy structures, Natural Language) and multi-task learning (UI Object Detection, Widget Captioning, Screen summarization, Language grounding, and Tappability).The proposed architecture consists of seperate transformer blocks to encode image and text modalities. The two transformer blocks attends to each other to produce multi-modal outputs, which is then used for downstream tasks. The authors that with the proposed architecture, training on all tasks simultaneously is better than training on individual task alone.