VUT: Versatile UI Transformer for Multimodal Multi-Task User Interface Modeling 
Authors: Yang Li
Gang Li
Xin Zhou
Mostafa Dehghani
Alexey A. Gritsenko
Venue: ICLR 2022 Submitted (2022)
Keywords: User Interface Modeling
Multimodal input
Multi-task learning
Transformer
Layout Detection
Language Grounding
Image Captioning
Screen Summarization
Tappability Prediction.

Abstract: User interface modeling is inherently multimodal, which involves several distinct types of data: images, structures and language. The tasks are also diverse, including object detection, language generation and grounding. In this paper, we present VUT, a Versatile UI Transformer that takes multimodal input and simultaneously accomplishes 5 distinct tasks with the same model. Our model consists of a multimodal Transformer encoder that jointly encodes UI images and structures, and performs UI object detection when the UI structures are absent in the input. Our model also consists of an auto-regressive Transformer model that encodes the language input and decodes output, for both question-answering and command grounding with respect to the UI. Our experiments show that for most of the tasks, when trained jointly for multi-tasks, VUT has achieved accuracy either on par with or exceeding the accuracy when the model is trained for individual tasks separately.

Under review as a conference paper at ICLR 2022
VUT: VERSATILE UI TRANSFORMER FOR MULTI- MODAL MULTI-TASH USER INTERFACE MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
User interface modeling is inherently multimodal, which involves several distinct types of data: images, structures and language. The tasks are also diverse, including object detection, language generation and grounding. In this paper, we present VUT, Versatile UI Transformer that takes multimodal input and simultaneously accomplishes 5 distinct tasks with the same model. Our model consists of a multimodal Transformer encoder that jointly encodes UI images and structures, and performs UI object detection when the UI structures absent in the input. Our model also consists of an auto-regressive Transformer model that encodes the language input and decodes output, for both duestion-answeree grounding with respect to the UI. Our experiments show that for most of the tasks, when trained jointly for multi-tasks, VUT substantially reduces the number of models and footprints needed for performing multiple tasks, while achieving accuracy exceeding or on par with bascline models trained for each individual task.
and command
INTRODUCTION
Modern graphical user interfaces specifically touchscreen mobile UIs enable rich problem space for modeling where the input is inherently multimodal, which consists of several distinct types of data. A user interface screen exists in both a visual form, i.e., screenshot, and structural representation, i.e., a tree-like view hierarchy. Based on graphical user interfaces, there is wide spectrum of modeling tasks that either directly enhance user experiences or advance the development of intelligent user interfaces. For example, previous work developed models and datasets for grounding a language command into an executable UI action (Li et al., 2020a), generating language description for acces- sibility on mobile devices (Li et al. 2020b Wang et al. 2021), and understanding the usability of user interfaces (Swearngin & Li| 2019) or identifying the objects on the screen (Zhang et al. 2021). Previous work has also started learning effective representation of user interface screens (He et al.) 2020
Li et al.
2021a), which can potentially benefit downstream tasks.
Although these previous works have made progress in addressing individual problems, it is important to investigate the feasibility of learning all these tasks with a single model. In addition to achieving a scientific understanding of how these UT tasks related, it is extremely valuable to obtain such multi-task model, which can potentially reduce the number of models that need to be developed and deployed. This crucial for mobile devices that have limited computing resources. In this work, we propose Ver-Versatile UI Transformer, which handles three types of data: images, structures (view hierarchies) and language, and simultaneously performs five unique tasks that are representative in the UI modeling literature, including UI object detection, natural language command grounding, widget captioning, screen summarization and UI tappability prediction.
A major challenge we need to address is how to unify these distinct tasks as well as their heterogencous datasets such that they can be learned by a single model. To this end, we devise a general formulation for UI modeling tasks based on five inherent types of information that define task. We also aim to design compact model architecture such that it remains stable for addressing a diverse and potentially growing set of tasks, for which make each model component multi-purpose. Specifically, VUT comprises two Transformer architectures (Figure T): the Image-Structte and the Question-Answer model. The Image-Structurn model encodes the entire screenshot of UI along its view hierarchy tree, with early fusion of the two modalities, which is guided by a foccus map when given object is inquired. In addition to being the UI encoder, the Image-Structie model
model,

---

Under review as a conference paper at ICLR 2022
Tasko Objectives
Ul Object Detection Task
Command Grounding Task
Widget Captioning Task Screen Summerizaiion Task
Tappabilty Prediction lask
Object Output Layer
Hidden Rupresentation
Tranaformer
Image: Structre Transtormer
Question Answer Transtormer
Aftertion
Foous Map Moduloting
P
Image Focus Map
Objject Focus Map
P
Ogy
Qgy
Question O
Quarition recamples:
Embedding
Image Embeder
Structure Embeder
Dearch reseby gas shallons.
What caption wecment?? Would object look luppebll??
Ul Object encodiig M,
Image encoding H, Question tokens
View Hierarchy
Screenshot
Target Object
Answer tokans
QA Transformer hidienn g)
absont
Pointer
Transform War
Text Head Linear Transform War
Figure 1: The VUT model architecture contains two Transformer models, which take image, structure and language input, and three task heads for achieving five distinet UI modeling tasks.
predicts UI objects when the view hierarchy is absent the which, achieves the UI object detection task. The Question-Answe model encodes a question while attending to the UI encodings from the StruetStructure model. It decodes a text answer when the task response is language, e.g.. widget captioning (Li et al. 2020c) or screen summarization (Wang et al.] (021). For the grounding task whose output and its hidden state is used to locate UI elements to be acted upon. We highlight the relation of VUT with previous works in Table D and discuss their differences further in the following section.
object reference, the Question-Answer model serves as the question encoder
We experiment with our model on datasets, and compare the accuracy of VUT when it is trained alone for each task and jointly with multiple tasks. Our experiments show that VUT is able to perform all the five tasks simultaneously and achieve the performance on par with or surpass that when each task learned alone. The main contributions of our work are as follows.
We formulate multi modal multi-task learning for a new domain- with one model to accomplish a wide range of tasks for enhancing mobile user experiences.
user interfaces-
We design VUT based on two-tower Transformer architecture, one for handling image- structure and the other for language data, where each Transformer is multi-purpose by both encoding and decoding its own modality, with cross-tower attention.
We experiment with VUT on 5 distinct UI tasks, and thoroughly investigated the effect these tasks when learned alone or jointly with ablations and analysis, which show the feasibility for achieving diverse UI tasks using single model, which offers the value for reducing the number of models and storage footprints needed for deployment (Appendix E)
RELATED WORK
Extensive work has been conducted in multi-modal modeling with vision and languages (Li et al. 2019; Lu et al.) 2019 Hu & Singh 2021 Lu et al.) 2020 Tann & Bansal 2019 Kim et al. 2021 Zhou et al., 2020 Gupta et al., 2021). Existing works differ in the form of input they consume and the objectives of modeling. One category of work focuses pretraining to learn an effective modalimy representation for downstream tasks and the other directly learns multiple tasks end-to-end (Table [.. VUT belongs to the latter. In terms of the forms of multimodal data these models consume, most models handle image and text input. However, a unique form of data in UI modeling is the structure input of view hierarchies, which only VUT, UlBert and ActionBert use.
2

---

Under review as a conference paper at ICLR 2022
Model
Image input
Structure input
Text input
Objectives
ViLBERT (L 2019
Object regions
None
Object captions
Pretraining
UlBertt (Bai ct all
20
Object regions
View hierarchy
Object text
Pretraining
ActionBert (He al.
2020
Object regions
View hierarchy
Object text
Pretraining
ViLT Kim al. 2021
Entire image
None
Image caption
Pretraining
VLP (Zhou et all 20200
Object regions
None
Image caption
Pretraining
LXMERT Tan Bansal
2019
Object regions
None
Image caption
Pretraining
12-in-1 (Lu et al. 2002))
Object regions
None
Task prompts
Multi-task
UniT (Hu & Singh 2021
Entire image
None
Task prompts
Multi-task
GPV-I el al.
Entire image
None
Task prompts
Multi-task
VUT (our model)
Entire image
View hierarchy
Task prompts
Multi-task
Table 1: Comparison of VUT with several existing multi-modal modeling works.
Many existing works feed object regions, instead of the entire image to the model, which requires a
pretrained object detection model (Lu et al. 2019 Bai et al.) 2021 He al. 2020 Zhou et al. 2020
Tan & Bansal 2019 Lu et al. 2020) or address tasks only regarding the entire image Kim et al.
20 & Singh 2021). Although ActionBert (He et al., 2020) and UlBert (Bai et al. 2021 also address the UI domain, they are targeted for representation learning, and do not support multiple tasks simultaneously. As result, they do not deal with language input of task descriptions. Their text input is those scraped from the UI screen, c.g., using OCR. In addition, these models require predetermined object regions similar to many BERT-based multi-modal models. In contrast, object detection is one of the tasks that VUT addresses.
In terms of modeling techniques, we designed novel Transformer architecture for multi-task modeling of the UI domain, based on building blocks previously proposed for natural images and language, e-g., (Hu et al. 2020 Lu et al. 2020). the work that is closely related to ours GPV. I (Gupta et 20211) which uses DETR (Carion et al. (020) for object detection, and ViBERT (Lu ct al. 2019) for multimodal modeling. In addition to the obvious deviation our work, c.g., VUT uses structure input but GPV-I does not, there are several important architecture differences. While GPV-I directly embeds DETR, an encoder-decoder model, into its architecture, VUT uses a single tower design where both the image and object queries are fed to the same Transformer encoder. This design choice is motivated by our goal to achieve a compact architecture, which the Image-Structur model serves both image-structure encoding and object detection (when the structure input is absent in the input). As shown in our experiment, the single tower architecture of VUT's Image-Structure model showed clear advantage over the encoder-decoder architecture in DETR for the UI object detection task. To address the unique domin of UI tasks, we also introduce focus map to guide the model towards the object being inquired. VUT's question-answe Transformer is designed based on existing auto-regressive multi-task language models (Raffel et al.) 2019 Brown et al.. 2020) where a question or a command is fed to the model as prefix, and the responses are decoded token by token. One difference is that for the language command grounding task, instead of generating a language response, the last hidden state of the language model used, as a question encoding, to retrieve UI object on the screen.
3 PROBLEM FORMULATION
A graphical user interface contains a collection of UI elements for fulfilling a coherent set of tasks. There are often five types of data involved to formulate UI task: S, V,T, Q. (Figure D is the screenshot image that captures the visual appearance of the UI screen. V is the view hicrarchy tree that represents the underlying structure of the screen. T the target object on the screen to be inquired. Q is the natural language description of the task, which can be an open-ended question such as "What is the caption of the element?", a yes-or-no question such as "Does the object look clickable?" ?" or command such "Click on the Next button.". ". See the full list of Q used in our experiments in Appendix| A Finally, A is the natural language answer to the question Q when the form of the response for the task is supposed to be natural language. Depending on each task setup, these data types appear as either input or output. We elaborate on the formation of each task here, and use F to denote the function for achieving each task.
3

---

Under review as a conference paper at ICLR 2022
3.1 UI OBJECT DETECTION
Given the screenshot image, S, the task is to detect each UI element on the screen, such as Text Field, Toggle Button, Image View. This task is similar the typical object detection task in natural images (Carion et al. 2020) or recent UI object detection work (Zhang et al. 2021). However, our task is more challenging in that it needs to detect different types container objects, which determine how UI objects are visually structured of the screen, such as Linear Layout, Frame Layout or List. In total there are 21 types of leaf non-leaf objects view hierarchy. See Appendix [D for the full list of objects we detect. UI object detection is important for improving accessibility and enabling other intelligent features such as UI adaptation when view bierarchy is not available. As screen understanding task, UI object detection is beneficial to other UI modeling tasks will show in our experiments. The task is formulated as the follow (Equation I.
V F(S, Vø, Tz, Q2)
Note that this task achieved solely based on the single-tower Image-Struettn Structure Transformer (Figure and does not rely the question-answer model. Vz. To and Qe represent each type of data masked out or missing in the input.
3.2 WIDGET CAPTIONING
Generating natural language description for user interface elements is important for accessibility and language-based interaction in general. The widget captioning task was initially proposed by Li
al. (2020b) and extends the classic image captioning tasks (Xu et al.) (2055) to the UI domain. In this task, given the UI view hierarchy, V, the screenshot image, S, and the target element to be
captioned, T,, the model predicts a natural language phrase, A, that best describes the functionality of the object (Equation
A = F(S, ==F(S,,, V,,,
(2)
The model uses the information of S, V and T via the Image-Structure model. The examples of Q "What is the caption of the element?" and "What best describes the object?", and the examples of A are "Forward", and "Shopping Cart".
3.3 SCREEN SUMMARIZATION
Instead of focusing on an individual element as the widget captioning task. screen summarization that is recently proposed by Waan et al. (201)) is the task that describes the entire UT screen (Equation 3) by producing a summary phrase.
A = Q)
(3)
The examples of task prompts, Q, for screen summarization are "What is the description of the screen?" and "What best summarizes the UI?" Q signals the model the type of responses it should generate when it is jointly trained with other tasks such as widget captioning. The screen summariza- tion task broadly related to multimodal summarization tasks in the literature, but specific the user interface domain.
3.4 LANGUAGE COMMAND GROUNDING
An important feature of modern smartphone interfaces is to interpret the natural language command of users as executable actions, e.g., Voice Control Previous work has investigated language grounding on user interfaces (Pasupat et al., 2018 Li et al. 2020a). In this task, given the UI, S and V. and the language command, Q, the model needs to predict which object on the screen can fulfill the command, which is the opposite to the two preceding tasks of language generation.
T= F(S,
=en
https::/support. http://pp....aaapoocooooooa///aa
4

---

Under review as a conference paper at ICLR 2022
Note that instead of generating natural language response like widget caption and screen summa- rization, this task locates the target object, T. on the screen. In our dataset, we asked labelers to refer to object
the screen in an unconstrained manner. The possibility of Q is unbounded, which can be any phrase the user feels like using for manipulating the UI, e.g., "Go to the next screen", or "Tap on the checkout button". A command can also refer an object indirectly using the relation to others, such as "Click the icon to the right of the search box."
3.5 TAPPABILITY PREDICTION
Lastly, we include a task related to automatic usability assessment. Whether a user perceives Ul object tappable is important usability issue. The mismatch between the tappability of an object as perceived by the user and its actual clickability has constantly plagued mobile user experiences. binary classification task based on the view hierarchy and the screen appearance of the element. In this work, to avoid introducing additional task heads, formulate tappability prediction as a yes-or-no QA task (Equation (5)
(Weeangin & Li (2019) proposed tappability prediction
A = F(S,V,T.Q)
which outputs "yes" when the perception of object is predicted as tappable, and "no" otherwise. Note that all the tasks share the Image-Structure model. Except the UI Object Detection task, they also share the Question-Answer model. We do not introduce special task token. Instead, Q is served as the natural language task indicator for signaling the model to produce different answers for each task. Q also carries the actual task specifies for the grounding task to find the object on the UI.
These five tasks are very different in nature. UI object detection only takes image input and generates UI objects. Command grounding leverages all the input modalities and outputs an object reference. Tappability, UI summarization and widget captioning share the same input and output modalities but are fundamentally different. Tappability concerned with the usability of specific UI object. In contrast, UI summarization and widget captioning generate functional descriptions about UIs, with the former focusing on the entire screen and the latter concentrating on a specific object. These heterogeneous tasks pose challenges for multi-task learning.
4 MODEL ARCHITECTURE
As shown in Figure I the architecture of VUT includes two Transformer models, i.c., the Image- Structure model and the Question-Answer model, and three output heads for three types of responses that accomplish 5 tasks.
4.1 SAGE-STRUCTURE TRANSFORMER
The Inaue-Structure Transformer is a two-modal model, which takes an image and the corresponding view hierarchy, and outputs the hidden representation of the image and each node in the view hierarchy. For the image modality, we compute the content embedding, C.,, and its positional encoding, P.. according DETR (Carion et al. 2020): C, Dense (Reseet(())))) and P. PE(Smaek). Smack is the binary non-padding mask of the image S. We omit details such as tensor reshaping and broadcasting in our discussions here. C2 E M and P. E RRM MxD where M the flattened size of the feature map after ResNet and D is the dimension of the representation.
For the view hierarchy modality, when V absent in the object detection task (Section 3.1 the content embedding for the modality, C.. is all zeros, and the positional encoding, P.. is a learned embedding vector for each query position. When the view hierarchy is present the input, each object in the view hierarchy tree, V. is embedded based on the set of attributes it possesses. The discrete attributes such as type, clickability, and text content are embedded separately to the same dimension and then combined via addition to form the content embedding of each element, C.- When there are multiple tokens in text content, max pooling is used to acquire a single fixed-size representation of text content.
The positional encoding of each object, P... is computed based on its bounding box [top, left, right, bottom] and DOM positions (pre-order idx, post-order idx, depth]. Each is treated continuous
5

---

Under review as a conference paper at ICLR 2022
vector position that is encoded via learnable Fourier representation (Li et al. 2021b) and then summed together to form the positional encoding of the object. When not present in the input, the position encoding is simply a learned embedding for each index position in the input. The final embedding of the view hierarchy modality includes two tensors: C E ND and P. E NRD
P. and P. positional encoding within each modality. Because the embeddings from the two modalities will jointly participate in the self attention of the Transformer encoder, important make their positional encoding global instead of local each modality. To this end, add learnable modal embedding each of these modality positional encoding: P. Pa E, and P. P. + E,, where Eg E RIxD and E, E RIID are the learnable embeddings for the image and view hierarchy modality respectively.
passed to the model so that it can pay more attention to the object. To achieve this unique need of UI tasks, we modulate the content embedding of both modalities, C2 and C2 using a focus map as produced from T as follows.
When the task with respect to a specific object on the screen (Section 3.2 and 3.5)
0g = SotmaxxFFaateee(eegiooMssst + T)M
(6)
C aa C2
RegionMask(-) creates 2D binary mask from the bounding box of the target object Toke where is all ones inside the the box and all zeros outside. Flatten(-) flattens the 2D mask to ID. 3 and are the hyperparameters that regulate how much the model should focus on the target object versus the rest of the image. @ denotes element-wise multiplication. We perform a similar modulation for the target object V.
G₁ Softmax(OneHo((Taa)) + T)N C² = at @ C
OneHot(-) produces one-hot vector from the index of the target object, Tates in the view hierarchy. We then concatenate the embeddings of the two modalities along the first dimension to form the input the Transformer encoder: C Concat C.] and P Concat| P P.). where C' E RMM 4M4-M N) and P (M are the final content embedding and positional encoding respectively, which fed multi-layer Transformer encoder: H Transformer. Btarmmerre__noee(CC.. P), where the hidden representation H IR (M We then split H for the hidden representation for each modality: H. H|: M] and H, = H[M :] while result in the hidden representations for each modality: H. MMD and H, NND
4.2 QUESTION-ANSWER TRANSFORMER
The Question-Answed Transformer is language model that encodes the question Q and decodes the answer A and produces the hidden representation for the grounding task. The input of the model, X Ti:t. where is the length of the sequence, is either just the token sequence of Q for the grounding task (Section 3.4). or the concatenation of Q and the decoded answer A' when language answer is to be generated. During training with teaching forcing, A' = A. During auto-regressive inference, A' is the predicted token sequence up step.
Iı.t = Transformer DeneinnrrDeeooe(((((eeeea PE(1 t); H,, H,,HH H.)
(8)
where is the ith token in the sequence (1 < < t), and E(-) and PE(-) compute the content embedding and the positional encoding of each token the sequence. H, and H, are accessed via Transformer encoder-decods attention. The sequence of hidden states, 91:t. 9t E R,,, are used for predicting the next token for generating an answer or for retrieving a target UI object in the view hierarchy for the grounding task.

---

Under review as a conference paper at ICLR 2022
4.3 OUTPUT HEADS
Object Detection Head: For the UI Object Detection task (Section 3.1 we borrow the object output layer from DETR (Carion al. 2020), using H2 as the input the layer: Yeppe He teppe and Yike cHH,. Oota Wibe bbe where W type E R is the linear projection
output the object type logits. An additional P PADDING type is included on top of the original 21 UI object classes. Φ(-) the multi-layer perceptron parametized by Oske and W bbu E Nx is the linear projection for generating the coordinates. The logits Y Yeppe E N and the coordinates Yboz E R x4 are for both generating object predictions and computing optimal compound loss using Hungarian Matching during training Carion et al. (2020).
Text Head: For the three tasks that require a text response, A, (Section 3.3 and 3.5 we apply a softmax layer on top of the decoder hidden state, I1.t (Equation 8). to generate each answer token.
0, arg Arr.aaxa)(fttnna(gg:==:)aa)))))
where a, is the ith token in the answer sequence A, and [Q] is the length of the question. Wat E Dx (vocab) is the learnable weights and |vocab| is the vocabulary size. For the three tasks, we optimize the model for the cross cross-entropy loss over the predicted and ground-trute answer token sequences.
For the grounding task (Section 3.4). we use the last hidden state from the Transformer decoder as "pointer" |Vinyals et al.) (2015) match against all the objects in the UI based on their hidden representations, using dot product similarity (Equation TOp.
Pointer Head:
i= arg max (ax((oftmmxx(11)) W Wtr h2) I5JEN
(10)
where h is the jth row in Hy that is the hidden representation of the jth object in the view hierar- chy. Wf,, Wpr E R DxD is the learnable projection and 9iQ1 is the last hidden state from the decoder (Equation 8), which is able to access the entire question (command) sequence, Q, via the decoder self attention. Compared to previous UI ground work (Li et al. 2020a), we used the last hidden state as the "pointer" instead of embedding pooling of bag of word in span. We optimize the model by minimizing the cross-entropy loss between the predicted and the ground-truth object index.
5 EXPERIMENTS
5.1
DATASETS
For the UI Object Detection task, we use RICO, a public corpus of mobile user interfaces (Deka et al., 2017) that contains 64,462 unique Android screens from 9,362 different apps. Each screen comes with an RGB screenshot and a corresponding view hierarchy. A view hierarchy is tree structure of nodes with 21 unique types, which we consolidated from the Android View class attributes in the original dataset (see Appendix D) Plus the special P PADDING type, there are 22 types to be predicted. view hierarchy has a maximum of 128 nodes in our dataset.
For the Widget Captioning task, we used a public dataset (Li et al. 2020b), which includes more than 200k human annotations for over 46k unique UI objects from 17k RICO screens. For Screen Summarization, we used public dataset (Wang et al. (021) that consists of 1 112k human created summaries for 22,301 unique Android screens from RICO.
We created the Language Grounding dataset, which has 10k human annotations for operating UI objects of 1432 unique screens from 26 Android build-in apps like Settings. human rater generates commands such as "Click the button below battery info", and the maximum length of command phrase is 20 words. The Tappability Prediction dataset includes tappability annotations for more than 18,669 UI elements from 3,218 Android screens. In the data collection, given target UI element highlighted on a screen, 5 different human raters are asked to answer yes or no for whether the target object looks tappable to them. We use the majority voting to determine the label of each element.
We split each dataset into training, validation and test sets (Table 2 and ensure there is no overlap of apps (or screens) between training set and any of the test of different tasks. This is important
7

---

Under review as a conference paper at ICLR 2022
Dataset
Train
Validation
Test
Ohject Detection (Deaa et al. 2017)
54,611
2,518
2,627
Widget Captioning (I.i ct al. (2200c)
39,951
3,436
3,531
Screen Summarization (Wang al. 2021
17,569
2,298
2,434
Language Command Grounding
7,822
1,024
987
Tappability Prediction
14,783
1,854
2.029
Table 2: Datasets.
because in the multi-task learning condition, VUT learns from all the training sets. Thus the union of apps and screens across all the training sets should not overlap any of the test set. We also ensure our test datasets are the same with the released benchmarks for widget captioning and screen summariztion so that the results are comparable with reported SOTAs. We unify these heterogeneous datasets follow the same feature taxonomies based on problem formulation that they can be consumed by the same model.
5.2
RESULTS
5.2.1
COMPARING VUT WITH DETR & CENTERNET FOR OBJECT DETECTION
In this experiment, we compare VUT with two benchmark models for UI Object Detection. The standard DETR architecture (Carion al. 2020) uses a 6-layer Transformer encoder and 6-layer Transformer decoder. To have similar number of parameters in the model, we let VUT Image- Structure transformer to use a 12-layer Transformer encoder this experiment. CenterNet (Duan) al. 2019) has been popular choice for object detection, which achieved SOTA results on natural images. this experiment, we let CenterNet use ResNct-101 as its backbone reach similar shows that VUT's Image-Structure model clearly outperforms DETR and CenterNet on the UI Object Detection dataset. Previously, Carion et al. (2020) found more encoding layers lead better accuracy. It is worth noting that VUT's Image-Structioe model uses encoder-only architecture that achieves better accuracy.
parameter size. Table shows
Model
#Params
AP
AP
AP7s
AP smaall
AP medium
APtarge
CenterNet
49M
31.9
44.3
33.0
1.2
12.6
33.0
DETRe+s
50M
37.8
49.1
39.6
1.8
21.1
38.1
VUT12
48M
39.3
50.1
40.9
3.3
21.6
39.6
Table 3: Comparison of three models for UI Object Detection on the validation dataset.
Note that these results not be compared directly with those previously reported Zhang al. (202T). Our task more challenging for detecting 21 different object types including several container elements, instead of 12 objects in previous work. In addition, previous work used dataset that is manually labeled by human and also employed heavy post-process to improve the accuracy.
5.2.2 COMPARING MULTI-TASK WITH SINGLE TASK LEARNING
For each task, we report its accuracy when it is learned alone and with other tasks jointly. We also show the benchmark results on these datasets reported by previous works for comparison, when they are available. In this experiment, we use a 6-layer Image-Structur Structure model and 6-layer Question- Answer model. Each task head and model parts are used only for batches that are specific for each task during training. For Widget Captioning, Screen Summarization and Tappability, text head (Equation 9 is used for generating answers. For Language Command Grounding, the grounding head (Equation 10) is used instead. See Appendix B.I for training schedule details.
As shown in Table 5 and 7 multi-task learning, though more challenging than single-task learning, can often perform on par with single-task learning. It consistently outperforms single-task learning for most configurations and metrics. There is a decrease of accuracy for the Grounding task when text-gent eext-generation related tasks are involved. We suspect that the grounding task, which relies the last hidden state the Question-Answw model, probably competes with the three text- eext-generttion tasks by "pulling" the hidden representations of the Question-Answa model towards different
8

---

Under review as a conference paper at ICLR 2022
Configurations SOTA, Li et al. 2020c)
BLEU-1
BLEU-2
BLEU-3
BLEU-4
ROUGE
CIDEr
44.9
32.2
44.7
97.0
Widget Captioning alone
45.8
30.2
19.6
12.9
46.0
94.8
Widget Caption + Object Detection
46.7
31,6
21.9
15.0
45,9
98.3
4 tasks (without Object Detection)
43.3
28.5
18.7
14.0
44.0
88.9
All tasks
32.3
22.7
46.8
99.3
Table 4: Results for the Widget Captioning task
Configurations
BLEU-1
BLEU-2
BLEU-3
BLEU-4
ROUGE
CIDEr
SOTA (Wang et al. 2021) Screen Summanzzation alone
63.3
45.8
32.4
25.1
48.6
61.3
68.7
49.4
31.6
19.4
53.8
64.3
Summarization Object Detection
68.9
50.8
33,5
21.4
54.9
65.6
4 tasks (without Object Detection)
68.2
49.4
32.2
20.2
53.5
56.8
All tasks
67.7
49.2
32.1
20.1
53.9
65.1
Table 5: Results of the Screen Summarization task.
Configurations
Ground accuracy (%)
Command Grounding without image input
Command Grounding alone Command Grounding Object Detection
75.5
82.1
4 tasks (without Object Detection)
77.3
All tasks
80.8
Table 6: Results for the Language Command Grounding task.
Configurations
Precision (%)
Recall (%)
F1 (%)
Tappability alone Tappability + Object Detection 4 tasks (without Object Detection)
81.9
91.0
84.6
87.7
86.5
84.8
86.7
All 5 tasks
90.1
86.5
88.3
Table 7: Results for the Tappability task.
Configurations
AP
AP
50
AP75
Object Detection alone
37.0
47.6
38.8
Widget Captioning + Object Detection
36.6
47.8
38.5
Screen Summarization Object Detection
36.3
47.8
38.3
Command Grounding Object Detection
36.9
48.4
38.8
Tappability Prediction + Object Detection
37.6
48.4
39.5
All 5 tasks
35.2
46.8
36.8
Table Accuracy for the UI Object Detection task when different tasks are jointly learned.
directions. One phenomenon that we consistently observed is that having the Object Detection task in multi-task learning often outperforms the configuration without involving Object Detection. For the Object Detection task itself, we found there is drop of accuracy when batch-alteration for multi-task learning starts. Yet, mostly recovers its accuracy (see Table g). These experiments show that instead of treating Object Detection as standalone pretraining task, it is feasible for it to be part of the multi-task learning where VUT achieves all the tasks through a single model.
CONCLUSION
We present VUT, multi-modal Transformer for multi-task modeling of user interfaces. Our model takes in three types data, i.e., UI screenshot images, view hierarchy structures, and natural language questions. Our experiments based 5 datasets show that VUT achieves five types of UI tasks simultancously, and show the promise of providing unified modeling for the user interface domain.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Chongyang Bai, Xiaoxue Zang. Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Agüera y Arcas. Uibert: Learning generic multimodal representations for UI understanding. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI 2021, Virtual Event Montreal, Canada, 19-27 August 2021, pp. 1705-1712. ijcai.org, 2021. doi: /1996////aai.2221/2335 URL https: //cp::://do... org 211.:/....449/9/1 2021/235
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL CC/ /t///ppe//2020/11ee11//666666363 2ttp:///00......0..110/S11122222
neurips
96 1ww.16/11118811111111111111
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, 2020.
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST 17. pp. 845-854 New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781 450349819 19. doi: 10.1145/31 05553331665444116666
htRp:htps:///ot .rg/
10. 45 26594.69... 316651
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 2019, doi: 10.1 10.1109/CCV.2019900066 URL https: :/tp:://dooo.. / 111.11.1111//CCCC1 2019 00667
IEEE,
Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems. CoRR, abs/2104.00743, 2021. URL /
00743
Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby B. Lee, and Jindong Chen. Actionbert: Leveraging user actions for semantic understanding of user interfaces. CoRR, abs/2012. 2020. UtLlattss://axiit/.
12350
Ronghang Hu and Amanpreet Singh. Transformer's is all you need: Multimodal multitask learning with unified transformer. CoRR, Abs/2202.10072. 2021. URL htt.:/tttt///trtt wtg.aii.oorr........//a/../bo
2102.10777
Ronghang Hu, Amanpreet Singh, Trevor Darrell, and Mareus Rohrbach. Iterative answer prediction with mhinter-ugmenter multimodal transformers for textvqa. In 2020 JEEE/CVF Conference on
Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp.
9989-9999. Computer Vision Foundation / IEEE, 2020. doi: 102100999924200022222222 https://openaccest
URL
thec com/ wwwpm///onetttttoeeeeeeeee CVPR 2020/html
Iterative wwwpsaeeeeeeeeettiaoeoe Betiiiioo__iittttt Po Pnnneeerreeeee Augmented Mult Muldimodal.
Transformers_ wot_ffffoo___e____e_______ CVP.
CB____CVR_______ 2020 2019200_pperhhtmm.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and--anguaage transformer without convo- lution or region supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp.
PMLR, 2021. URL
ngs mm www.rrp.......... /t3./v13///// m2 html
10

---

Under review as a conference paper at ICLR 2022
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. In Arxiv, 2019.
Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. Screen2vec: Semantic embedding of gui screens and gui components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY. USA, 2021a. Association for Computing Machinery. ISBN 9781450380966. doi: 10.1145/3411764.444550000 1tt./////...11113111611611111
URL
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 8198 8210, Online, July 2020a. Association for Computational Linguistics. doi: 10.
URL https://www
aclweb. wwwprr/aannooooo////222222. acl main. 729
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. Proceedings of the
2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5495-55911151 55100, Online, November 2020b. Association for Computational Linguistics. doi: 10. 18653/v1/2020.
Monlp-main.445 URL httph: /twp:///wwww..//.o
www.waclwwwtn.. aclweb. wwrggannhoooggy////
emn 1p main
443
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning:
Generating natural language description for mobile user interface elements, 2020c.
Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. Learnable fourier features for multi- dimensional spatial positional encoding. CoRR, abs/2100.00295, 2021b. httts:/:/:aaa
org/abs/210s 02795
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-ane ussion-andlaan and-language tasks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dÁlché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httpps::/prccceeeiiiiiiee/ htt.://aeaeaa.eaaaaaaaaaeaeaaaa2
wwwrrerpp... cc/ pdf
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Panupong Pasupat, Tian-Shun Jiang, Evan Zheran Liu, Kelvin Guu, and Percy Liang. Mapping natural language commands to web elements. In Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun ichi Tsujii (eds.). Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 4970 4976. Association for Computational Linguistics, 2018. URL httph:ttt/.......... //ac antho 1 logy- org DD1 8- 1540/
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs//910..ti.. 1910.10683, 2019, htLpttp: //axxiv/ orq/ abs tabbs/1990 191011911100 100.111..111111111
Amanda Swearngin and Yang Li. Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning. pp. 1-11. Association for Computing Machinery, New York, NY, USA, 2019. ISBN 9781450359702. URL htttp:://doi..rg// 1tt.://.....15/3229655
330C305
Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP). pp. 5 1100-111,,10 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/D19-111 D19-1514]
URL https:: //wp::://www.... ac wwb....eb..... orge:/.orr//aao htrg/anthhoogyy
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015. htopo///oooottttt///eeeeeeeeede
www.iiss.euuipssp cC/

---

Under review as a conference paper at ICLR 2022
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile UI summarization with multimodal learning. 21 2021. URL https?
ataxiiv.org/ abs/ / 2108 03353
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2048-2057, Lille, France, 07-09 Jul 2015. PMLR. URL xuc15.html
ml m.l.... pwess/v33.
Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey Bigham. Screen recognition:
Creating accessibility metadata for mobile applications from pixels. In Proceedings of the 202 CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 9781450380966. doi:
174434441844
httthttps: / /oi (adii.ooooo wwt. 100 1ee.11111111111 3445186
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu. Jason J. Corso, and Jianfeng Gao. Unified vision- lsionnlanguage pre- veeetraining for image captioning and VQA. In The Thirty-Foure Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Sheroy-Second Innovative Applications of Artificial Intel- ligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 13041-13049, AAAI Press, 2020. URL httttt://eaatt //tp://aaaaaaaa///a//aaa org ojjs index. http:ppppaaatt.aeeeeeeaeaaaaaaaa
A QUESTIONS
As discussed in the paper, there are three types of question input, Q, fed into the Question-Answer model: commands, questions for yes-or-no responses, and questions for open-ended answers. When VUT is joint-learning all the tasks, it is required to recognize all the questions for different types of tasks. For questions that expect textual response, e.g., Widget Captioning, Screen Summarization, and Tappability Prediction, VUT also needs to generate answer corresponding to each type of question.
A.1 COMMANDS
For the Language Command Grounding task, commands that refer specific object in the screen are fed to the model by which the model is trained to locate the referred object. Example commands shown below. These commands are created by human annotators for a target UI object shown on a screen. A human annotator is asked to come with different commands referring to each highlighted target object.
Click on the notification bar above the status option.
Press the back arrow button.
Select the icon above the clock option.
Swipe down the notification bar.
A.2 QUESTIONS FOR YES-OR-NO ANSWERS
For the Tappability Prediction task, we generate synthetic Yes-or-No questions based on the following regular expression pattern. The model is trained to decode yes the to the question.
Is the [object lel ement wwiget control] [cl [cli cicbble e tappable]
The question examples that are generated based on the regular expression are the following.
Is the object tappable?
Is the widget clickable?
Is the element tappable?
12

---

Under review as a conference paper at ICLR 2022
A.3 QUESTIONS FOR OPEN- ANSWERS
For Screen Summarization summary and captioning tasks, the model will need to generate an open- ended answer. We use the following regular expressions to generate questions for these tasks. VUT is trained to decode screen summary or widget caption following the question.
What is the [summary description) of the [screen UI]
What is the [caption ion description) of the [object element widget control]?
Below are the question examples generated from the above regular expressions.
What is the summary of the screen?
What is the description of the UI?
What is the caption of the widget?
What is the description of the object?
B MODEL & TRAINING DETAILS
For the comparison with DETR and CenterNet on the UI Object Detection task, VUT uses 12-layer Transformer encoder as the Inrue-Structure model that amounts 48M trainable parameters, which is fewer than 50M trainable parameters of DETR with 6-layer encoder and 6-layer decoder, and 49M parameters of CenterNet. Image Embedder (Figure [ the ResNet backbone. For the remaining experiments, VUT uses 6-layer Transformer encoder for the Image-Structure model, and a 6-layer Transformer decoder for the Question-Answer model. When all the tasks are jointly trained, there 64M parameters. Task Fask-specific heads and word piece embeddings and projections the main contributors to the growth of the parameter size. When only a subset of these tasks involved in the training, e-g-, Widget Captioning Object Detection, there will be fewer trainable parameters involved because only part of the full model in use. All the VUT variants use the following configurations: #Attention Heads=8, Hidden_ Dimension=226, Transformer MLP Dimension=2048, Transformer QKV Dimension=256.
Note that in our Image-Struue Strgct-t model, positional encoding added the input of each layer of the Transformer: This is in contrast to our Question-Answer model where the positional encoding of each token is only added the input of the first layer. We use the learned embedding for positional encoding in the Question-Ami Answerr model. During training, we use 10% for both the attention and the MLP dropout the Anesseron-Ansswrr Answer Transformer, and we also apply 10% dropout on the encodings from the -ngge-Structure model before cross attention. During the 5-task joint learning, the attention and the MLP dropout rates are 10% for the Image-Structure Transformer and the encoder-decoder cross-altention dropout rate is 20%. During auto-regressive decoding for interference, the maximum decoding length is 30 that covers the total length of a question and an answer.
We use the same method BERT for tokenizing phrases into sequence of word pieces which results in a vocabulary size of 28,536. The maximum size for a screenshot image is 1080 x 1080. We randomly resize each image for image augmentation. The maximum number of UI objects and containers on each screen capped to 128. We implement VUT based JAX library for machine learning. We train each VUT model with batch size of 64 screens/examples, which the training is parallelized across 64 TPU v3 cores.
B. TRAINING SCHEDULE DETAILS
Because the UI Object Detection task requires many more iterations to converge than other tasks, we start our multi-task learning by training VUT for the UI Object Detection task, and then training VUT jointly for multiple tasks by alternating batches from each dataset. This learning strategy is sensible because by learning from the UI Object task, the model can learn useful information about
token zer
13

---

Under review as a conference paper at ICLR 2022
how to encode the screen pixels. As it is consistently shown in our experiments, joint learning that involves Object Detection can often boost the learning of the other four tasks.
We start the training by learning the Object Detection task that takes 300k steps, using DETR's default training schedule (Carion et al., 2020). For joint-learning of all the tasks, we sample batches from the Object Detection, Widget Captioning. Scren Summarization, Command Grounding and Tappability datasets with the weights [15, 10, 10, 20, 1] and the model trained with 100k steps. The learning rate le-4 and decayed to le-5 after 50k steps.
For individual task training, we train each model until it converges with a batch size of 64. For the UI Object Detection task, we train the model with the most default setup of DETR with 6-layer encoder and 6-layer decoder and 8-head attention and 256 hidden size, using 300k steps. The learning rate schedule includes one learning rate decay from le-4 to le-5 at the 200k steps. The Widget Captioning task uses 6-layer Image-Struce Structure model and 6-layer Question-Answer model plus the Text head. Similarly, the model was trained 45k steps converge. For the Screen Summarization task, we used the same model configuration as Widget Captioning, and the model converges at 50k steps. The Language Command Grounding task uses a similar model setup as the Widget Captioning and the Screen Summarization tasks, except that it uses the Grounding Head instead of the Text Head. It took the model 27k steps to converge. For training the model for each of these tasks, we decay the learning rate once from le-4 to le-5 at 15k steps. For learning the Tappability Prediction task alone, we used the same model setup as the two text related tasks (Summarization and Captioning). We found the model is very prone overfitting in spite of using large dropout rate. So we train the model with early stopping.
For Object Detection another individual task, we sampled batches from the two datasets with weights similar to the ones used for 5 task joint training. For Widget Captioning, Screen Summary and Grounding, the models are trained converge with 35k, 50k and 50k steps, with lc-4 initial learning rate which decays to le-5 at 25k, 15k and 25k steps, respectively. For the Tappability task, we use a initial learning rate of le-5 and train the model with early stopping.
For 4 task joint training (without Object Detection), we sampled batches from the 4 tasks (Widget Captioning, Screen Summary, Grounding, Tappability) with the weights [5, 5, 10, 1]. We trained the model with 150k steps and batch size 64. The learning rate schedule includes one learning rate decay from le-4 to le-5 at the 50k steps.
550956
506694
406673
314164
287983
272033
Leaf
Inner
Count (log10)
57517
52663
20038
18233
16275
15399
14045
11002
9624
8050
6083
3714
3084
5264
1862
2
LINARRLYOUUT
TEXVVEW
FRAMELAR AYOUT
VEW
MAGGEVE
RELATVVELAOOUT
MAGEBUTTON
BUTTON
WESVEN
CDITERY
SCROLVIER
CHECKEDTERTVEEE
MEWSWITCHER
HORIZONTALSC SRIZONTALSCROLVVEE SCROLUEE
CHECKBOX
RADIOBUITON
PROGRESSAA
SPINERR
TOGGLEBUN RECOLEBUTTON
SEERDAR
Figure 2: The distribution of the 21 UI elements the logio scale. The proportion that each type is used as an inner versus a leaf node is shown within each bar.
C ABLATION STUDY ON FOCUS OBJECT
For Widget Captioning and Tappability task, VUT predicts the caption or the tappability answer of a given object on the screen- tth focus object. To inform the Image-Structure Transformer the location of the given object, we apply focus map the Image Embeder outputs and the embeddings
14

---

Under review as a conference paper at ICLR 2022
of the view hierarchy objects so that the focus object has more weight on its pixel area and object embedding than the others (see Equationnaand D. Specifically, we used 3 2.0 and T -1.0 in our experiments. To show the effectiveness of the tocus modulation, the Widget Captioning model with three settings: 1) applying the focus map to both the image and structure modalities, 2) applying the focus map the structure modality only, and 3) concatenating an embedding of {0, } to the object embedding for the structure modality with I indicating the focus object.
conduct ablation study using
As shown in Table 9 the model performs the best when focus map is used on both the image and the structure modalities, indicating the necessity and effectiveness to inform the model the focus object location in the pixels and the object list. Using the focus map only on the object embedding is not sufficient because the Strre-Str model needs to learn from the data to find the corresponding pixel areas of the focus object. Yet, using (0, 1) embedding is the least effective.
Configurations
BLEU-1
BLEU-2
ROUGE
CIDEr
{0, 1} embedding the structure modality
39.7
26.7
38.6
80.7
Focus map on the structure modality only Focus map on both the image and structure modality
41.0
27.3
39.8
84.9
46.4
31.4
45.1
98.5
Table Ablation Study Results for Focus Weight
D UI OBJECT TYPES
We process the RICO dataset for the UI Object Detection task. As discussed the paper, among the 64,462 screens of the original dataset, we particularly use those verified by human raters for validation and test datasets. For each element on the screen, we extract its attributes such as its UI object type, its bounding box position the screen, whether it is clickable enabled. We exclude all the elements that are marked as invisible as they have no correspondence with pixels on the screen. There are many custom object types in the dataset and many of them inherit from common Android widget type We consolidate rare object types (such as TABWIDGET or VIDEOVIEW) to their closest ancestor type that are in the common widget library. With the consolidation, there are 21 UI object types in the dataset (Figure 2 which has long tail distribution. As we can see from the distribution, some elements are dedicated as leaves in a view hierarchy, e.g., ImageButton or RadioButtion, and some elements are primarily for determining the layout or as non-terminal nodes, e.g.. Linear Layout There are cases that a type is used for both leaves and non-leaves, c.g., View. The View type often catches UI elements that cannot be easily classified into specific type. Together, these make UI Object Detection a challenging task.
E MODEL SIZE AND INFERENCE TIME COMPARISON
In this section, we compare model size and inference time of the 5 tasks when they are learned jointly and individually. We can see the joint model has the largest parameter size, includes all the task heads. However, the size the 5-task joint model substantially smaller than the total size of each individual single-task model (Table 1OP. The inference time of the joint model for each task is similar that of the task-specific model (T). as only the part that is required by the task is activated in the joint model during inference. Altogether. VUT can achieve the five tasks with little inference time overhead and substantially small model footprint.
F EXAMPLES OF PREDICTION RESULTS
We here show examples of predictions versus ground-truth for each task, on the test data, as achieved by a single model of VUT, when it learns all the tasks jointly.
15

---

Under review as a conference paper at ICLR 2022
Ui Diod, Strbudo, Wil..
Diod Bebudd Wi..
Conact you User Potfle the Fit Braine botween denies and othe Brains apps.
Conect your your Profle the Rit Braind Jchwenn deties and other Brains apps
Sitrr al locadon..
al a1003001
PANL: AUSERNAME
Becert Seacchen
Recer Seardwes
Stanfiyy
Ficcettre
Facceiine
SUBHIT
SUBMIT
Locgpig
Lodging
Eufflee
Eufflee
CONMACT AP ONO NEOKEE
CONNUCT NNDOO NEOKU
Liiccdos
Lricodes
Ground Truth
Prediction
Ground Truth
Prediction
INC
T 1955
C B 8i
1955
Nii
Crards
XDV
X0W
Ground Truth
Prediction
Ground Truth
Prediction
Figure 11: Examples for the Object Detection task. In the ground-truth screens, the bounding
boxes of inner objects are highlighted in orange and those of leaf objects are shown in blue. In the predictions, we render the bounding boxes of predicted objects as inner (orange) versus leaf (blue) based on the dominant of the predicted object type, according to Figure) 2
16

---

Under review as a conference paper at ICLR 2022
+ 1:32
0 3
253
Notifications
G
0
Google
All apps
aBc
support.muttcooeeo Dusi
android support.test orchestrator
Drake
API Demos
NBA reportedly Drake language
Bootstrap Service
improve your search esperlerce can change sop preferel bes in your desice's setüngs.
Google apo your
Calculator
Myps
Calendar
NBA Reportedly Warns Drake
Draki Was Reportal
Command: icon"
"tap on the voice
"press on the icon below calculator icon"
Command:
"select the weather text which below the notifica- tion bar"
Command:
Figure 15: Examples for the Language Command Grounding task. The object located by the model is highlighted with blue bounding box in each screenshot.
12:00
5:22
CALLERIDFAKER
Locations
Today's Top 100
Tikce
Lai
Crudla
CALLERIDFAKER
Setup
Premium
Denerl
Amabl Cara
NETJETS
Prediction: "search bur to search for the location" "
Prediction: "page displaying
"pop-up showing to create an account"
Prediction:
music track in music app"
Reference; "page displaying search box in the app"
Reference: "screen shows mu-
Reference: "pop-up displaying
sic playing on an app"
to setup the account details"
Figure 19; Examples for the Screen Summarization task. We here display one of the 5 references (ground-truth summaries) created by human annotators for each screen.
17

---

Under review as a conference paper at ICLR 2022
0 203
4
1:51
English
French
Helpy Gracn Chan
LOGDUT
Text
Thank You!
Name
Groce
Chan
HELLO!
4155791638
FOOOOD
bpprrwwerdggmallcomm
deuvre
profession
ouvrage
emploi
boulot
mission
métier
résultat
nnine
Prediction: "copy text"
Prediction:
"select the emoji"
Prediction: "enter password"
Reference:
"copy clipboard
Reference:
"select
guitar
Reference: "input confirm pass- word"
option"
lizard"
Figure 23: Examples for the Widget Captioning task. The target element is highlighted via a blue bounding box. We here show of the three references (ground-truth captions) created by human annotators for each target element.
BUY
Tell about yoursell
Customize
Name
INSTALLED
MIDSET
LEVE WALLIAFRR
All
Support
Emeil
Your e- wariiriggrrall.
Default dial code
Change peasword
Tearppane
Divelooud
Ranstam
Dowerod
Man
phone
Date binch
06 Der
version
Logout
Wadeet
WetherOock
Dowctoad
Autafil Fasebook
Prediction: "yes" Ground Truth: "yes"
Prediction: "no"
Prediction: "yes"
Ground Truth: "no"
Ground Truth: "yes"
Figure 27: Examples for the Tappability Prediction task. blue bounding box.
The questioned element highlighted with
18

---

Under review as a conference paper at ICLR 2022
Task
5-Task Joint Model
Single-Tase Task Model
Object Detection
63.6
39.5
Widget Captioning
63.6
56.2
Screen Summarization
63.6
56,2
Command Grounding
63.6
63.5
Tappability tasks
63.6
56.2
63.6
271.6
Table 10: Parameter size (millions) comparison between the 5-task joint model and single task model.
The 5-task joint model performs the five tasks simultancously, which otherwise require five separate models that amount to much larger model footprint.
Task
5-Task Joint Model
Single- Mangle-Task Model
Object Detection
12.21
11.59
Widget Captioning
30.02
30.00
Screen Summarization
43.68
42.97
Command Grounding
70.53
71.29
Tappability
51.55
51.09
Table 11: Inference time (ms) comparison between the 5-task joint model and single task model for each task. The time calculated by averaging the inference times on the There is little time overhead for the joint model to perform each task, compared to each task-specific model.
19


Reviews:
[{"review_id": "hNn9VfKmF9r", "reviewer": "Reviewer_e7Hg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ Formulated a multi-modal multi-task learning for graphical user interfaces\n+ A new two-tower Transformer architecture is designed\n\nWeakness\n- By jointly trained on 5 tasks, the model is on par or slight improves over models trained on individual data sets. This does not match the observation on other image-language data joint training where significant improvement is achieved.\n- The 5 tasks are closely related, and it is hard to see how the model can generalize to a different task, such as generating/editing new UI layout\n- The main architecture of the model is new, while all the building blocks are not. The technical contribution is somewhat lacking.", "summary_of_the_paper": "A multi-modal multi-task model is proposed for UI understanding. It has an Image-Structure Transformer and Question-Answer Transformer to encode/decode image, structure and language information. The method is evaluated thoroughly on 5 different data sets and achieves good results.", "summary_of_the_review": "The paper demonstrated an interesting problem addressed by a standard transformer based model. The potential of generalization to more tasks is not clear, and the improvement over single task models falls below expectation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7_iPaRPN95h", "reviewer": "Reviewer_j2ye", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In summary, their main works can be summarized as followed:\n(1) The authors formulate multi-modal multi-task learning for graphical user interfaces and design a VUT model.\n(2) The authors create a so-called Language Grounding dataset for language grounding task.\nThis topic is interesting and it is pioneer to incorporate muti-task learning in graphical user interfaces. The manuscript is easy to follow. And the technical contribution is significant. The experimental results conducted on 5 distinct UI tasks demonstrate its superior performance over some state-of-the-art methods.\n\n\nThe reviewer suggests borderline. The weakness of this paper is as follows.\n(1)\tTables 5,6,7 and 8 compare the multi-task learning with the single task counterpart. However, their results show that the multi-task learning cannot significantly facilitate the studied tasks. For example, in the Table 5, the setting of “Screen Summarization alone” performs better than the setting “all 5 tasks”. This is contrary to the original intention of this paper. More explanations should be given.\n(2)\tThere are some typos in this manuscript, e.g., “We experiment with VUT on 5 distinct UI task”. \n", "summary_of_the_paper": "This paper presents a multi-modal Transformer for multi-task modeling of user interfaces. It is potentially worthy to investigate how to incorporate multiple UI related tasks into a model. The authors hence design a so-called Versatile UI Transformer model which involves three modals inputs to handle five unique tasks. ", "summary_of_the_review": "I am a researcher interested in UI and published many paper of UI on top conferences.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1E-QONO6Og", "reviewer": "Reviewer_69JK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\n1. Overall, the formulation of the problem and various design choices (global positional encoding, focus map, pointer head for the grounding task) make sense. The authors also seem to have carefully designed their experiments. For instance, ensuring that comparisons are fair (same number of parameters, etc) for the object detection task.\n2. Given the nature of the problem statement (with multiple tasks, inputs and outputs), the authors have done a good job in explaining each of them properly. Overall, I thought the paper was easy to read and understand. \n3. The authors provide several reasonable insights that might be relevant to the user interface modeling community — using object detection as part of multi-task learning instead of standalone pre-training task, design choices to create a single unified architecture for all the tasks, multi-task learning outperforming single-task learning etc. \n\n**Weaknesses**\n\n1. I understand that a single model is helpful for multiple UI tasks, but I wonder if this approach is scalable beyond the 5 tasks and 5 modalities mentioned. The authors comment that the model architecture is designed to remain stable for a growing set of tasks, but this seems to be under the assumption that the input and output modalities would remain constant. A discussion on how to add new tasks to the same framework might help, or a discussion on why the current framework is enough.\n2. One of the important motivations of multi-modal multi-task learning mentioned was to achieve better or on-par performance with a single model (and supposedly fewer computations) which is crucial for devices with limited computing resources. But a direct head to head comparison for the computational cost of a multi-task model and individual models is not provided. Are there any overheads/disadvantages because of multi-task learning (Like a larger model size, inference time for individual tasks etc)? The model sizes are varied across experiments to achieve on par performance which makes the comparison of the computational cost not so obvious. Such a comparison would highlight the advantages of the multi-task model and would be helpful for the relevant audience.\n3. I also believe that the downstream tasks are also somewhat similar (language command grounding, tappability, UI object detection, UI summarization, widget captioning). It is therefore not surprising that multi-task learning should help these tasks. \n4. There is no comparison provided with any baseline for 2/5 tasks (Language grounding, Tappability)\n5. The novelty in the paper is somewhat lacking. The techniques used in the paper (multi-branch transformer, pointing mechanism, cross-modal attention, global positional encodings, etc) have been shown to work in the past for image-text tasks [1, 2]. The paper is taking all the lessons from past works and applying it to a new domain. \n\n**Clarifications:**\n\n1. The training procedure mentioned in section 5.2.2 talks about joint training but the procedure followed for training for individual tasks or a subset of tasks is not described in detail. Were the same hyperparameters used for all configurations? \n\n**Updates after rebuttal period** \nThe authors addressed some of the concerns -- showing inference time, model size and a discussion about training details and hyperparameters in the appendix. However, I am not convinced that the paper presents new insights that are relevant for the broader ICLR community. Concerns around novelty and the multi-task setup was also raised by another reviewer (e7Hg). Responses to reviewer e7Hg also aren't convincing. If the tasks are not similar, and the learning objectives are not aligned, then the motivation for multi-task learning is solely for reducing memory footprint and computational cost. I believe that this contribution isn't enough for me to recommend acceptance.   Thus, I am going to stick to my original rating. \n\n[1] R. Hu, A. Singh, T. Darrell, M. Rohrbach, *Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA*. in CVPR, 2020\n\n[2] J. Lu, V. Goswami, M. Rohrbach, D. Parikh, S. Lee, 12-in-1: Multi-Task Vision and Language Representation Learning. in CVPR, 2020", "summary_of_the_paper": "The paper proposes an architecture for graphical user interfaces which involve multi-modal inputs (UI screenshots, Hierarchy structures, Natural Language) and multi-task learning (UI Object Detection, Widget Captioning, Screen summarization, Language grounding, and Tappability).The proposed architecture consists of seperate transformer blocks to encode image and text modalities. The two transformer blocks attends to each other to produce multi-modal outputs, which is then used for downstream tasks. The authors that with the proposed architecture, training on all tasks simultaneously is better than training on individual task alone.", "summary_of_the_review": "Overall, the paper showed how multi-task learning can improve performance on several downstream UI modeling tasks. These insights might be relevant to the UI modeling community. The authors use several popular ideas in the field of multi-modal, multi-task learning and showed it's effectiveness on multiple downstream tasks. The paper is well written, and the ideas are clearly presented but it lacks in technical novelty. Additionally, the paper lacks in a few other areas -- comparison to single-task models with respect to computational cost, inference time etc, the downstream tasks are similar but the performance gains aren't significant. Due to these reasons, I think the paper is marginally below the acceptance threshold. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a neural architecture for tasks involving user interfaces. Tasks involve detecting objects on screen, writing captions about UI components, attribute recognition, etc. The reviewers for this submission found the proposed model to be reasonable and effective. They also found the paper to be well written and easy to understand. However, they did have one major concern, before and after rebuttal: While the model and design choices were reasonable, they questioned if the insights gained from this paper were of interest and relevance to the broader vision community. They also had other concerns/suggestions including adding inference costs and adding more detail, which were addressed in the rebuttal. Another concern was the fact that multi-task did not provide large gains over single task. I agree with the authors in this regard. I think the goal here was to produce a multi-task model that attained at least parity with respect to single task, because a single model would provide large benefits when running on a device, and hence I think this concern was well addressed.

My takeaway from the paper, reviews and discussion, however, continues to focus on the major concern of the reviewers. I think this paper would have benefited from answering at least one (if not more) of the following questions to the reader: (1) Why should the broader community work on this task ? (2) If this task is of limited interest, are there instead, aspects of this task that serve as a useful testbed for multimodal research ? (3) If the task and testbed are not directly applicable, are there new techniques developed in this paper that are broadly applicable to other problems or domains ?

Unfortunately, I think that this paper does not presently address either of these questions strongly to the reader. The paper proposes a method for their task, but readers who aren't directly interested in that end task may find this submission less interesting, in terms of insights for their own work. Given the above, I encourage the authors to address this concern and resubmit. I recommend rejection.
Decision: Reject