Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus
Authors: Gang Li
Yang Li
Venue: ICLR 2023 poster (2023)
Keywords: vision-language
UI
few-shot
finetuning
multi-task

Abstract: Mobile UI understanding is important for enabling various interaction tasks such as UI automation and accessibility. Previous mobile UI modeling often depends on the view hierarchy information of a screen, which directly provides the structural data of the UI, with the hope to bypass challenging tasks of visual modeling from screen pixels. However, view hierarchies are not always available, and are often corrupted with missing object descriptions or misaligned structure information. As a result, despite the use of view hierarchies could offer short-term gains, it may ultimately hinder the applicability and performance of the model. In this paper, we propose Spotlight, a vision-only approach for mobile UI understanding. Specifically, we enhance a vision-language model that only takes the screenshot of the UI and a region of interest on the screen---the focus---as the input. This general architecture of Spotlight is easily scalable and capable of performing a range of UI modeling tasks. Our experiments show that our model establishes SoTA results on several representative UI tasks and outperforms previous methods that use both screenshots and view hierarchies as inputs. Furthermore, we explore multi-task learning and few-shot prompting capacities of the proposed models, demonstrating promising results in the multi-task learning direction.

Published as a conference paper at ICLR 2023
SPOTLIGHT: MOBILE UI UNDERSTANDING USING VISION-LANGUAGE MODELS WITH A FOCUS
Gang Li Google Research Mountain View, CA baebird@googles com
Yang Li
Google Research Mountain View, CA 1 Liyang@googee.com com
ABSTRACT
Mobile UI understanding is important for enabling various interaction tasks such as automation and accessibility. Previous mobile UI modeling often depends the view hierarchy information of screen, which directly provides the struc- tural data of the UI, with the hope to bypass challenging tasks of visual modeling from screen pixels. However, view hierarchies are not always available, and are often corrupted with missing object descriptions or misaligned structure informa- tion. As result, despite the of view hierarchies could offer short-term gains, it may ultimately hinder the applicability and performance of the model. In this paper, propose Spotlight, a vision-only approach for mobile UI understanding. Specifically, we enhance a vision-languaal of the UI and a region of interest on the screen the focus-as the input. This gen- eral architecture of Spotlight is easily scalable and capable of performing a range of UI modeling tasks. Our experiments show that our model establishes SoTA results on several representative UI tasks and outperforms previous methods that use both screenshots and view hierarchies as inputs. Furthermore, we explore multi-task learning and few-shot prompting capacities of the proposed models, demonstrating promising results in the multi-task learning direction.
model that only takes the screenshot
INTRODUCTION
Computational understanding mobile user interfaces (UT) a crucial step for achieving intelligent UI behaviors such as UI automation, and addressing diverse interaction scenarios such as those requiring accessibility features. Recently, mobile UI understanding has attracted numerous research interests. Previous works have proposed various UI modeling tasks and datasets, including widget captioning (Li et al. 2020b), screen summarization (Wang et al., 2021), command grounding (Li let al.) 22200a] Bai et al. 2021 Burns al.) (022) and other tasks (Li al. 2022] He et al., (0200 on the mobile screen. Many of these works focus on bridging natural language and graphical user interfaces, which have shown potential for enabling language-based interaction.
mobile UI screen can come with a view hierarchy-a structural representation of the screen-ih addition to the screenshot image. Using view hierarchy input allows model to directly acquire detailed information of UI objects such as their types, text content and positions on the screen, by- passing challenging visual modeling tasks such as inferring object information from screenshots (Li et al. 202T Zhang et al.) 2021). Previous works have shown the benefit of using view hierarchy Ul modeling in several tasks. For example, models using view hierarchy have achieved better performance than their vision-only counterparts in UI captioning tasks (Li al. 2020b Wang et al.,
2021.
However, recent work has revealed that mobile UI view hierarchies often contain inaccurate in- formation about the UI screen, e.g., missing object text and misaligned structure information. Li et al. (2022) showed that about 37.4% of the screen view hierarchies contain objects with invalid bounding boxes. Ross et al. (018)) showed that 92.0% of Floating Action Buttons had missing text labels, compared 54.7% of Image Buttons and 86.3% of Clickable Images. These object text la- bels (c.g., content desc) are among the most important features in view hierarchies. Removing text features resulted in drop by 17 CiDER points for the widget captioning task (Li al. 2020b).

---

Published as a conference paper at ICLR 2023
Therefore, such inaccurate information in the input can seriously hinder models in realizing their full potential in UI modeling. Although recent work has proposed methods for repairing view hierar- chies (Li et al. 2022), substantial effort is still needed to robustly denoise raw view hierarchies. On top of these, Ul screen data does not always have view hierarchies available in them, such as mobile UI images crawled from the web. Fetching view hierarchy at runtime in a mobile environment also imposes additional system constraints for the applicability of models that rely on view hierarchies.
In this paper, we investigate the direction of using only visual UI screenshots as input (i.e., without including view hierarchies) for UI modeling tasks. We observe that many UI modeling tasks essen- tially aim to learn a mapping between the UI objects and text. As result, vision- language models, a class of models that encode visual (and language) modalities and decode text answers, become a natural choice for the model architecture. Although previous works show that vision-only mod- els generally perform worse than the models using both visual and view hierarchy input (Li et al.) 2020b] WWang et al., (02T), we believe that visual language models offer two unique opportunities: 1) the simple architecture enables model easily scalable, and 2) many heterogeneous tasks can be universally represented by the two core modalities of vision and language. These advantages have been evidenced by the recent successes of the vision-language models (Chen et al. 2022 Alayrac ct al. 2022 Yu et al. 2022; Wang et al. 2022).
tasks in the general domain, which usually use entire image as input, UI modeling tasks are often concerned with a specific object or area on the screen. This requires a vision-language model to be able to focus the object
In contrast to previous visual-languee
of interest. Thus, we propose S which enhances a vision-language model to generate text responses with respect to focus object or region to support various UI modeling tasks (see Figure Dp. In our experiments, we initialize Spotlight by leveraging pretrained large ViT (Dosovitskiy et al. (022] and T5 (Raffel ct al. 2019) checkpoints. We then pretrain Spotlight with unlabeled datasets consisting of about 2.5 million mobile UI screens and 80 million web pages, which is followed by one of the three modeling strategies: single-task finetuning, multi-task finetuning or few-shot learning.
Our main contribution is three-fold. First, we propose a novel vision-language model architecture that capable of finetuning, multi-task learning and few-shot learning for mobile UI tasks. The model can easily scale and generalize to other tasks without architectural changes. This model advances the art of UI understanding without needing to use view hierarchies as inputs that has many drawbacks in practice. Secondly, we develop method for creating large-scale pretraining datasets from automatically collected mobile screens and web pages. These pretraining datasets and methods are crucial for vision-language model learn the prior knowledge of the unique domain of mobile screens and Uls. Finally, we conduct extensive experiments over the proposed model, including using various focus region representations and modeling strategies. Our experiments show that the proposed models obtain new SoTA performance in both single-task and multi-task finetuning for the four tasks, including widget captioning, screen summarization, command grounding and tappability prediction. We also examine the feasibility of using the proposed model for few-shot prompting.
2 RELATED WORK
UI modeling problems have drawn widespread interest from researchers in both the MI. and HCI fields (Li et al. 22020b: Wang et al. 202T; [Li et al. 2020a Burns et al.) 2022; Bai et al.) 2021 Zhang ct al. 1 Wu et al. 2021 He et al. 2020). With the overarching goal of enabling intelligent UIs and addressing mobile accessibility, previous works have proposed rich set of mobile UI modeling tasks, along with datasets and benchmarks. Widget captioning (Li et al. 2020b) aims to generate natural language description for UI objects the screen. The capability can enable accessibility features such as the TalkBack screen reader to improve user experience for vision-impaired users. Screen2Words expands UI captioning by proposing the task for summarizing the entire screen (Wang) et al. 2021). Command grounding maps natural language command to a UI object on the screen, via single or multi-step interactions (Li et al. [2020a] Burns al., [022 Bai et al.) (02TT). Tappability prediction predicts whether a UI object is tappable when perceived by human (Swearngin & Li, 2019 Schoop et al., 2022). which is useful for UI design validation. Most of these previous works
'The name draws the analogy that spotlight illuminates a target region.
https: htttpp::/supppoe http:/ttpppptttr/////e http:///pppppptogoooee.eo http:p/wwww.......ccccccccccaaaa
=en
2

---

Published as a conference paper at ICLR 2023
Screen Rag on Encoder
Orpa
Focus Region
Extracter
Model
T5 Transformer Decoder
MT
sonweoccc
Input
moarpple regien
proreet
region
Exarplet
More corr plic
Pronst
Example screen region
Ecample
Ecample ooreen region
Examplo text
Prompt screen mejion
Prompt text
Output
Widget Captioning
Delace the syspttet
Open DIECU
Show pazawood
Pige duslayis categores
Summarizzatin
kocefon suspping
abscnete rifooes
shoppingg
Mobile UI Tasks
Command Grounding
Select
oiti en the aecond
Mos
Mo
bothee swcend
Yee
O
1
Tappability Prediction
Tappatlle
Tappadde
Ye
Hecslale
Figure 1: An illustration of the model architecture and UI task examples. VIT (Dosovisskiy et al, 2021) and T5 (Raffel et al. 2019) are initialized with checkpoints pretrained in the general domain of natural image and language.
used both the screenshot and the view hierarchy of screen as input in their models, and revealed that multimodal input achieves better performance than vision-only variants (Li et al.; 2020b Wang
ct al. 2021). Prior works have also borrowed language modeling techniques such as BERT (Devlin
al. 2019 for representation learning to facilitate downstream tasks (Bai et al. 202 He et al.)
2020).
Recently, Li et al. (2021) investigated the feasibility multi-task modeling for achieving range of UI tasks simultancously. To combat the lack of accurate view hierarchy, prior works have investigated model- based approaches for repairing noisy view hierarchies (Li et al. (002), or UI object detection sereenshots to derive information such object attributes and structure
3

---

Published as a conference paper at ICLR 2023
information (Zhang al. 2021 Wu et al. 2021). In this work, investigate rision-input-on approaches for several representative UI modeling tasks that have been established by prior works, including widget captioning, screen summarization, command grounding and tappability prediction.
Many of these mobile UI tasks are concerned with specific objects or areas on the UI screen. Pre- vious works have employed different methods to allow model to focus on object or region of interest on the screen. [i et al. (2020b); Wang et al. 2202T); Bai et al. (2021) used the screenshot pixels cropped for individual objects as input the model. Li al. (2021) I used focus map to bias the ResNet outputs to pay more attention to the pixels corresponding to an object. [Schoop et al.) (2022) added a fourth channel to an RGB screenshot image that indicates the target region with bi- nary mask where the Is correspond to the area of the object. The cropping and blas-based approach might result in losing the context of the object, despite providing direct localization of the object. While adding a fourth channel also keeps the information of the entire image, it is incompatible with existing vision models that are based on 3-channel RGB images, which makes it difficult to leverage their pretrained checkpoints. In this work, we investigate variety of methods for region representations.
Recently, rapid progress has been seen for the large vision and language models, such GPT-3 (Brown et al. 2020). T5 (Raffel et al. 2019). PaLM (Chowdhery et al. 2022). CLIP (Radford et al,, 2021). DALL-E (Ramesh et al. 2022) and Imagen (Saharia al., 2022). These models exploited large unlabeled data crawled from the web and demonstrate impressive finetuning and few-shot learning capacities. Vision-language models that encode images and decode text have leveraged large pretrained vision and language components and observed significant performance gain, e.g., Flamingo (Alayrac et al. 2022) and PaLI (Chen al.. 2022). Specifically, Flamingo showed that the few-shot prompting ability of the large language model can generalize to multimodal cases using multiple pairs of image and text as input. We adapt this model architecture to explore task- specifcc finetuning and multi-tast task learning as well as few-shot capability for mobile UI tasks.
DATA
3.1 PRETRAINING DATASETS
Two kinds of UI data are used to pretrain Spotlight models. First, we use the publicly available C4 corpus (Raffel et al. 2019) which contains a large amount of web pages that can be rendered into screenshots, similar to mobile UI screenshots. We use 80 million web page screenshots for pretrain- ing. Secondly, we use large-scale UI corpus that consists of 2.69 million mobile screenshots with paired view hierarchies. The dataset was collected by crawling various mobile apps. Screenshots and their view hierarchies are captured after performing random clicks in the Android emulator. Although being much smaller than the C4 dataset, the mobile data is critical as they are in-domain unlabeled data of our downstream tasks. As mentioned in Section [ many UT modeling tasks aim to learn mapping between UT object and a language description. Therefore, we propose to pretrain Spotlight models to auto-regressively decode text-based attributes of individual objects on the web or mobile UI, with only the screenshot image and the object bounding box the input.
We select a list of HTML attributes as the text descriptions an object based on a manual examination of their qualities. The selected attributes contain text either corresponding to the rendered text on the screen (e.g., a button label), or describing the functionality of the HTML object (e.g., alt-texth Text consisting only of generie words (e.g., "image", "text") ") are removed. We also use the t tit e of a web page as the description of the full screenshot. The statistics of the most common attributes with more than 10M instances summarized in Table [ The comprehensive statistics and list of generic words are reported in Appendix D
Web Page Data:
Web Pages Objects 2.6B
text 2.3B
alt text aria label
title
placeholder
80M Avg word #
136.1M
90.4M 2.7
80M
33.1M
5.1
3.8
10.7
2.2
Table 1: Statistics of the C4 pretraining dataset.
4

---

Published as a conference paper at ICLR 2023
Mobile Data: Similar prior work eLi al., 2020b), we only use leaf objects they more likely to represent the functionalities of the UI. For each object, we use three text-based attributes from a view hierarchy node: text, content description
and resource id. These attributes are showed to be important features for models using view hierarchy as inputs (Li et al.] 2020b Wang al. (02T). Note that derive the output labels from view hierarchies but do not use the view hierarchies themselves as input.
To compensate for missing text in view hierarchies, further run OCR the screenshot to ex- tract additional text that might be present on the screen. Recognized single-line text with 80% or higher confidence are used. To improve the data quality, we preprocess the dataset remove URLs, Unicode, angle-character and non-alphaiee -on-alphabetical strings. We also remove text that rarely occurs, i.e., fewer than times in the entire dataset, or consists of generie phrases (e.g., "text view ") that do not convey useful information about the object. Table summarizes the statistics of the mobile dataset.
Screen/View Hicarchy
Objects 51.6M
Text
Content Desc
Resource id
OCR Text
2.5M
12.
2.2M
13.9M
29.1M
Avg word
2.6
2.0
2.1
3.0
Table 2: Statistics of the mobile pretraining dataset.
3.2 UI TASK DATASETS
We select four representative UI modeling tasks that previous work proposed as downstream tasks (see Figure D. We list these tasks below.
Widget Captioning (Li et al. 2020b); Generating a natural language description for the functionality of given object on the screen.
Screen Summarization (Wang et al. 202T): Generating high-level summary of a UI screen to describe its contents and functionalities.
Command Grounding (Li al.. 2021): Finding the UI object the screen that matches the intent of a natural language command.
Tappability Prediction (Schoop et al. 2022): Predicting whether given object on the screen is tappable or not.
These four tasks require models lear the mapping between UI object and natural language description in either object -> languoge or language - object direction. Furthermore, it is essential for the model to be able to both widget captioning, and zoom out to grasp the full screen for tasks such as screen summarization. We discuss these tasks in detail Appendix In our experiments, use the same dataset splits for comparison with these benchmarks.
in to focus specific region for tasks such as
4 MODELS
We design Spotlight based on the general encoder-decoder architecture that has been widely used for vision-language modeling (Chen al. 2022 Alayrac et al. 2022: Wang al.) (022). We em- ploy ViT (Dosovitskiy et al. 2021 for encoding images (screenshots) and Transformer decoder for generating language. In contrast to previous lingun-language tasks, UI modeling tasks are often concerned with specific region or object on the screen, instead of the entire image. This unique as- pect of modeling requires model to able to concentrate on a specific region on the image. We investigate different methods for realizing the Focus Region Extractor (see Figure I for computing region representation.
Region-of- Legioneof-Interest (ROI) Align: ROI Align is parameter-free approach previously introduced for extracting a feature representation of a specific region on an image (He et al.) 2017). This method performs average max pooling the feature map of an image region, and handles low spatial resolutions in the latent feature space using bilinear interpolation. In our case, we perform ROI Align for target region over the image encodings from ViT outputs, which results in fixed-length
5

---

Published as a conference paper at ICLR 2023
vector representation for the focus region. This method provides direct localization of the focus region in the encoded visual signals, and embodies soft cropping as the resulted representation carries contextual information beyond the target area due to ViT self-attention. This is in contrast to hard cropping that carves out the raw pixels of the target area (Li et al. 2020b), which loses the access to the sereen context.
Previous works have used learnable vectors as input in Transformer models to query visual modalities for specific tasks, For example, DeTR (Carion et 2020) used learnable queries for retrieving visual objects on an image. Flamingo (Alayrac et al. (2022) summarizes image encodings by using learnable query vectors to resample the image encodings. Inspired by these methods, we propose Region Summarizer, which acquires latent representation of region based on VIT encodings, by using attention queries generated from the bounding box of the region or object. The bounding box of region, B E R includes the four scalar coordinate values, i.e., [left, top, ri iht,, bottom]. For each coordinate value, we generate n vectors (Equation []. and having multiple descriptor vectors allows us to control the capacity of the region representation.
Region Summarizer:
E Reshape4xmd. -4xmxx (GeLL((W..)WWa) (
X = tesshapexxxd=ddxddd (E+ C)
In Equation I_ we first project each coordinate a scalar value of the bounding box to nde- dimensional dense vector using perceptron: GeLU(BW.). where W. E IR' 1xnd, We next reshape the perceptron activation from x nd., to x n de to spawn n de-dimensional vectors for each of the four coordinates. Then, W₂ E IRd projects each vector to the hidden dimension, d, of the Focus Extractor transformer model. Thus E E g4xaxd and each of the coordinates produces n d- dimensional vectors. To indicate from which coordinate value vector comes, add the coordinate type embedding, C' E g4x1xd, Each vector in C' represents the embedding of a coordinate type in {left, top, right, bottom). In sum, We. Wa and C' are learnable parameters. is hyperparameter that controls the number of bounding box vectors to use and the number of region summaries generate, which determines the capacity of bottleneck.
Based on the bounding box descriptor vectors, X, we then use a Transformer to extract region representations from ViT encodings, H Rmxxd in similar way Flamingo (Alayrac et al., 2022). The query of cross attention q = Q, and the memory kv H || Qi R (m)xxx which is the concatenation of and Q; on the first dimension. The ith layer of the Region Summarizer Transformer is formulated in Equation 2 where Qo X and 0 < i < I and I is the number of Transformer layers used.
Y = Qi + CrossAttennion((q = Q kv = H Q1)
0++1 Y241 DDnsseY++.1)
where Y E Ra and Q+ E Raxd_ As shown in experimental analysis, the Region Summarizer allows Spotlight to focus on a target region indicated by a bounding box while taking into account of the screen context. Our experiments show that Region Summarizer offers superior performance over ROI Align. We conducted a series of ablations for Region Summarizer in Appendix B and the pseudo code is shown in Appendix C
We use an auto-regressive Transformer decoder to generate token sequences, by attending to the target region representation, Qi E Raxd via cross-attention. We formulate all the tasks sequence decoding (see Figure D.. During pretraining, our model generates text content associated with a target region. For widget captioning and screen summary, they are naturally sequence decoding tasks. For command grounding and tappability prediction, given a target region, we ask the model to predict either Yes or No tokens following prompt text, which can be grounding command or a question for tappability. For command grounding, we let the model inspect each object on the screen by predicting Yes or No following a given command, and the object that yields the largest probability for the Yes token is used as the prediction. During both pretraining and finetuning, the entire Spotlight model is trained end to end by minimizing the cross entropy loss for next- token prediction during sequence decoding. Although there is direct supervision for Region Summarizer, our analysis shows that the learned attention weights of Region Summarizer not only captures the target region but also relevant areas in the context of the screen (see Figure

---

Published as a conference paper at ICLR 2023
5 EXPERIMENTS
Similar to previous vision-language models (Chen et al. 2022 Alayrac et al. 2022). we use existing checkpoints of ViT and T5 models to initialize corresponding components our models in all the experiments. We experiment with two different sized ViT models, B/16 and L/16, pretrained on 300M natural images (Dosovitskiy et al., 2021 ViT L/16 is larger than B/16 and has similar parameter size as the m'55 base model (Xue et al. 2020). for which we also reuse the pretrained checkpoint The hyperparameters of the models and model variant details can be found in Appendix
A
5.1 PRETRAINING
During pretraining, we train the model to decode the text of an object in the screenshot. As this learning objective generic, we can combine the heterogeneous C4 web dataset and mobile dataset by sampling objects and their text attributes. For C4 dataset, we randomly sample from three text attributes: text and alt-text as well as misc-text that includes all other text attributes, which occur much sparser than text and alt-texts We also sample the title attribute of a webpage with a small weight (0.01) to train the model with the ability to summarize the entire screenshot. For the mobile dataset, we randomly sample from the four text attributes of an object: text, content www.eendddddrreeeeeeeeeeeeeee pesscription, resource_io and OCR text. We found that the text in these two datasets contains very frequent words phrases, e.g- submit and inpul password. To counter the imbalance, we subsample the frequent phrases as word2vec (Mikolov et al. 2013) with / = le 5. As want to explore few-shot prompting and multi-task learning in addition task- specific finetuning, we sample a random number of [1 5) screen-object-te
tuples and pad the input sequence tuples, similar Flamingo (Alayrac et al.) 2022). As C4 dataset much larger, we sample batches from C4 and mobile dataset with a ratio of 9:1. The VIT encoder encodes each screenshot independently, while the decoder decodes the full sequence of the text of all the pairs, concatenated by BOC (Beginning of the chunk) and EOC' (End of the chunk) tokens. This is to encourage the model to generalize to using sequences of multiple screenshot-object--ss tuples in multi-task and few-shot prompting scenarios for UI tasks, similar to previous work for the general domain.
5.2 FINETUNING, MULTI-TASK LEARNING AND FEW-SHOT PROMPTING
Once we pretrain the Spotlight model, we investigate three modeling strategies for downstream tasks, including task-specific finetuning, multi-task learning and few-shot prompting.
5.2.1 FINETUNING
We finetune a pretrained Spotlight model for each of the four downstream tasks separately. Each example used for both finetuning training and evaluation contains a single screen-object-text tu- ple, because the model is tuned to perform the specific task and does not require task prompts or switchers. As shown Table 3 we can see that our vision-only approaches obtain state-of-the-arr performance for all the four tasks, significantly outperforming best previous models that use both screenshots and view hierarchies as input.
5.2.2 MULTI-TASK LEARNING
Multi-task learning is valuable for reducing model footprint, but developing a robust multi-task model can be more difficult. We train Spotlight model to learn the four tasks simultaneously, which is more challenging than finetuning for each specific task. We sample training batches from the widget captioning, screen summary, command grounding and tappability tasks with the weights of [3. 2, 15, 1J. which are tuned on the development set. Each example in the batch contains two screen-object-tex tuples during training so that the model preconditioned work with one prompt tuple and one prediction tuple. During evaluation, we applied the same model onto each of the four tasks with a ocreen- tuple sampled from each training set to prompt the model to perform
https: htttaa///atttub.coo/o/ooooeeeeee

---

Published as a conference paper at ICLR 2023
Model
Captioning
Summarization
Grounding
Tappability
Widget Caption Screen2Words VUT Taperception Swearngin & Li (019
97.0
Baselines
61.3
99.3
65.6
82.1
85.5
87.9*
B/16 L/16
136.6
103.5
95.7
86.9
Spotlight
141.8
106.7
95.8
88.4
Table 3: Finetuning Performance. CIDEr scores are reported for widget captioning and screen summarization while accuracy for command grounding and F1 score for tappability prediction. is obtained by Schoop et al. (2022) using both image and view hierarchy (Swearngin & Li 2019).
different tasks, As shown in Table multi-task Spotlight models still outperform previous models, and perform on par with task-specific models (Table 3) on widget captioning and tappability tasks. Although multi-task performance sereen summarization and grounding is lower than finetunning performance, they are still significantly better than the published benchmark results.
Model
Captioning
Summarization
Grounding
Tappability
VUT multi-task
99.3
65.1
80.8
Spotlight B/16
140.0
102.7
90.8 94.2
89.4 ,4
Spotlight L/16
141.3
99.2
89.5
Table 4: The performance of Spotlight models for multi-task learning.
5.2.3 FEW-SHOT LEARNING
Few-shot learning is an important capability demonstrated by large vision and language mod- els (Alayrac et al. 2022 Brown et al. 2020). in which large model can be adapted for specific downstream task solely using examples or prompts without further training. For few-shot prompt- ing, we pretrained Spotlight model directly for evaluation, without model parameter finetuning. We test Spotlight on widget captioning tasks with a different number of shots: (0, 4, 8, 16, 32} to see how the model performance is impacted by the number of example prompts (Table 5). When there shot, the pretrained Spotlight model is directly tested on the widget captioning task, without example prompts. Although at most five screen-object-texx tuples are used in each example during pretraining, our model can generalize to more shots during inference. We observed marginal im- provement over the zero-shot condition when more shots of the task are given for the L/16 model, but not for the B/16 model, which implies that models with larger capacity seems more likely to succeed on few shot learning. The models did show meaningful few-shot performance for other tasks. We conjecture that the other tasks are too far from the pretraining data distribution, pretraining datasets are still much smaller than the ones in general domain (Alayrac et al.) [2022). Using a larger and more diverse pretraining dataset and larger text decoder might improve the few-shot learning performance for UI tasks.
our
ViT
4
16
32
B/16
57.1
56.7
55.6
55.5
54.9
L/16
61.6
61.9
62.0
61.9
62.1
Table 5: The performance of few-shot prompting on the Widget Captioning task.
DISCUSSIONS
To understand how the Region Summarizer is able to enable Spotlight to focus on a target region as well relevant areas on the screen, analyze the attention weights for both Widget Captioning
8

---

Published as a conference paper at ICLR 2023
and Screen Summarization task. In the Widget Captioning example (Figure 2ap, model lears to attend to not only the target region of the check box, but also the text "Chelsea" on the far left to generate the caption. For the Screen Summarization example in Figure where the target region is the entire screen, the model learns to attend to important parts on the screen for summarization.
the
Popudar Teams
Munchottrr United
Chetees
Areenal
Ou Biwwn
HDT
(a) Predicted caption: select chelsea team
(b) Predicted summary: page displaying the tutorial of learning
Figure 2: Visualization the attention weights of the last cross- cross-attention layer of the Region Sum-
marizer in the L/16 fintuned models for Widget Captioning (a) and Screen Summarization (b).
Spotlight models advance the of UI understanding by using a large-scale pretraining dataset and a general architecture that can support different use cases. As shown in our ablation study (Appendix(B). pretrained models significantly outperformed the models trained from scratch. There various ways of using the unlabeled dataset to pretrain UI models (He et al., 2020 Bai et al.) 2021]. In our early exploration, we also experimented with decoding a serialized view hierarchy directly instead of only text of individual objects. However, this led to poor pretraining performance, likely due to the large amount of noise in view hierarchy structures (Li et al., 2022). Nevertheless, the structural information encapsulated in the view hierarchy is valuable and investigating the usage of such information for better pretraining remains an interesting direction for future work.
Compared to recent large vision-language model efforts (Alayrac al. 2022 Chen al.) 2022) that use (10B) parameters, the model size that we investigated is relatively small (Appendix A Our experiments consistently show the trend of larger model yielding better performance, which promises that simply scaling up the model sizes would potentially lead to further performance gains. For this purpose, Spotlight can easily adopt larger ViT and T5 checkpoints. Our model architecture is generic, which takes screenshot and a region bounding box of interest as well as text prompts, and outputs responses. Spotlight can be easily applied more UI tasks, e.g., icon classification (Deka et al. (017)) and multi-step grounding (Li et al. 2020al Burns et al. 2022 as many UI tasks can be formulated as sequence decoding task.
CONCLUSIONS
We presented Spotlight, vision-only approach for mobile UI understanding, which alleviates the need to use view hierarchy. We design Spotlight based on general vision-language model archi- tecture and enable the model to focus on region of interest on the screen. This architecture is easy to scale and can benefit from the success of recent large vision-language models pretrained for the general domain. Spotlight is able to fulfiil range of UI modeling tasks. Our experiments show that Spotlight obtains SoTA results on several representative UI tasks and outperforms previous works that both screens and view hierarchies. We conduct ablations and report our findings on these variants, as well as a variety of modeling strategies and use cases, including finetuning, multi-tasking learning and few-shot prompting.
range of model variants,

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We thank the anonymous reviewers for their constructive feedback for improving the paper. We thank Mandar Joshi and Tao Li for their help in processing the C4 dataset. We also thank Chin-Yi Cheng and Forrest Huang for their feedback.
REFERENCES
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning, 2022. URL BtLphtts::/ pattps://arxive ptgg/ab.......o. 2204.14192
Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Agüera y Arcas. Uibert: Learning generic multimodal representations for UI understand- ing. In Zhi-Hua Zhou (ed.). Proceedings of the Thirtieth International Joint Conference Arti- ficial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pp. 1705- 1712. ijcai.org, 2021. doi: 10.24963/jccai222212333 URL https: org/ 24.29333 ijcai. 2021/235
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Horbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL /tRp:tttpp/ppccccedsssssot
cc cc/papprr/02////ieeeee
11570066100000711111bee101221211 Paper.. pdf
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. A dataset for interactive vasion-language navigation with unknown command feasibility, 2022.
URL
wtt.oorggabb///.a../// 02322
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, 2020. URL https:
7/arxiv.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Se- bastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thap- liyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scaled multilingual language-image model, 2022. URL https:
//arxiv.org/abs/2222 httpg//bb/220900677 067944
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski. Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov. Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstemn Douglas
10

---

Published as a conference paper at ICLR 2023
Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL https: // arxia. org abs///b///// www.abb..a.....ee/aa./a 02311
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jef- frey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven de- sign applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST '17, pp. 845-854, New York. NY. USA, 2017. Association for Computing Machinery. ISBN 9781459349819. doi: 10.114531125944.12266 bittp:://di.orgg/
URL
11 3126651
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-11 1423. URL hetps:
www. www.ww.wwwbb...................w wttp:///aoaoooo////oeoo----
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openzeview.neee wwtp://enn/oooomeeee (icbFCNNTT
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE ternational Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 2980-2988. IEEE Computer Society, 2017.
doi: 10. 11////C/..211..3222.. URL https:
Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schu- biner, Ruby B. Lee, and Jindong Chen. Actionbert: Leveraging user actions for semantic under- standing of user interfaces. CoRR, 1bbs/2012.12350, 2020. URL / http://axxvvv..oo. /
2012. 12350
Gang Li, Gilles Baechler, Manuel Tragut, and Yang Li. Learning to denoise raw mobile ui layouts for improving datasets scale, 2022. URL / wwtp://wwrr//bb/////o/// 04100
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Proceedings of the 58th Annual Meeting of
the Association for Computational Linguistics, pp. 8198-8210, Online, July 2020a. Associ-
ation for Computational Linguistics. doi: 10. wwtp:///02000.mmi...2... wwww.aclwwed
URL https:
mai 729
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning:
Generating natural language description for mobile user interface elements, 2020b.
Yang Li, Gang Li, Xin Zhou, Mostafa Dehhani,, and Alexey A. Gritsenko. VUT: versatile UI transformer for multi-modal multi-task user interface modeling. CoRR, abs/21 12.0522,, 2021. URL httph: /ttp:s////aa.voo/a org/
05692
Tomas Mikolov, llya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed represen- tations of words and phrases and their compositionality, 2013. URL https::/arxiv..or
abs/1310.4546
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
htttps://arxi>> wwtp:////////eeeooooooooo/.ee 20000
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, 0tts/1900.108.. 2019. hRR.http::///at /tctt:///axxt www..oorggaaae abs/ 11t.//99111...111111111111 10683.
11

---

Published as a conference paper at ICLR 2023
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URL https: /ttps///axiv.... org/ /bs/
1204.06122
Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, and Jacob O. Wobbrock. Examining image- based button labeling for accessibility in android apps through large-scale analysis. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS '18. pp. 119-130, New York, NY. USA, 2018. ACM. ISBN 978 1-4503-56500--3 doi: 10.1145/ 3234695.32366
URL http:tttp://ottt acm. org /0 114./ 11111..111//333453333 3234695 3236364
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sal- imans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image dif- fusion models with deep language understanding, 2022.
2205.11487
Eldon Schoop, Xin Zhou, Gang Li, Zhourong Chen, Björn Hartmann, and Yang Li. Predicting and explaining mobile ui tappability with vision modeling and saliency analysis, 2022. URL
https://arxi:: org/ abs/ 2204 02448
Amanda Swearngin and Yang Li. Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning, pp. 1-11. Association for Computing Machinery, New York, NY. USA, 2019. ISBN 978 178555999770. URL
http:///doooog11..11111111
605. 3300305
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile UI summarization with multimodal learning. U1S 2021. URL https:
2108 03353
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as foreign language: Beit pretraining for all vision and vision-language tasks, 2022. URL https:
axxiv.org/abs 2208 10442
Jason Wu, Xiaoyi Zhang. Jeff Nichols, and Jeffrey P. Bigham. Screen parsing: Towards reverse enginecring of ui models from screenshots. 2021. URL httttp:://axxiv.r////eeet
08763. pdf
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. CoRR, abs/2010.1 1934, 2020.
http://awwwwwa.aaaaaaa/// wtt.://rr/abbb./.../ooo 11934
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022. URL http:::tt////tt //axxvi.///
org/ 2b5//bs/220 ab////bs.2222 2205. 01917
Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey Bigham. Screen recognition: Creating accessibility metadata for mobile applications from pixels. In Proceed- ings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI 221 New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978 97834903. doi: 10.1145/3411744.4444
URL htttts::////iii.. www.oorgg...... 1145/ 10t...11111111111000 3411764 34117644344411818 3445186
A HYPERPARAMETERS & MODEL SIZES
We base Spotlight models in our experiments on T5 and VIT. We used two ViT configurations: B/16 and L/16 (Table 6). We use an image resolution [740, 740] for all our experiments. The mobile screenshots are resized with aspect ratio preserved and padded to /740, 740] if necessary. The paddings in the pixels are ignored by the ViT encoder using masking. The object bounding boxes normalized to [0. 1]. We use 0.1 dropout in all our models. For all the training experiments, use a batch size of 128 and linear warmup/rsqrt decay for learning rate. During inference decoding, beam size of is used. The maximum text decoding length is 64 for each screen-object-tex tuple.
12

---

Published as a conference paper at ICLR 2023
TS
ViT
VIT Enc Layers
Dec Layers
# Parameters
base
B/16
12
12
619M
base
I/16
24
12
843M
Table 6: The Spotlight model variants in our experiments.
Region Summarizer: For both the B/16 and L/16 model, we use n = -,,d. 64, = -66 for Equation I and
Pretraining: We pretrain the Spotlight models with a learning rate of 9c-3 for both B/16 (164K steps) and L/16 models (156K steps), with an initial linear warmup to 10k steps. Each training example consists of [1.5] image-object-tex tuples sampled from the pretraining datasets. The model with VIT B/16 is trained using 128 Google Cloud TPU v3 cores for 54 hours. The model with ViT L/16 is trained using 256 Google Cloud TPU v3 cores for 86 hours.
Finetuning: For finetuning, we use learning rate of 1e-3 and 20k steps for the Command Ground- ing task and le-4 and 10k steps for the other three tasks.
We use learning rate of 3e-4 for multi- maulti-task learning, and train the multi-task Spotlight models for 30k steps. The sampling weights are [3, 2, 15, 1] for the Widget Captioning, Screen Summarization, Command Grounding and Tappability tasks. We use screen-object-tex ples in each example during training. During evaluation, a screen-object-text tuple from the training set of the target task is used to prompt the model to perform the task.
Multi-Task Learning:
B ABLATION STUDY
To explore different options for designing the Spotlight models, we conduct an extensive ablation study, We use the model with B/16 ViT the full model. Each ablated version has one option changed compared to the full model, which is listed in the first column in Table 7 We pretrain all the ablation variants for 100K steps and finetune them on the four downstream tasks using the same hyperparameters and training schedule (see Appendix A The full model in Table 3 has better results as it is pretrained with more steps (164K).
Most notably, the models that use frozen ViT or are trained from scratch did not perform well for all the four tasks. This indicates the pretraining on the mobile and web datasets is crucial for the downstream UI tasks. Using only one of the two pretraining datasets also led to degraded perfor- mance, which demonstrates the value of combining heterogeneous datasets.
Region Summarizer is important component that we proposed for enabling Spotlight to focus on a specific region on the screen. We compare a few options for the design of Region Summarizer. In Equation 2 the region representation is concatenated with ViT encodings to form the memory for cross attention and the region representation updated after each layer: kv = H || Q; and Qo = X. We investigate static region representation in kv, i.e., ky H || X, which is not updated by each cross-attention layer and no region representation kv. i.c., kv = H. We also investigate embedding four bounding box coordinates jointly instead of individually. These alternatives perform reasonably on the UI tasks except the Widget Captioning task. This indicates that more flexibility in the bounding box queries, e.g., embedding each coordinate individually and updating its embedding using cross- attention per layer, helps the Widget Captioning task. In the Section 6 we can see that the bounding box queries not only attend to the area of the bounding box itself, but also the relevant context on the screen needed for decoding the captions.
Lastly, using ROI Align directly performed worse for the screen summarization task than using Region Summarizer, possibly due to that some information of the entire screen can be lost during the average pooling. In contrast, using the vector from ROI Align as the query to Region Summarizer instead of the bounding box led to improved results for text decoding tasks, but performed worse for the Command Grounding task.
13

---

Published as a conference paper at ICLR 2023
Ablation
Widget Caption
Screen Summary
Grounding
Tappability
Full model
125.1
95.7
95.6
86.9
Freeze VIT
37.5
19.6
72.6
72.3
From scratch
18.5
23.1
36.7
79.5
C4 only Mobile only
120.3
94.2
96.3
87.8
105.7
90.3
89.8
87.3
Static bbox queries
118.0
96.2
95.1
87.4
No bbox in KV
114.7
93.7
95.9
87.8
Joint bbox embedding
119.0
94.0
96.7
87.8
ROI Align as query
124.4
94.9
89.4
87.4
ROI Align
121.8
87.5
89.0
86.8
Table 7: Ablation study results.
C
PSEUDO CODE
def bbox, # The bounding bex coordinates in [batch, 4, 1].
vit_outputs, namerayers, hiacen_ dim, output_dim, Output dimension for the first dense layer. num_query,
# The ViT outputs in [batch, vit_seq_Len, hidden_dim)
Number of cross-attention layers.
Hidden dimensien for the cross-attention layers.
The number of queries generated from each coordinate.
# Pirst, map bounding bbox gelu (Dense (bbox, features=output_cm _acptttppt__mm)) ddm))
coordinates to dense vectors.
# Split each bbox vector into multiple queries. bbox abox.resher
[batch,, 4, nun nun_query, -1]
# Mар to the dimension required by crose-attention layers. bbox Dense (bbox, foatures-hidden_dii
# Compute type mabeddings for the four coordinates.
coord_types [[]], [2], [3], [4]]
type_embedding Embed (coord_types)
Reshape for proper proadcast.
sype_embedding = type wwweeeeeeddiiggeeteeeeee (11, 4, 1, hidden dim
Combine coordinates with the type embeddings.
bbox bbox type_embedding
# Perform cross-attention to extract region information. bbox
for in range (num num_layers):
kv concat (vvitouupuus, x], axis=1)
x = x CrossAttention (q=xx kv) = x Dense_i (x) return
Listing 1: Region Summarizer Psuedo Code
D PRETRAINING DATASETS
In the mobile pretraining dataset, view hierarchy is a tree structure of mobile UI, like the DOM tree to a webpage. In View hierarchy, each leaf node corresponds an object the screen, which contains a collection attributes such as object type, text content, visibility and clickability. Each non-terminal node represents group of objects. However, in reality, view hierarchies often acquired from the runtime render tree UI screen, which can contain much noise and might not be aligned with the visual representation.
14

---

Published as a conference paper at ICLR 2023
Attribute
Count
Avg word
title
2.6B
10.7
text
2.3B
5.1
alt-text
136. 136.1M
3.8
aria-label
90.4M
2.7
placeholder
33.1M
2.2
data-icon
7.2M
1.5
data-service
874K
1.1
data-caption
241K
9.7
uk-icon
167K
1.6
data-hint
134K
3.6
data-svg
126K
1.6
data-social
114K
1.9
uk-tooltip
68K
4.0
voq-icon
4K
3.0
Table 8: Statistics of the HTML attributes in the C4 pretraining dataset.
We list all the HTML attributes used for pretraining and their statistics in the C4 dataset (Table For both the web and mobile dataset, we remove invisible objects (based on CSS or view hierarchy attributes) and objects with bounding box of uniform color, and filter out object text consisting of only generic words. The list of generic words are the following: action, bar, menu, title, and, ans, app, icon, name, arg, background, element, btn, but, bottom, button, content, desc, text, item, empty, fab, image, grid, header, img, imgfile, lbutton, label, letter, list, view, pic, placeholder, random, row, single, raw, small, large, sub, template, navbar, banner, test, textinput, error, texto, todo, toolbar, tool, track, txt, unknown, stub, web, left, right, tlb, nan, page, feature, menugrid, picture, tabs, number, node, iconimage, entity, webview, heading, logo, tbl, tab, primary, footer. We also remove object text that contains only single characters or URLs. Continuous spaces and underscores are replaced with single space. All the text are lowercased.
DOWNSTREAM TASKS & DATASETS
We list the dataset statisties of the four downstream UI tasks in Table 9 and show one example of each task in Figure 3 For Widget Captioning, Command Grounding and Tappability. the models use individual UI objects, e.g., decoding the caption of object. For Screen Summarization, the models are trained to decode the summary of an entire screen. More detailed information of the four datasets can be found in their original papers.
Task
Split
Screens
UI Objects
Train
14,878
41,221
Widget Captioning Li et
2020b
Dev
1,292
3,483
Test
1,265
3,621
Train 15,743
Screen Summarization (Wan et al.
2021
Dev
2,364
Test
4,310
Train 7,822
178,783
Command Grounding Li et al.
202
Dev
1,024
24,007
Test
987
22,974
Train
2,567
14,781
Tappability Schooop et al.
2022
Dev
309
1,857
Test 342
2,029
Table 9: The dataset statisties of the four downstream tasks.
15

---

Published as a conference paper at ICLR 2023
Top
you take andiicttinn??
1184499
Ser Diggo CA
Dookinge
Busiddyy moneging twen wiww you bugant
Fetur del
GevEN
Resevry
00 2
For Formectorr
CONFIRM
CD PU MiiiiN
(a) Widget Captioning: open drawer menu
(b) Screen Summariza- (c) Command Grounding: tion: multiple options of click on location icon left medicine in health upp to san diego
(d) Tappability: tappable
Figure 3: Examples of the four UI tasks. For Widget Captioning, Command Grounding and Tappa- bility Prediction, the target object highlighted for illustration purposes only, i.e., these red bound- ing boxes are not marked on the images that are fed to the model.
16


Reviews:
[{"review_id": "h-oNi1x3Np-", "reviewer": "Reviewer_V2L4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a model that incorporates bounding box queries in the visual encoder to extract region-specific features for UI modeling tasks. With the proposed framework, no view hierarchy would be required during the above modeling process, which thus alleviates the potential concerns of missing object text or bounding box misalignment.\n", "summary_of_the_review": "This paper proposes a model that incorporates bounding box queries in the visual encoder to extract region-specific features for UI modeling tasks. With the proposed framework, no view hierarchy would be required during the above modeling process, which thus alleviates the potential concerns of missing object text or bounding box misalignment. However, the paper is not well presented (with insufficient technical contributions and details, and missing or over-claimed experimental results). I would reconsider my ratings if the above issues can be properly addressed.", "strength_and_weaknesses": "Strength\n1. The idea of incorporating bounding box queries to obtain region-specific features provides a unique way to extract fine-grained features.\n2. Compared with existing approaches, impressive performance on finetuning experiments for three mobile UI understanding tasks were achieved.\n\nWeaknesses:\n1. Overall, the novelty of this work is limited. The model architecture is similar to Flamingo (Alayrac et al., 2022). The only difference is that the inputs of Flamingo are visual tokens and learned latent queries, while SPOTLIGHT takes visual tokens and bounding box queries.\n2. The main idea of this work is that, the bounding box queries have capacities to learn how to represent specific image regions, while jointly taking into account of the screen context. However, there is no further analysis, explanation, or qualitative/quantitative supports for this claim.\n3. The experiment comparisons are potentially unfair. The improved results reported by SPOTLIGHT might come from large-scale pretraining, overwhelming parameters, and the uses of more powerful models (i.e., ViT and T5). All the baseline models (e.g. Widget Caption (Li et al., 2020b) and Screen2Words (Wang et al., 2021)) are composed of ResNet and vanilla Transformer with 128 dimensions. \n4. Missing comparisons to SOTAs in multi-task (Table 4) and few-shot learning tasks (Table 5).\n5. In Section1, the paper states its second contribution: “we develop a method for creating a large-scale pretraining dataset from automatically collected mobile screens.” However, this newly dataset is not discussed or presented later.\n6. In Sect. 5.3, the authors note that SPOTLIGHT outperforms previous models. However, in Table 4, there are no other experiments to support this claim.\n\n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_ZFoliePeGu", "reviewer": "Reviewer_cKCz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For the mobile UI understanding tasks, the authors achieve advanced performance on multi-task learning and few-shot learning by pretraining the vision-language model on the proposed 2.69M dataset. This pretraining-finetune framework is easily scalable to other UI modeling tasks and not needs the view hierarchy as auxiliary information. Meanwhile, the authors proposed a pipeline for automatically creating the large-scale UI understanding dataset.", "summary_of_the_review": "It is recommended to add some details and ablation study in paper. Moreover, since there is very little research in this field, I encourage the authors to open source the pretraining dataset,model checkpoints, and code, which will promote the development of this field and attract more researchers to make the field more prosperous.", "strength_and_weaknesses": "Strength:\nThe authors attempted to study UI understanding tasks by multi-task learning with pretrained multi-modal models, which is an independent, innovative and laborious work. The experiment results prove the generalization and validity of this method. Meanwhile, The author also proposes a large-scale UI modeling pretraining dataset. \n\nWeakness:\n1. Although the motivation is innovative in constructing the pretraining model for the UI modeling field, the overall pretraining pipeline may lack appropriate innovation and some aspects are similar to Flamingo, e.g., the vision-language model architecture, the evaluation method on the multi-task and few-shot learning, and the computational pipeline of Region Summarizer is similar to the Perceiver Resampler of Flamingo. \n2. The illustration of the pretraining dataset is too brief. It is recommended to add more statistical analysis and construction details, e.g., The text and content description are human-annotated or not?\n3. Considering the readability, it is recommended to add the introduction of test datasets about downstream tasks. Perhaps the authors can add these in the supplementary material rather than suggesting the readers read the previous literature.\n4. The ablation study is limited and maybe authors can consider the following ablation to make the paper clear and complete.\n(1)\tRegion Summarizer:\n①\tWhy choose the bbox coordinates as q rather than the region feature of bbox?\n②\tWhat is the result when kv is just the vit_outputs rather than the concatenation of vit-outputs and bbox.\n(2)\tPretraining:\n①\tCan the text input is concatenated by the four text elements of an object?\n②\tThe ViT is freezing as Flamingo or not?  If not, The much screenshots of examples will bring the burden of GPU memory or not?   \n5. The 2.69M pre-training data as extra knowledge may obscure the fairness of the method compared with the baseline method. Maybe author can try to further finetune the comparison methods combined with single-modal encoders of the pretraining model? This is just a suggestion out of curiosity and not influence the final judgment.\n6. In the Discussion section, these “early exploration” are supposed to illustrate hypotheses by some experimental data. \n7. The paper of the comparison method “Widget caption” is cited repeatedly.\n8. Maybe the authors can consider making the dataset and code public to promote the development of this field. It is just advice and not influence the final judgment.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "R3EEEXw7waK", "reviewer": "Reviewer_WFS1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduced a novel vision-language multi modality model (Spotlight) for mobile UI understanding tasks. Specifically this model is designed for tasks for mapping visual components in the UI to natural language text. In the context of UI understanding models, this paper presents 2 major contributions.\n\n1. The authors challenge the recent belief that view-hierarchy based UI understanding model is a hands-down choice for accuracy. They proposed the Spotlight model which only uses visual information at inference time; this can potentially lead to higher accuracy by working around the typical incomplete hierarchical view information for prediction.\n\n2. The authors developed a pretrained model for the UI understanding tasks. They present SoTA accuracy on UI understanding tasks under single-task finetuning, single-task few shot and multi-task finetuning. They show that the pretraining is crucial to the high accuracy in widget captioning, screen summarization, command grounding tasks.", "summary_of_the_review": "Overall I vote for a marginally-below-acceptance. I think the empirical results are overall great and the visual input only model is a novel design over the conventional view-hierarchy based models [Based on my limited understanding of UI view-hierarchy]. However I think the writing quality is below the bar for top tier conference submissions. Disclaimer: :) I am experienced in developing large language model and multi modality models. But I am not an expert in the UI understanding domain.", "strength_and_weaknesses": "Strength\n\n1. The technical effort is solid. The creation of the pretrained model involves the collection and curation of millions of UI screen shots. Such endeavor combined with the novel pretraining objective for UI understanding task enables doubling the CIDEr score from no pretraining efforts on the same model.\n\n2. The empirical results is impressive. It attains SoTA accuracy on the 3 tasks with only visual info input during inference time.\n\nWeakness\n\nThe paper is a bit hard to follow and needs modification to enable a self-containing manuscript for readers to understand.\n\n1. There are technical concept/facts missing their intuitive or formal definitions in the related work / background sections. Representative missing information are\n     a. What is an intuitive definition of view-hierarchy information which is the primary input to existing models. \n     b. How severe is the information missing problem in view-hierarchy information? Could you slightly quantify it w.r.t. the datasets the author used or collected? This is important to evaluate the claim on the problem of previous view-hierarchy info based models. \n\n2. The model design is a bit hard to fully understanding with pure text. Overall, I would recommend a symbol based formal model description which could answer the following questions.\n    a. In listing 1, it seems the Dense function encodes the coordinate into vectors. Is this encoding the normal way in VIT which just maps the coordinates to the 1D index of the image patch in the patch sequence?\n    b. Also why the coordinate is encoded twice via both the Dense function and the SinusoidalEmbed function?\n    c. What is the training objective? My rough guess from the relevant description in 4.1 is that it use the self contrastive loss type thing to promote the patch and text description pairs. But it is very unclear to me if this is the case.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster