Venue: NAACL 2025

# ScreenQA: Large-Scale Question-Answer Pairs

Over Mobile App Screenshots

 Yu-Chung Hsiao\({}^{*}\) &Fedir Zubach\({}^{*}\) &Gilles Baechler &Srinivas Sunkara

Victor Carbune &Jason Lin &Maria Wang &Yun Zhu &Jindong Chen\({}^{\ddagger}\)

Google DeepMind &Cisco Systems\({}^{\dagger}\)

yohsiao@cisco.com, {fedir, baechler, srinivaskssun, vcarbune, jasonjlin, mariawang, yunzhu, jdchen}@google.com

###### Abstract

We introduce ScreenQA, a novel benchmarking dataset designed to advance screen content understanding through question answering. The existing screen datasets are focused either on low-level structural and component understanding, or on a much higher-level composite task such as navigation and task completion for autonomous agents. ScreenQA attempts to bridge this gap. By annotating 86k question-answer pairs over the RICO dataset, we aim to benchmark the screen reading comprehension capacity, thereby laying the foundation for vision-based automation over screenshots. Our annotations encompass full answers, short answer phrases, and corresponding UI contents with bounding boxes, enabling four subtasks to address various application scenarios. We evaluate the dataset's efficacy using both open-weight and proprietary models in zero-shot, fine-tuned, and transfer learning settings. We further demonstrate positive transfer to web applications, highlighting its potential beyond mobile applications.

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 9427–9452

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Recent advancements in machine learning, especially Visual Large Language Models (VLMs) or Multimodal LLMs (MLLMs), have catalyzed numerous applications centered on mobile screens. These applications range from personal assistants enabling hands-free or eyes-free interaction to code generation from UI design mock-ups, adaptive device interfaces, automated ad creation, and mobile app testing. All of these require reliable screen content understanding as the foundation to ensure quality applications.

Mobile app screenshots have been analyzed using machine learning from multiple perspectives. These include pixel-level understanding, such as layout structural analyses, UI issue detection and correction Li et al. (2022), UI element semantics like icon recognition, button action prediction Sunkara et al. (2022), to even higher-level functional analyses such as accessibility support Li et al. (2020), screen description Wang et al. (2021), and screen type classification Deka et al. (2017). However, the specific aspect of understanding screen contents from pixels remains comparatively understudied.

Screen contents encompass diverse information, from restaurant ratings to system settings and chat messages. This capacity is crucial because: 1) screens primarily function to present information, and 2) autonomous agents and task completion systems require precise screen content understanding to achieve reasonable success rates for multi-step processes.

In this work, we propose using pixels as the sole representation of UI screens, to eliminate dependency on unreliable structural representations such as view hierarchies (VHs) Zang et al. (2021) and maximize applicability to various visual understanding tasks. To this end, we annotated the RICO dataset Deka et al. (2017) with 85,984 questionnaire pairs (Section 4) and defined four tasks with evaluation metrics (Section 3), creating _ScreenQA1_.

Footnote 1: ScreenQA dataset is released at [https://github.com/google-research-datasets/screen_qa](https://github.com/google-research-datasets/screen_qa)  2 We deliberately chose not to combine RICO with other datasets for this reason.

RICO remains the preferred screenshot dataset due to its 1) extensive collection of 66k screenshots, 2) diverse app distribution spanning 9.3k apps across 27 categories, 3) numerous benchmarking tasks built upon RICO that enable cross-referencing and comparison2Wang et al. (2021); Li et al. (2020, 2022); Ahmed et al. (2023); Lu et al. (2024), and 4) widespread use as a training dataset in recent document/screen understanding and VLM research Cheng et al. (2024); Liu et al. (2024); Xie et al. (20

newer datasets, RICO's comprehensive nature and established presence in the field make it an optimal choice for this work. See Table 1 for comparison.

This work offers the following contributions:

1. We create and release ScreenQA, the first large-scale question answering dataset and benchmark for mobile screens.
2. ScreenQA provides rich annotations: short answers, full-sentence long answers, and supporting UI element contents with their bounding boxes (Section 4).
3. We propose four tasks with evaluation metrics to satisfy various application needs in extraction, text generation, and grounding (Section 3).
4. We establish zero-shot, fine-tuning and cross-domain transfer learning baselines for both closed and open-weight models quantifying the effectiveness of ScreenQA (Section 6).

## 2 Related Work

Closest to ScreenQA is work that tackles multimodal understanding and question answering, as discussed below.

### Multimodality

We discuss the research on text-heavy images3 and provide an overview in Table 2.

Footnote 3: Natural image VQA and video VQA are omitted due to the space constraints.

Screen UI-based understandingUnlike natural images, UIs are designed to be informative and actionable. Examples of work dealing with informativeness include 1) UI element identification, such as icon detection Deka et al. (2017), widget captioning Li et al. (2020); Chen et al. (2020), and 2) referring expression to UI elements in classification Wu et al. (2023), representation learning Bai et al. (2021) and generatio Hong et al. (2023).

\begin{table}
\begin{tabular}{l c l l l} \hline \hline Dataset & Size & Unit & Apps & Categories \\ \hline RICO Deka et al. (2017) & 66k & screensh & 9.3k & 27 \\ LabelDroidChen et al. (2020) & 13k & screensh & 7.6k & 25 \\ MoTFB Burns et al. (2022) & 4.7k & trace & 0.1k & n/a \\ AiW Rawles et al. (2024) & 715k & trace & 0.4k & n/a \\ \hline \hline \end{tabular}
\end{table}
Table 1: The RICO dataset remains the most suitable mobile app screenshot collection for this work due to its diversity, large scale, and established presence in the field. Although AitW is larger in scale, it lacks app and screenshot diversity due to its sampling method.

Figure 1: ScreenQA examples. (a) Four corresponding tasks (Section 3): 1) Short answer (SQA-S): “1,1”. 2) Long answer (SQA-L): “There is 1 like and 1 comment”. 3) UI Content (SQA-UIC): “[1”]. 4) UI Content w/ Bounding Boxes (SQA-UIC-BB): also with bounding box coordinates (in green). (b) Both high and low temperatures in two Saturdays are relevant answers. (c) Lacking the content to answer the question.

 

Actionability is related to task completion and autonomous agents. MoTIF Burns et al. (2022), VisualWeb Arena Koh et al. (2024), and Android-in-the-Wild Rawles et al. (2024) provide interactive app environments for evaluating visually grounded screen agents. This work is focused on the information retrieval aspect.

Visual Document UnderstandingVisual Document Understanding concerns understanding scanned or photographed documents. DocVQA Mathew et al. (2021) uses an extractive QA format for span/segment extraction. TextVQA Singh et al. (2019) and several other domain-specific datasets Mishra et al. (2019); Gurari et al. (2018); Huang et al. (2019); Abdallah et al. (2024); Jaume et al. (2019); Harley et al. (2015); Lewis et al. (2006) relate the 2D arrangement of texts to semantic meanings. Beyond text alone, infographics understanding Mathew et al. (2022); Masry et al. (2022); Kahou et al. (2017); Kafle et al. (2018); Chaudhry et al. (2020) focus on charting with text around.

### Question Answering

We focus on closed-domain question answering grounded in specific contexts. Answer formats include span Rajpurkar et al. (2016), entity Talmor and Berant (2018), multiple choice Mihaylov et al. (2018), and generation Xiong et al. (2019). Capacities range from reading comprehension Yang et al. (2015), multi-hop reasoning Yang et al. (2018), Chen et al. (2021b), logical reasoning Yul et al. (2020), and commonsense reasoning Talmor et al. (2019).

## 3 Problem Setting: Tasks and Metrics

As each example in the validation and test sets may contain alternative ground truths from multiple annotators, we compute the evaluation metric by: 1) calculate the maximum metric value across all annotator-provided ground truths for each example, and 2) average these maximum values across the entire dataset:

\[{\text{avg}}(metric)=\frac{1}{N}\sum_{i=1}^{N}\max_{j}[metric(A_{i},A_{i,j}^{g })],\]

where \(N\) is the number of questions, \(A_{i}\) is the predicted answer for \(i\)-th question, and \(A_{i,j}^{g}\) is the \(j\)-th ground truth for \(i\)-th question.

ScreenQA (or SQA in short) presents four tasks encompassing diverse application scenarios. Examples are illustrated in Figure 1.

ScreenQA Short answer (SQA-S)Given a screenshot and a question, output a short (concise) answer to this question using the information presented on the screen. If the screenshot doesn't contain the answer, output "<no answer>".

Similar to SQuAD Rajpurkar et al. (2016), we propose to use _Exact Match_ (EM) and _F1-Score_ for accommodating acceptable permutations and rephrasing of the same content. We also apply SQuAD pre-processing to normalize answers before computing the metrics. This task is the core capability enabled by the dataset.

\begin{table}
\begin{tabular}{l l c c l} \hline \hline Dataset & \# Images (k) & \# Examples (k) & UI? & Task Type \\ \hline ScreenQA (this work) & \(35\) * & \(86\) & ✓ & Question Answering (QA) \\ \hline TextVQA Singh et al. (2019) & \(28\) & \(45\) & ✗ & QA \\ DocVQA Matthew et al. (2021) & \(12\) & \(50\) & ✗ & QA \\ InfographicVQA Mathew et al. (2022) & \(5.5\) & \(30\) & ✗ & QA \\ ChartQA Masry et al. (2022) & \(22\) & \(33\) & ✗ & QA \\ WebSRC Chen et al. (2021a) & \(6.5\) & \(440\) & ✓ & QA on Webpage Segments \\ ComplexQA Bachelter et al. (2024) & \(10\) & \(12\) & ✓ & QA on Counting, Arithmetic \\ VisualWebBench - WebQA Liu et al. (2024a) & \(0.3\) & \(0.3\) & ✓ & QA on Webpages – Eval Only \\ LabelProdChen et al. (2020) & \(13\) & \(19\) & ✓ & Text Generation \\ Screen2Words Wang et al. (2021) & \(22\) & \(112\) & ✓ & Summarization \\ ScreenAnnotation Bachelter et al. (2024) & \(22\) & \(22\) & ✓ & Object Detection \\ MoTIF Burns et al. (2022) & \(62\) & \(4.7\) & ✓ & Navigation \\ AitW Rawkes et al. (2024) & \(5,690\) †  & \(715\) & ✓ & Navigation \\ \hline \hline \end{tabular}

* A subset of the 66k images from RICO Deka et al. (2017) as described in Section 4
* Limited app and screenshot diversity as described in Table 1

\end{table}
Table 2: Comparison of ScreenQA with Related Datasets. ScreenQA is the largest QA dataset for mobile screenshots, using entire screenshots rather than cropped answer regions (as in WebSRC) to better represent real-world applications, and including unanswerable questions and bounding boxes. ComplexQA complements ScreenQA with its focus on counting, arithmetic, and comparisons. Not an exhaustive comparison due to space constraints.

 

ScreenQA Long answer (SQA-L)This task is identical to SQA-S, with the distinction that it requires a long, full-sentence answer instead of short answer phrases. This task facilitates the generation of coherent responses suitable for direct human interaction, particularly in the context of virtual assistant applications.

As the task resembles summarization of pertinent information, we propose to use _ROUGE-[1,2,L]_Lin (2004) as the evaluation metric.

ScreenQA UI Content (SQA-UC)Given a screenshot and a question, output a list of UI elements that contain the answer, where each element is represented by its text representation. If the screenshot doesn't contain the answer, output an empty list.

Section 4 will further details the UI elements corresponding to questions, their listed order, and their contents represented in text. Except for icons with predefined textual descriptions, most content resembles OCR output: text within UI elements Qin et al. (2019). However, unlike OCR, the output should be evaluated as a list rather than a continuous sequence of symbols or words.

We use _Exact Match_ and _F1-score_ as metrics. Commonly, text in screenshots is directly extractable. Therefore, we perform UI-element-wise matching without additional pre-processing.

ScreenQA UI Content with Bounding Boxes (SQA-UC-BB)Given a screenshot and a question, output a list of UI elements that contain the answer, where each element is represented by its bounding box _and_ text representation.

This task extends the previous SQA-ULG, additionally facilitating answer highlighting and action automation over the screen. The detection of bounding boxes, especially in screen contents, is rarely available in existing datasets and challenges model capabilities.

We recommend evaluating the bounding box detection quality using _F1-Score_, where two bounding boxes match if their Intersection over Union (IoU) Rezatofighi et al. (2019) score is higher than \(0.1\). _Exact match_ and _F1-score_ are evaluated in a similar way, but restricting the list of matches to only those where text representation also matches. This threshold is justified due to the annotation methods of ground truth bounding boxes: selection from view hierarchies (accommodating UI elements with substantial no-content area) or manually drawing (tightly fitting text) (Section 4.3). The simultaneous use of these approaches by different annotators typically results in low IoU values.

## 4 Data Annotation

We performed a five-stage process to annotate ScreenQA: 1) Prefiltering (Section 4.1), 2) Question annotation (Section 4.2), 3) Answer annotation (Section 4.3), 4) Not-answerable question annotation (Section 4.4), and 5) Short answer generation (Section 4.5). See data statistics in Table 3, data annotation details and UI tools in Appendix A, and data examples in Appendix B.

### Prefiltering

Prefiltering involves one round of human rating to remove screenshots with the following conditions:

1. Screenshots from non-English apps.
2. Screenshots with unsynchronized view hierarchies (VHs). Removing these data helps avoid language-related obstacles and potential view hierarchy issues Zang et al. (2021), which could otherwise introduce noise into subsequent annotation stages. We employed 27 annotators to perform this stage. Occlusion and ghosting during screen transitions are deemed acceptable if the UI elements in the main content area remain clean and accurate, resulting in different numbers from Li et al. (2020). Examples of such VH symptoms are provided in Appendix A.1.

\begin{table}
\begin{tabular}{l l r c} \hline \hline Stage & Step & \# SS & \# Q \\ \hline  & RICO Original & 66k & – \\ \multirow{2}{*}{Prefiltering Section 4.1} & Non-English apps† & 11k & – \\  & Out-of-sync VHs† & 13k & – \\ \cline{2-4}  & For Question Anno. & 51k & – \\ \hline Question Anno. & 1st pass of Q anno. & \multirow{2}{*}{35k} & 46k \\ Section 4.2 & 2nd pass of Q anno. & \multirow{2}{*}{36k} & 46k \\  & Lack of content & 15k & – \\ \cline{2-4}  & For Answer Anno. & 35k & 82k \\ \hline Not Ans. Anno. & 3rd pass of Q anno. & \multirow{2}{*}{5k} & 5k \\ Section 4.4 & & For Data Splitting & 35k & 86k \\ \hline Data Splitting Section 5.1 & Train & 28,378 & 68,951 \\  & Validation & 3,485 & 8,614 \\  & Test & 3,489 & 8,419 \\ \cline{2-4}  & Total & 35,352 & 85,984 \\ \hline \multicolumn{4}{l}{† Not mutually exclusive.} \\ \end{tabular}
\end{table}
Table 3: Counts of distinct screenshot (SS) and questions (Q) for annotation stages and data splitting.

 



### Question Annotation

We employed the same 27 annotators to formulate questions based on provided screenshots, simulating real-world app usage queries. Questions were restricted to directly observable information on the screen, deliberately excluding those questions requiring reasoning, computation, or advertisement-related content. This is because recent multimodal reasoning benchmarks Yue et al. (2024); Han et al. (2023) unveil that the multimodal LLM reasoning capacities are heavily influenced by their underlying LLM performance (Section 6 of Wang et al. (2024)), which can be more efficiently achieved through inference time scaling Brown et al. (2024); Snell et al. (2024); Wu et al. (2024); Kumar et al. (2024), followed by pretraining data and model parameter scaling Zhang et al. (2024); Owen (2024). By focusing our scope on screen content understanding, we aim to investigate multimodality independent of testing other types of intelligence.

Question annotation was a two-pass process. First, an annotator composed up to five questions per screenshot. Then, a second annotator, with access to the initial questions, added up to three more, resulting in a maximum of eight questions per screenshot. This two-pass approach promoted a diverse and comprehensive set of questions, while reducing redundancies. To encourage diverse questioning styles and a varied set of questions, each annotator was required to complete their own initial set of questions before they could see questions created by others. Screenshots lacking interesting content (e.g., login pages, ads) may be skipped.

### Answer Annotation

Given a screenshot and a question, the annotators are tasked to

1. Correct grammatical errors in the question.
2. Answer the question by: a) Annotating bounding boxes that constitute the answer. b) Ranking bounding boxes by relevance or reading order.
3. Provide a full-sentence long answer.

The annotator who composed a specific question is excluded from answering it to prevent potential bias. Our UI tool offers two bounding box annotation methods: direct selection of UI elements from view hierarchy(VH) leaf nodes, or manual drawing when VH structure is inadequate for selection.

We further consider two exceptions: 1) Mark "invalid question" if incomprehensible. 2) Mark "not answerable from the screenshot" if the screenshot does not contain the answer. The "invalid question" answer annotations are first removed for each question. Then questions without any remaining answer annotations are excluded from the ScreenQA dataset, as they are deemed invalid. To enhance evaluation quality, three answer were annotated for validation and test sets, while only one for training set (details in Section 5.1).

### Not-Answerable Question Annotations

A third pass of question annotation was implemented to increase "not-answerable questions" to approximately 10% (Figure 2). This tests models' ability to refrain from answering unanswerable questions, similar to Rajpurkar et al. (2018). Approximately 5k screenshots without existing "not answerable" questions were randomly sampled. We instructed annotators to compose one question per screenshot that was related to the screen content but unanswerable. See an example in Figure 0(c).

### Short Answer Generation

As UI design of apps may fragment a short, semantic answer into multiple bounding boxes (e.g., "Oct. 15, 2024" fragmented into "Oct.", "15", and "2024" in a date selector), we perform a post-processing step to make the annotations into a coherent, human readable short answer suitable for common question answering purposes. This also partly accommodate answer variants that are semantically identical, such as numerical representations (digit-s/words), date/time formats, and optional descriptors/units (e.g., "3 reviews" vs. "3"). We employed PaLM 2 Anil et al. (2023) to generate answer variants via few-shot prompting with inputs including question, UI element descriptions, and full-sentence answer, with a subset verified by authors. See Appendix A.6 for the prompt details.

Figure 2: Distribution of question answerability. Each split contains approximately 10% questions that have at least one “not answer” answer annotation, to test models’ ability to refrain from answering unanswerable questions.



Dataset Analysis

### Dataset Splitting

The ScreenQA dataset comprises 85,984 questions from 35,352 unique screenshots, split at the screenshot level into training, validation, and test sets in an approximate 80-10-10 ratio (see Table 3).

### Question Analysis

Among the 86k collected questions, 47.5k are unique in term of SQuAD EM. Common questions often recur across screenshots (e.g., "Which option is selected?" or "What is the email address?"). Screenshots with more information typically generate more questions. The histogram of questions per screenshot follows a mild exponential decay (Figure 2(a)). We further categorize questions using regular expressions, providing a preliminary overview in Table 4. It is worth noting that the category distribution is implicitly influenced by RICO's crawling process. For example, the crawling process commonly includes a step at the login page, resulting in a higher percentage of questions about app names, email addresses, and login permissions.

### Answer Analysis

We analyze the answer annotations in two aspects: 1) How often multiple bounding boxes are required for answers, indicating task complexity, and 2) How often VH is sufficient for bounding box annotation, indicating the reliability of VHs.

Figure 2(b) illustrates the histogram of bounding boxes per answer. About 84% of answers contain a single bounding box, with 51% utilizing VH leaf nodes and 49% using manually drawn boxes. When all answers are considered, 51% rely solely on VH leaf nodes, 48% on manual boxes, and 0.8% on a combination. Combinations typically occur for multi-part answers involving diverse UI elements, scattered components (e.g., dates), or measurements with units. Despite VHs being commonly used in prior screen tasks Burns et al. (2022); Li et al. (2020), the near-equal preference for manual bounding boxes reflect VH limitations: VMs cannot capture UI elements in WebViews and Canvas, and are inconsistent for certain screen designs. This justifies our decision to use screenshot as input in ScreenQA.

## 6 Experiments and Baselines

We conducted three sets of experiments: zero-shot, fine-tuning, and transfer learning, each of which is explained below.

### Zero-Shot Experiments

We evaluated Fuyu-8B Bavishi et al. (2023), Gemini 1.5 Flash, Gemini 1.5 Pro Gemini Team Google (2023), and GPT-4o OpenAI et al. (2024) on SQA-S in a zero-shot setting. Results are summarized in Table 5. We prompted each model using an instruction followed by a question. For the Fuyu-8b model, we used the instruction that model's authors recommended in their examples. For GPT-4o, we iterated on the prompt design using a set of 10 examples from the ScreenQA validation split and then reused the prompt for Gemini 1.5 models. See Appendix D.1 for the detailed prompts. Despite the prompt being originally designed for GPT-4o, Gemini 1.5 Pro outperforms GPT-4o in this particular setup.

### Fine-Tuning Experiments

We fine-tuned three series of models spanning open-source, domain specific, and general purpose proprietary models on the ScreenQA tasks as described below.

PaliGemma 3BWe used the pre-trained PaliGemma 3B model checkpoints Beyer et al. (2024) with three input resolutions, 224\(\times\)224, 448\(\times\)448 and 896\(\times\)896, and fine-tuned for 10 epochs with a learning rate \(1.0\times 10^{-5}\) using the Adam optimizer and a cosine decay schedule. Both vision and language backbones were trained during fine-tuning.

ScreenAI 670M and 5BThis VLM specializes in UI and infographics understanding Baechler et al. (2024). We used a dynamic \(812\times 812\) input resolution and fine-tuned this model until convergence, using a learning rate of \(1.0\times 10^{-3}\). Only the language backbone was trained during fine-tuning.

Gemini 1.5 FlashWe also fine-tuned Gemini 1.5 Flash Gemini Team Google (2023) model4 on the downstream tasks for \(\sim 7\) epochs with a learning rate of \(1.0\times 10^{-4}\) for SQA-S and SQA-L and \(1.0\times 10^{-6}\) for SQA-UIC and SQA-UIC-BB. Only the language backbone was trained during fine-tuning.

Footnote 4: [https://cloud.google.com/vertex-ai/generative-ai](https://cloud.google.com/vertex-ai/generative-ai) /docs/models/gemini-use-supervised-tuning

Note that we did not test PaliGemma and ScreenAI models' zero-shot performance as they 

are not instruction-tuned, therefore, not suitable for evaluation in a zero-shot setting.

We use the metrics introduced in Section 3 to measure the performance of models along various dimensions: i) Extracting relevant information for answering a question in the SQA-S and SQA-UIC tasks, ii) Ability to provide fluent answers in the SQA-L task, and iii) Identifying relevant UI elements through their bounding box coordinates in the SQA-UIC-BB task.

We present the results in Table 6. We observe a slightly higher performance for the ScreenAI 5B model compared to PaliGemma 3B, and attribute it to its larger model capacity and specialized pretraining mixture that includes a rich variety of UI elements. The ScreenAI model also performs on par with the Gemini 1.5 Flash model in a fine-tuned setting. We also notice a larger difference between the ScreenAI model and other approaches in the SQA-UIC-BB tasks involving bounding box prediction. PaliGemma performance is however very competitive, and by fine-tuning both language and vision backbones we enable better use of the entire model capacity.

In Appendix E we present a few examples of the different errors made by the models and a categorization of these errors.

### Cross-Domain and Transfer Learning

We conducted cross-domain learning (CDL) and transfer learning (TL) by fine-tuning PaliGemma models (Table 7). For CDL, we fine-tuned the models using either SQA-S or DocVQA Mathew et al. (2021) and evaluated on the other. These experiments aim to characterize the differences between ScreenQA and other VQA datasets from related domains like document images by investigating the performance difference by training on related datasets vs. training on the corresponding training split. We observe that the use of in-domain training data results in significant improvements for both SQA-S and DocVQA datasets. However, we do note that for the SQA-S task, using training data from the related domain of document images results in performance gain over the zero-shot setting with a larger Fuyu-8B model (Table 5), showing that they are related but distinct domains.

For transfer learning, summarized in Table 7, we used VisualWebBench-WebQA Liu et al. (2024), a recently released QA task on web screenshots,

Figure 3: Histograms for various data and model performances. (b) Approximately 91% of questions are either not answerable or can be answered by a single bounding box, hence, omitted to emphasize the long tail distribution. (d) The model exhibits consistent SQuAD-F1 performance across app categories, except for Events and Videos, which present higher scores due to lower support for those categories.

 

as the target evaluation dataset. We fine-tuned the models using DocVQA alone and using both DocVQA and ScreenQA, demonstrating positive transfer when incorporating the ScreenQA dataset.

Additionally, we tested the hypothesis that training models to predict bounding boxes would enhance their performance on tasks not explicitly involving bounding boxes. For example, fine-tuning models on both SQA-S and SQA-UIC-BB tasks and evaluating performance on SQA-S task. However, in our experiments with smaller models such transfer benefits did not materialize. One possible explanation could be that this is due to increased complexity caused by introducing an additional task. While this may seem like a limitation, it also reflects the standalone challenges each task presents, reaffirming the value of our dataset.

### Importance of Multimodality and Fine-Tuning

We used Gemini 1.5 Flash (Gemini Team Google, 2023) model and SQA-S and SQA-L tasks to demonstrate the complexity of the screen data for screen understanding. Experiments in Table 8 show that using text representation of the screen information (OCR) as model input produces significantly worse results than using multimodal (image & text) setup. Noticeable difference in performance between zero-shot and fine-tuned setups suggests that screen understanding domain presents its own challenges on top of generic image understanding.

## 7 Conclusion

We introduced ScreenQA, a rich dataset that enables training and evaluating models on question-answering tasks on screen content. We described the annotation process, statistics of the collected dataset, which contains 85,984 question-answer pairs. In addition to answers, our dataset contains extensive annotations of the UI elements, enabling the ability to train or probe models for their holistic understanding of the screen, a necessary capability for high-level reasoning and automation using UI interfaces. Compared to other vision-language tasks, such as document understanding or visualisation answering, the four constructed tasks on the ScreenQA dataset pose unique challenges: rich in text, diverse in mobile applications, blended with icons and symbols. The tasks not only evaluate content quality, but also UI element identification quality. Furthermore, we conducted a diverse set of experiments, including zero-shot, fine-tuned, and transfer learning, on a series of open-weight and proprietary models to assess the capacity of our ScreenQA dataset. We encourage the community to tackle screen content understanding challenges present in our benchmark, fostering new technologies and user experiences.

## 8 Limitations

We acknowledge limitations of our work.

LanguageReleased data is only in English and further work would be necessary to create a multilingual variant of our dataset. This is further amplified by the need of screenshots with phones configured in the different locales.

\begin{table}
\begin{tabular}{l l|p{160.7pt} p{160.7pt}} \hline \hline Category & \% & Examples \\ \hline UI selection \& config & 18.1 & Which option is selected? \\ Quantity number & 11.7 & How many messages? & How many pictures are there in Western Europe? \\ App name & 10.4 & What is the name of the application? & What is the app name? \\ Date time & 9.4 & When was “Heal the Living” released? & When is happy hour? \\ Price & 3.4 & How much is the gift bonus in 3rd place? & What is the price? \\ Name of item & 3.3 & What is the name of the drug? & What is the name of chef? \\ User name & 2.8 & What is the name of the user? & What is the username on telegram? \\ Duration & 2.5 & What is the duration of video? & How long is the stop? \\ Ennum. of avail. options & 2.5 & Which social media options are given there? & What are the options available for logging in? \\ Address and direction & 2.4 & What is the current location? & What is the service zip code? \\ Email address & 2.4 & What is an email address? & What is customer service email? \\ Person’s name & 2.1 & Who song the song? & What is the last name? \\ Others & 12.8 & What’s the average speed? & What is the user’s middle initial \\ Subtotal & 83.8 & What is the spending limit? & Which team has 41 points? \\ \hline \hline  & & \\ \end{tabular}
\end{table}
Table 4: Top question category distribution (\(\geq 2.0\%\)) and examples (See Appendix C for remaining categories).

\begin{table}
\begin{tabular}{l c c} \hline \hline  & SQuAD-EM & SQuAD-F1 \\ \hline Fuyu-SB & 39.5 & 47.3 \\ Gemini 1.5 Flash & 80.6 & 86.4 \\ Gemini 1.5 Pro & **81.4** & **87.2** \\ GPT-4o & 77.8 & 86.6 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Zero-shot public models evaluation on SQA-S. Bold is best performance.

 

Rich LayoutsOur data and annotation process focuses on rather static content, facilitating us to focus on information extraction from canonical layouts. There are a number of challenges our dataset does not capture, specifically rich layouts coming from gaming or other highly interacting applications that may be present on a user's device. Screenshots that contain natural images and videos are also missing.

ReasoningThe type of challenges our dataset focuses on are a lot more related to information look-up and rewriting. We therefore capture challenges grounded in UI content and composition understanding, rather than arithmetic or other forms of complex compositional reasoning challenges. See Section 4.2 for details.

Multi-ImageAn example in our dataset always consists of a single screenshot. Several challenges that come with multi-image understanding such as a trace of screenshots are therefore not covered by our tasks in the dataset. Similarly, no UI animations are included or scrolling actions.

PlatformScreenQA contains screenshots only from the Android ecosystem and the phone form factor. As all of the composed questions are focused on the main content area without involving any gestures and actions, we do not anticipate major performance differences across ecosystems. In addition, our experiments show positive transfer for VisaulWebBench-WebQA when additionally using ScreenQA (Table 7), indicating the efficacy of ScreenQA for other form factors.

## 9 Ethical Considerations

Information Retrieval OnlyScreenQA focuses on information retrieval of screen contents, with the primary intent of improving screen content understanding. ScreenQA dataset does not involve any

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{SQA-S} & \multicolumn{3}{c}{SQA-L} & \multicolumn{3}{c}{SQA-UIC} & \multicolumn{3}{c}{SQA-UIC-BB} \\ \cline{2-10} Model & EM & F1 & R-I & R-Z & R-L & EM & F1 & BBOX-F1 & EM & F1 \\ \hline ScreenAI 670M & 51.2 & 60.6 & 77.3 & 68.4 & 76.7 & 47.8 & 49.4 & 62.7 & 41.1 & 42.6 \\ PaliGemma 3B 224 & 77.5 & 83.9 & 88.2 & 81.5 & 87.4 & 74.8 & 76.7 & 84.9 & 67.5 & 69.6 \\ PaliGemma 3B 448 & 88.3 & 92.2 & 91.1 & 85.5 & 90.3 & 86.0 & 87.7 & 89.4 & 79.1 & 81.6 \\ PaliGemma 3B 896 & 89.4 & 93.2 & 90.9 & 85.3 & 90.1 & 86.1 & 87.8 & 88.8 & 78.8 & 81.2 \\ ScreenAI 5B & **90.7** & 94.6 & **92.6** & **87.4** & **91.9** & 87.0 & 88.7 & **94.2** & **84.0** & **85.7** \\ Gemini 1.5 Flash & 90.5 & **94.9** & 92.4 & 86.2 & 91.7 & **88.2** & **89.7** & 92.4 & 83.9 & **85.7** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Model performance for ScreenQA tasks after fine-tuning. Bold is best performance.

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline Experiment i Task & \multicolumn{3}{c}{**CDL** i SQA-S} & \multicolumn{3}{c}{**CDL** i DocVQA} & \multicolumn{3}{c}{**TL I Visual WebBench-WebQA**} \\ \cline{2-10} Fine-Tuned on & \multicolumn{3}{c}{SQA-S} & \multicolumn{3}{c}{DocVQA} & \multicolumn{3}{c}{SQA-S} & \multicolumn{3}{c}{DocVQA} & \multicolumn{3}{c}{DocVQA} & \multicolumn{3}{c}{SQA-S + DocVQA} \\ \cline{2-10} Model v Metrics & EM & F1 & EM & F1 & ANLS & ANLS & F1 & F1 \\ \hline PaliGemma 3B 224 & 77.5 & 83.9 & 52.6 & 60.5 & 27.5 & 43.7 & 19.31 & **21.51** \\ PaliGemma 3B 448 & 88.3 & 92.2 & 66.2 & 72.9 & 55.1 & 78.0 & 48.33 & **49.11** \\ PaliGemma 3B 896 & 89.4 & 93.2 & 63.6 & 70.4 & 62.0 & 84.8 & 57.07 & **58.69** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Cross-domain learning (CDL) and transfer learning (TL) experiments via PaliGEMm model fine-tuning. For CDL, models were fine-tuned on either SQA-S or DocVQA and evaluated on the other. In-domain and cross-domain model scores exceed 60 for the 896\(\times\)869 model and differ by 20\(\sim\)25 points across the board (underlined). TL experiments revealed positive transfer when additionally using SQA-S for the WebQA evaluation (bold).

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline Evaluation setup & \multicolumn{3}{c}{SQA-S} & \multicolumn{3}{c}{SQA-L} & \multicolumn{3}{c}{SQA-UIC} & \multicolumn{3}{c}{SQA-UIC-BB} \\ \cline{2-10}  & EM & F1 & R-I & R-J & R-L & EM & F1 & BBOX-F1 & EM & F1 \\ \hline Zero-Shot, Text-Only & 64.4 & 72.5 & 78.2 & 67.8 & 75.6 & 38.6 & 41.2 & 30.1 & 49.7 & 28.5 \\ Zero-Shot & 80.6 & 86.4 & 83.8 & 70.8 & 79.3 & 62.4 & 66.8 & 26.2 & 33.3 & 24.0 \\ Fine-Tuned & 90.5 & 94.9 & 92.4 & 86.2 & 91.7 & 88.2 & 89.7 & 85.7 & 92.4 & 83.9 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Gemini 1.5 Flash performance against ScreenQA for various evaluation setups. Note that for SQA-UIC-BB, the Zero-Shot, Text-Only case (the 1st row) outperforms its multimodal counterpart (the 2nd row) because the bounding box coordinates are given as text input: the model just needs to select the right UI elements with the given coordinates, which makes bounding box prediction very precise when they are correct, hence, an easier task.

 

materials related to decision-making for or against users, nor does it execute actions on behalf of users, avoiding potential harm.

PrivacyThe technologies fostered by ScreenQA, when used on a mobile device, require access to information at a level comparable to that of a standard accessibility application. Furthermore, the promising results from fine-tuned PaliGemma models (Table 6) suggest that the technologies enabled by ScreenQA can be effectively hosted on mobile devices, significantly limiting potential privacy concerns.

App Category FairnessScreenQA contains questions across 27 categories of apps (Figure 3a), which is the most diverse set at the time of writing (Table 1). Also, our fine-tuned PaliGemma 896 model exhibits consistent SQuAD-F1 performance across these app categories (Figure 3d), demonstrating that ScreenQA has a fair support across app purposes.

## References

* Abdellah et al. [2024] Abdelrahman Abdallah, Mahmoud Abdalla, Mahmoud SalahEldin Kasem, Mohamed Mahmoud, Ibrahim Abdelhalim, Mohamed Elkasaby, Yasser Eldendary, and Adam Jatowt. 2024. CORU: Comprehensive Post-OCR Parsing and Receipt Understanding Dataset. _Preprint_, arXiv:2406.04493.
* Ahmed et al. [2023] Ali Ahmed, Alaa Zaki, Enas Elgeldawi, Mohamed Abdallah, and Moheb Girgiis. 2023. MASC: A Dataset for the Development and Classification of Mobile Applications Screens.
* Anil et al. [2023] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. PaLM 2 technical report. _arXiv preprint arXiv:2305.10403_.
* Baechler et al. [2024] Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Carbone, Jason Lin, Jindong Chen, and Abhanshu Sharma. 2024. ScreenAI: A vision-language model for UI and infographics understanding. _Preprint_, arXiv:2402.04615.
* Bai et al. [2021] Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Aguera y Arcas. 2021. UIBert: Learning generic multimodal representations for UI understanding. _Preprint_, arXiv:2107.13731.
* Bavishi et al. [2023] Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Tasrilar. 2023. Introducing our multimodal models.
* Beyer et al. [2020] Lucas Beyer, Andreas Steiner, Andre Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alguldumhosin, Michael Tychannen, Emanuclele Bugliarello, Thomas Unettrthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishab Kabra, Matthias Bauer, Matko Bosnjak, Xi Chen, Mathias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pineolj Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiath Harmsen, and Xiaohua Zhai. 2024. PaliGemma: A versatile 3B VLM for transfer. _arXiv preprint arXiv:2407.07726_.
* Brown et al. [2024] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Re, and Azalia Mirhoseini. 2024. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. _Preprint_, arXiv:2407.21787.
* Burns et al. [2022] Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. 2022. A dataset for interactive vision language navigation with unknown command feasibility. In _European Conference on Computer Vision (ECCV)_.
* Chaudhry et al. [2020] Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prunn Bansal, and Ajay Joshi. 2020. LEAF-QA: Locate, encode & Attend for Figure Question Answering. In _2020 IEEE Winter Conference on Applications of Computer Vision (WACV)_, pages 3501-3510.
* Chen et al. [2020] Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xivei Xu, Liming Zhuot, Guoqiang Li, and Jinshui Wang. 2020. Unblind your apps: Predicting natural-language labels for mobile GUI components by deep learning. In _2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)_, pages 322-334.
* Chen et al. [2021a] Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021a. WebSRC: A Dataset for Web-Based Structural Reading Comprehension. _Preprint_, arXiv:2101.09465.
* Chen et al. [2021b] Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021b. FinQA: A Dataset of Numerical Reasoning over Financial Data. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 3697-3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Cheng et al. [2024] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. 2024. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 9313-9332, Bangok, Thailand. Association for Computational Linguistics 

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Arefagn, Yang Li, Jeffrey Nichols, and Ranjith Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In _Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology_, UIST '17, pages 845-854, New York, NY, USA. Association for Computing Machinery.
* Google (2023) Gemini Team Google. 2023. Gemini: A family of highly capable multimodal models. _arXiv preprint arXiv:2312.11805_.
* Gurari et al. (2018) Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018. VizWiz Grand Challenge: Answering Visual Questions from Blind People. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 3608-3617, Salt Lake City, UT, USA. IEEE.
* Han et al. (2023) Xiaotian Han, Quanzeg You, Yongfei Liu, Wentao Chen, Huangjie Zheng, Khalil Mrini, Xudong Lin, Yiqi Wang, Bohan Zhai, Jianbo Yuan, Heng Wang, and Hongxia Yang. 2023. InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models. _Preprint_, arXiv:2311.11567.
* Harley et al. (2015) Adam W. Harley, Alex Ulfkes, and Konstantinos G. Derpanis. 2015. Evaluation of deep convolutional nets for document image classification and retrieval. In _2015 13th International Conference on Document Analysis and Recognition (ICDAR)_, pages 991-995.
* Hong et al. (2023) Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2023. CoGgAgent: A visual language model for GUI agents. _arXiv preprint arXiv:2312.08914_.
* Huang et al. (2019) Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthennis Karatzas, Shijian Lu, and C. V. Jawahar. 2019. ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction. In _2019 International Conference on Document Analysis and Recognition (ICDAR)_, pages 1516-1520.
* Jaume et al. (2019) Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019. FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents. _Preprint_, arXiv:1905.13538.
* Kafle et al. (2018) Kwalsh Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018. DVQA: Understanding Data Visualizations via Question Answering. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 5648-5656, Salt Lake City, UT. IEEE.
* Kahou et al. (2017) Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. 2017. FigureQA: An annotated figure dataset for visual reasoning. _arXiv preprint arXiv:1710.07300_.
* Koh et al. (2024) Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. _arXiv preprint arXiv:2401.13649_.
* Kumar et al. (2024a) Aviral Kumar, Vincent Zhuang, Rishbha Agarwal, Yi Su, John D. Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M. Zhang, Kay McKinney, Disha Shirvastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. 2024a. Training Language Models to Self-Correct via Reinforcement Learning. _Preprint_, arXiv:2409.12917.
* Lewis et al. (2006) D. Lewis, G. Agam, S. Argamon, O. Frieder, D. Grossman, and J. Heard. 2006. Building a test collection for complex document information processing. In _Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '06, pages 665-666, New York, NY, USA. Association for Computing Machinery.
* Li et al. (2022) Gang Li, Gilles Baechler, Manuel Tragut, and Yang Li. 2022. Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale. In _CHI Conference on Human Factors in Computing Systems_, pages 1-13, New Orleans LA USA. ACM.
* Li et al. (2020a) Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020a. Mapping Natural Language Instructions to Mobile UI Action Sequences. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8198-8210, Online. Association for Computational Linguistics.
* Li et al. (2020b) Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. 2020b. Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5495-5510, Online. Association for Computational Linguistics.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Liu et al. (2024a) Jungpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. 2024a. VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding? _Preprint_, arXiv:2404.05955.
* Liu et al. (2024b) Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. 2024b. TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document. _Preprint_, arXiv:2403.04473.
* Lu et al. (2024c) Yuwen Lu, Yuewen Yang, Qinyi Zhao, Chengzhi Zhang, and Toby Jia-Jun Li. 2024c. AI Assistance for 


* Masry et al. (2022) Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. _arXiv preprint arXiv:2203.10244_.
* Mathew et al. (2022) Minesh Mathew, Viraj Bagal, Ruben Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. 2022. Infographicvqa. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_, pages 1697-1706.
* Mathew et al. (2021) Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. DocVQA: A dataset for vqa on document images. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_, pages 2200-2209.
* Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tsushar Khot, and Ashish Sabharwal. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2381-2391, Brussels, Belgium. Association for Computational Linguistics.
* Mishra et al. (2019) Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. OCR-VQA: Visual question answering by reading text in images. In _2019 International Conference on Document Analysis and Recognition (ICDAR)_, pages 947-952. IEEE.
* Openi et al. (2021) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ajeman, Diogo Almeida, Janko Alteschmidt, Sam Altman, Shyami Andakat, Red Avila, Igor Babuschkin, Sucht Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosise Campbell, Andrew Cann, Brittaury Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yuxnia Dai, Cory Decareaux, Thomas Degery, Noual Deutsch, Daminen Deville, Arka Dhar, David Dolan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simon Posada Fishman, Juston Forte, Isabella Fullford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goloh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidelberg, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhu Jin, Denny Jin, Shinno Jonto, Billie Jorn, Hewoo Jun, Tomer Kaftaukas Zaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Arts Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jane Leibke, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Lalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Melby, Ashvin Nair, Reitichiro Nakano, Rajeev Nagak, Arvind Neelakanta, Richard Ngo, Heyounwo Nob, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emyn Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbutte Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokromy, Michelle Polcars, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prechl, Raul Puri, Alec Radford, Jack Rao, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsm, Kyla Sheppard, Toki Sherbakov, Jessica Shield, Sarah Shocker, Pranav Shyam, Symon Sidor, Eric Sigler, Madele Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonehan, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Ceron Uribe, Andrea Vallone, Arun Vijayergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Weimher, Jiayi Weng, Lilian Weng, Matt Witrighthor, Dave Willmer, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. GPT-4 technical report. _Preprint_, arXiv:2303.08774.
* Owen (2024) David Owen. 2024. How predictable is language model benchmark performance? _Preprint_, arXiv:2401.04757 

Siyang Qin, Alessandro Bissacco, Michalis Raptis, Yasuhisa Fujii, and Ying Xiao. 2019. Towards unconstrained end-to-end text spotting. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
* Short Papers)_, pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
* Rajpurkar et al. [2016] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Rawles et al. [2024] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2024. Android in the wild: A large-scale dataset for android device control. In _Proceedings of the 37th International Conference on Neural Information Processing Systems_, NIPS '23, pages 59708-59728, Red Hook, NY, USA. Curran Associates Inc.
* Rezatofighi et al. [2019] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. 2019. Generalized intersection over union: A metric and a loss for bounding box regression. _Preprint_, arXiv:1902.09630.
* Singh et al. [2019]Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 8317-8326.
* Snell et al. [2024] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. _Preprint_, arXiv:2408.03314.
* Sunkara et al. [2022] Srinivas K. Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Jindong Chen, Abhanshu Sharma, and James W. Stout. 2022. Towards Better Semantic Understanding of Mobile Interfaces. In _Proceedings of the 30th International Conference on Computational Linguistics_, Oyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Talmor and Berant [2018] Alon Talmor and Jonathan Berant. 2018. The Web as a Knowledge-Base for Answering Complex Questions. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 641-651, New Orleans, Louisiana. Association for Computational Linguistics.
* Talmor et al. [2019] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
* Wang et al. [2021] Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021. Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning. In _The 34th Annual ACM Symposium on User Interface Software and Technology_, UIST '21, pages 498-510, New York, NY, USA. Association for Computing Machinery.
* Wang et al. [2024] Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jinbo Yuan, Quanzeng You, and Hongxia Yang. 2024. Exploring the Reasoning Ability of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning. _Preprint_, arXiv:2401.06805.
* Wu et al. [2023] Jason Wu, Siyan Wang, Siman Shen, Yi-Hao Peng, Jeffrey Nichols, and Jeffrey P Bigham. 2023. Webui: A dataset for enhancing visual ui understanding with web semantics. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_, pages 1-14.
* Wu et al. [2024] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2024. An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models. _Preprint_, arXiv:2408.00724.
* Xie et al. [2023] Jinheng Xie, Kai Ye, Yudong Li, Yuexiang Li, Kevin Qinghong Lin, Yefeng Zheng, Linlin Shen, and Mike Zheng Shou. 2023. Learning Visual Prior via Generative Pre-Training. _Advances in Neural Information Processing Systems_, 36:70562-70580.
* Xiong et al. [2019] Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulkarni, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. TWEETQA: A Social Media Focused Question Answering Dataset. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5020-5031, Florence, Italy. Association for Computational Linguistics.
* Yang et al. [2015] Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiqQA: A Challenge Dataset for Open-Domain Question Answering. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 2013-2018, Lisbon, Portugal. Association for Computational Linguistics.
* Yang et al. [2018] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for 

Diverse, Explainable Multi-hop Question Answering. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
* Yu et al. (2020) Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning. _Preprint_, arXiv:2002.04326.
* Yue et al. (2024) Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Boitao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. _Preprint_, arXiv:2311.16502.
* Zang et al. (2021) Xiaoxue Zang, Ying Xu, and Jindong Chen. 2021. Multimodal icon annotation for mobile applications. In _Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction_, pages 1-11.
* Zhang et al. (2024) Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. 2024. When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method. _Preprint_, arXiv:2402.17193.

## Appendix A Data Annotation Details

### VH Out-of-Sync Rules

We asked annotators to mark screenshots which are 1) from non-English apps and 2) not synchronized with VHs, as described in Section 4.1. The out-of-sync symptoms are oftentimes harder to distinguished than expected. UI elements may be occluded (Figure 3(a)) and "ghosting" VMs appear in addition to the in-sync VHs because of UI animation effects such as menus sliding/popping in and out (Figure 3(b)). These two cases are deemed to be acceptable in-sync scenarios as a user or an annotator can still select the correct bounding boxes from VH. However, the example in Figure 3(c) is an out-of-sync case, in which the text "No Alarms. Set an alarm and start your ..." is not supported by a VH bounding box. All the other bounding boxes that appear on the screen are all irrelevant to the current main content, hence, determined as an out-of-sync case. We trained our annotators to distinguish these cases and only mark the third case to remove.

### Question Annotation UI

The question annotation interface is shown in Figure 4(a). Question annotation was performed in a sequential manner by multiple annotators. An annotator can see all previous questions to diversify question framing and avoid duplication. We also used the same sequential process to provide more feedback and training to the annotators for quality improvement.

### Answer Annotation UI

The answer annotation interface is shown in Figure 4(b). Answer annotators were tasked to determine if the question is valid and if the question is answerable from the screen context. If both are positive, the annotators need to answer the questions by 1) selecting or drawing the bounding boxes of UI elements, 2) filling the text for each selected/drawn bounding box on the right, 3) ranking them appropriately, 4) providing the full-sentence answer to the question. The annotators were also tasked to review and make necessary corrections if the question has grammatical errors or typos.

### Annotation quality control

We leveraged a rating platform for collecting human annotations to ensure that raters go through 

a training phase, and proceed to collecting annotations only after reaching 90% quality bar. Furthermore, output quality was monitored throughout the annotation process, consistently reaching more than 95%.

### Annotation post-processing

Aside from removing answer annotations marked as "invalid question" and questions with no remaining answer annotation, post-processing of the records in the train split included applying question modification, if there was one provided during answer annotation.

The same could not be easily done for validation and test splits, as those contained 3 answer annotations per question, and corresponding question modifications could vary. The inter-annotator agreement was therefore estimated. The 2 answer annotations were considered to be in agreement if any of the following conditions apply:

* The questions (after modifications if some were provided) are the same.
* The full-sentence answers are the same.
* The lists of bounding boxes that constitute the answer contain the same elements, ignoring permutations.
* Both answer annotations are in agreement with the same other answer annotation.

Here two bounding boxes correspond to the same element if they correspond to the same VH node, or if they intersect and have the same textual content. As a result, 97.9% of the questions in validation and test splits had full agreement of the answer annotations, 1.9% had partial agreement (2 out of 3), and 0.2% had no agreement.

We then removed the disagreeing answer annotations from questions with partial agreement, and questions with no agreement. And applied question modification with the biggest consensus (2-3 out of 3), or chose the latest if all modifications are different.

### Short Answer Generation Prompts

We describe below the prompts used in PaLM 2 Anil et al. (2023) to generate short answers for ScreenQA, as described in Section 4.5. ScreenQA dataset annotations (question, list of text of UI elements, and full-sentence answer) were used as input. To improve the prompt quality, we used an identical prompt template with two sets of few-shot examples for answers with 1) one or 2) more than one UI elements involved, each set of which is used and inserted into the template for the example with answers that utilize the corresponding number of UI elements. The prompt template is give below:

Figure 4: View hierarchies (VHs) are overlaid on the screenshots with class names and the first few characters printed to assist annotators to determine whether the VHs for the main contents are in sync.

 

List various ways to rephrase the answer. The answer should be as short as possible, without extra words from the question. Use all provided elements in each answer. Provide the output in square brackets. {examples} Now is your turn. Question: [question] Answer elements: (list of text of UI elements) Full answer: (full-sentence answer) Rephrases: An example of single-UI-element answer is as below: Here is another example: Question: "What is the gender? Answer elements: ['Rule'] Full answer: 'The gender is male.' Rephrases: ['male'] An example of multiple-UI-element answer is as below: Here is another example: Question: "What is the name? Answer elements: ['Jon', ''Brown'] Full answer: 'The name is Jon Brown.' Rephrases: ['Jon Brown']

## Appendix B Data Examples

Table 9 presents a few examples from the ScreenQA dataset. Each example contains

* A screenshot
* A question
* Multiple annotations of lists of UI elements relevant to the question, each annotation of which is completed by an annotator
* Multiple annotations of long full-sentence answers corresponding to the UI element annotations
* Multiple short answers, generated by the procedure outlined in Section 4.5 Note that the bounding boxes of selected UI elements are highlighted on the screenshot for the illustrated purposes of the corresponding annotated bounding boxes, but they are not present in the corresponding image from RICO Deka et al. (2017) or during our data annotation process outlined in Section 4 

* [14] M. C. Bader, A. D. Kessler, and R. J. Sorensen. The role of the user's guide for a new generation of wireless networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 235-238, 2009.
* [15] P. L. Barnett, G. E. Hinton, and I. Goodfellow. Generative adversarial nets. In _Advances in neural information processing systems_, volume 67. Springer, 2016.
* [16] T. H. Nguyen, Y. Bengio, and Z. Chen. Learning deep convolutional neural networks with applications to image recognition. In _NIPS_, pages 1332-1339, 2015.
* [17] W. Dong, X. Li, and Q. Weinberger. Deep residual learning for image classification. In _IEEE International Conference on Acoustics and Speech Processing Systems_, pages 1332-1339, 2015.

[MISSING_PAGE_POST]

 

* [198] J. M. C. Collins, R. S. Thorne and D. W. Duke, Phys. Rev. Lett. **63**, 2457 (1990)
* [199] A. K. Leibovich and V. I. Zakharov, Nucl. Phys. B **289**, 1 (1984)
* [199] P. F. Harrison and H. E. Haber, Phys. Rev. Lett. **64**, 1018 (1990)
* [200] G. L. Kane, Phys. Rev. Lett. **65**, 1018 (1990)

[MISSING_PAGE_POST]

 

* [36] M. C. Bader, R. J. Brouwer, and D. A. Kessler, "A new approach to the design of a smart device for smart devices in smart devices: The case of smart devices with applications in smart devices: The case of smart devices with application in 

Dataset Annotation Analysis

Additional dataset annotation analysis is provided in this section. Table 10 shows the remaining question categories and examples continuing from Table 4. Figure 6 shows distribution of question types (e.g., Wh- questions, Yes/No questions, etc.), regardless of the subject.

## Appendix D Evaluation Configurations

In Section 6 we report the performance of various models on ScreenQA tasks in a zero-shot setting and after fine-tuning. We provide additional details as to how we evaluated the baselines for the corresponding model sizes.

### Zero-shot

Question answering is one of the most common tasks for LLMs and VLMs. Since the output format of the _SQA-S_ task is similar to other existing benchmarks, we attempt evaluating publicly available models in zero-shot setting. Specifically, we focus our evaluations on Fuyu-8b5[23], Gemini 1.5 Flash6, Gemini 1.5 Pro7[19], and GPT-4o8[20]. We refer to Table 5 for the results.

Footnote 5: Fuyu-8b 6 Gemini 1.5 Flash 7 Gemini 1.5 Pro 8 GPT-4o 9 See Fuyu-8B discussion

Further, we describe for each model the prompt used in the evaluation is an instruction followed by a question. Fuyu-8B model came with an author recommendation for a prompt, which we made use of 9:

Answer the following DocVQA question based on the image. \n For GPT-4o we optimized the prompt using 10 examples from the validation split. The one we picked was:

Answer the question based on the screenshot only. Do not use any other sources of information. The answer should be succinct and as short as possible. If the answer is a text from the image, provide it exactly without rephrasing or augmenting. If there is no answer on the image, output "<no answer>*.n The same prompt was re-used for the Gemini model evaluation. Although this may represent a disadvantage for Gemini compared to GPT-4o, our results indicate that Gemini 1.5 Flash is on par with and 1.5 Pro outperforms GPT-4o using it.

### Fine-tuning

The training data for SQA-L, SQA-UIC and SQA-UIC-BB tasks has one annotation per example (with a few exceptions), so it's usage for fine-tuning is straightforward for all models. Short answers, on the other hand, were generated automatically (see Section 4.5), resulting in multiple answers per question for the SQA-S task. The fine-tuning setup for PaliGemma 3B[24] and ScreenAI 5B[24] models randomly select one answer each time. Through sufficient epoch, it is however likely that all answer variants are utilized. For Gemini 1.5 Flash fine-tuning on SQA-S task, however, one answer per question was randomly selected before training, limiting the diversity of answers observed by the model. We therefore ran fine-tuning twice for 2 different random selections, and reported the best obtained results among the two experiments.

## Appendix E Observed prediction errors on SQA-S task

While Section 6 contains evaluations of different models in different settings to provide baselines, in this paper we deliberately refrain from making conclusions about upslides and downshides of each individual model. Instead, in Table 11 we provide examples from SQA-S task of some errors models made during evaluation, as a demonstration of complexity of this dataset. While this may not be a complete list, we categorized those errors into:

* _Misinterpreted question_: the predicted answer answers a different question.
* _Misinterpreted screenshot info_: the predicted answer is based on the screenshot information that has a different meaning.
* _Hallucination_: the predicted answer is not based on the screen at all.
* _Modified or misread content_: the predicted answer originates from correct answer but contains typos or is incorrectly transformed into something different.
* _Jack of understanding/reasoning to identify/compose answer_: the predicted answer suggests poor model's screen understanding or reasoning capabilities.

 



\begin{table}
\begin{tabular}{l l l l} \hline Category & \% & Examples \\ \hline Subtotal from Table 4 & 83.8 & \\ \hline Signup/login & 1.6 & Which application can be used to sign up / login? & What are the alternative choices for signing up? \\ Version information & 1.6 & What is the version number? & What is the new feature in version v3.1.3? \\ Weather & 1.5 & What is the range of temperature shown on Sunday? & What is the weather forecast for Sunday? \\ Score \& value & 1.4 & What is height/weight of the person? \\ Yes/No & 1.1 & Is there any travel plans? & Is there any favorite? \\ Phone number & 1.0 & What is the phone number? & What is the prefix for the international mobile number? \\ \# of Stars & 0.8 & What is the star rating? & How many stars are given to the product? \\ Share/sharing & 0.8 & Which application can be used to share? & Where can I share this application? \\ Age & 0.8 & How old s...? & What is the age? \\ Percentage & 0.7 & What is the percentage of ... ? & What is the brightness percentage for foreground? \\ Settings & 0.6 & What is the setting of ... ? & Which settings are switched on? \\ Quantity amount & 0.6 & How much it has a timer? & What is the amount? \\ Permission & 0.5 & Which application is asking for permissions? & What permissions are required for MyCarTracks? \\ \# of Likes & 0.5 & How many likes for ... ? & How many likes does ... get? \\ Country & 0.5 & What is the name of the country? & Which country has the +54 code? \\ Distance & 0.5 & What is the visibility distance? & How far is it room ... ? \\ \# of Reviews & 0.4 & What is the number of comments on ... ? & How many comments? \\ Website & 0.3 & What is the url? & What’s the website address? \\ Gender & 0.3 & What is the gender? & Which gender is displayed on the screen? \\ How to & 0.3 & How to start on boot? & How to pronounce his name? \\ Currency & 0.3 & What is the currency? & What is the currency for the price? \\ Unit of measurement & 0.2 & What is the unit of temperature? & What is the unit of weight and length? \\ Language & 0.1 & Which language is used in the setting? & Which language is being translated into which language? \\ Color & 0.0 & What is the UI color? & What is the amount of green color? \\ \hline Total & 100.0 & \\ \end{tabular}
\end{table}
Table 10: Remaining question categories and examples (_cont._ from Table 4)

Figure 6: Histogram for the types of questions.



* [14] M. C. Bender, R. J. Keenner, and A. D. Murray, "A new approach to the design of a single-layer model for the transmission of a single layer with an arbitrary number of layers," in _Proceedings of the IEEE Conference on Decision and Control_, pp. 235-260.

[MISSING_PAGE_POST]

 

* [16] M. C. B. Anderson, J. A. Harvey, and R. D. Murray, "The statistical significance of the detection of a single-dish survey in the sky," _Journal of Geophysical Research: Space Science_**20**(4), no. 3, pp. 579-584, 1996.
* [17] S. G. Wilson, "A new method for detecting the presence of an unknown source at the time of observation," _IEEE Transactions on Geoscience and Remote Sensing_**20**(1), no. 1, pp. 1-10, 1978.

[MISSING_PAGE_POST]

 

* [14] M. C. Bock, J. A. Harvey, and R. H. Wilson, "The New Physics of the Universe: The Case of the Universe," _Physics Reports_**206**(3), 579-682, 1997.
* [15] D. W. Anderson, "A new approach to cosmology in cosmology," _Annals Phys._**198**(1), 103-126, 1998.
* [16] S. Weinberg, "On the nature of the universe," _Nature_**190**(1), 103-118, 1982.
* [17] G. L. Kane, "The Early Universe," _Astrophys. J._**198**(1), 103-118, 1983.
* [18] E. Witten, "Superstring Theory," _JHEP_**19**(1), 043, 1998.
* [19] P. N. Argyros, "An Introduction to Cosmology," _Physica D_**198**(1), 1-13, 1984.

[MISSING_PAGE_POST]

 

* [145] M. C. Davis, R. J. Reynolds, and D. A. Smith, "The New England Journal of the American Statistical Association," _Journal of Geophysical Research_**20**(3):179-186, 1979.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l l p{142.3pt}}  & Question & How many minutes does a beginner have to do the breathing basics session? \\ Correct answer & 10 to 15 minutes \\ Incorrect answer & 10 minutes and 15 minutes \\ Error category & Lack of understanding/reasoning to identify/compose answer \\ Explanation & The UI indicates that the breathing session is ”10/15 min”, which from the context should be interpreted as 10 to 15 mins, rather than 10 and 15 mins, separately. \\ \end{tabular}
\end{table}
Table 11: Examples of errors made by the models from ScreenQA dataset on SQA-S task.