Harnessing Webpage UIs for Text-Rich Visual Understanding
Authors: Junpeng Liu
Tianyue Ou
Yifan Song
Yuxiao Qu
Wai Lam
Chenyan Xiong
Wenhu Chen
Graham Neubig
Xiang Yue
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal
Instruction-tuning
Large Language Model
Primary area: datasets and benchmarks
TL;DR: Leveraging structured webpage UIs, the MultiUI dataset enhances multimodal large language models’ ability to interpret text-rich visual environments, overcoming challenges in existing methods and boosting performance across diverse tasks.

Abstract: Text-rich visual understanding—the ability to interpret both textual content and visual elements within a scene—is crucial for multimodal large language models (MLLMs) to effectively interact with structured environments. We propose leveraging webpage UIs as a naturally structured and diverse data source to enhance MLLMs’ capabilities in this area. Existing approaches, such as rule-based extraction, multimodal model captioning, and rigid HTML parsing, are hindered by issues like noise, hallucinations, and limited generalization. To overcome these challenges, we introduce MultiUI, a dataset of 7.3 million samples spanning various UI types and tasks, structured using enhanced accessibility trees and task taxonomies. By scaling multimodal instructions from web UIs through LLMs, our dataset enhances generalization beyond web domains, significantly improving performance in document understanding, GUI comprehension, grounding, and advanced agent tasks. This demonstrates the potential of structured web data to elevate MLLMs’ proficiency in processing text-rich visual environments and generalizing across domains.

Published as a conference paper at ICLR 2025
HARNESSING WEBPAGE UIS FOR TEXT-RICH VISUAL UNDERSTANDING
Jumpeng Liu, Tianyue Ou, Yifan Song; Yuxiao Qu,
Wai Lam, % Chenyan Xiong, *eenhu Chen, 9 Graham Neubig, 9 Xiang Yue
Carnegie Mellon University, Thhe Chinese University of Hong Kong Peking University, 4 University of Waterloo
jpliue ink. cuhk. edu.hk. hk xyue20 cma. edu
ABSTRACT
Text-rich visual understanding- the ability to process environments where dense textual content is integrated with visuals- crucial for multimodal large lan- guage models (MLLMs) to interact effectively with structured environments. To enhance this capability, we propose synthesizing general multimodal instructions from webpage UIs using text-based large language models (LLMs). Despite lack- ing direct visual input, text-based LLMs are able to process structured text rep- resentations from webpage accessibility trees. These instructions then paired with UI screenshots to train multimodal models. We introduce MultiUI, a dataset containing 7.3 million samples from million websites, covering diverse multi- modal tasks and UI layouts. Models trained on MultiUI not only excel in web UI acckevinhiiiing up 48% improvement on BehBeeh and a 19.1% boost in element accuracy surprisingly well to non-web UI tasks and even to non-UI domains, such as doc- ument understanding, OCR, and chart interpretation. These results highlight the broad applicability of web UI data for advancing text-rich visual understanding across various scenarios.
web agent dataset Mind2Web- but also generalize
INTRODUCTION
Text-rich visual understanding, the ability to interpret environments where textual content is densely intertwined with visual elements, is crucial cognitive skill in humans. For multimodal large lan- guage models (MLLMs) (OpenAl 2023] Liu et al. 2024b), replicating this ability essential for tasks that involve complex text-visual interactions, such as document processing (Mathew et al.) 2021 Singh et al.) 2019 Liu et al. 2023c), web navigation (Liu et al.) 2024c| Deng et al.) 2023), chart interpretation (Masry et al. (022) and text-rich visual reasoning Yue et al. 2024b). These
tasks require models to integrate dense textual information with surrounding visuals, enabling AI
systems to interact intelligently with the increasingly text-rich digital landscape (Koh et al., 2024..
To advance text-rich visual understanding in MILMs, we propose leveraging webpage Uls as a naturally structured, diverse, and text-dense data source. Web UIs, where textual content is often central and tightly integrated with visual elements and interactivity, offer an ideal setting for training models to interpret and navigate complex text-visual interactions.
Existing approaches to using web content in multimodal models have limitations (Fiuure 1). Rule- based extraction of images and their surrounding text (Zhu al.) 2023) Schuhmann et 2021 2022) often introduces noise and lacks contextual depth. Converting screenshots into simplified HTML structures (Lee et al., 2023 Gao al., 2024) imposes a rigid format that limits generaliza- tion across domains. Models like GPT-4 generate captions (Chen et al., 2023) for web images but frequently overlook the rich interaction between text and visuals.
Our approach addresses these limitations by synthesizing general multimodal instructions from web- puge UIs using text-based LLMs. Although text-based LLMs lack direct visual input, they can
"Co-second author. Order determined by dice roll.
'Corresooning Author.

---

Published as a conference paper at ICLR 2025
Previous Methods
MuleUT (Ours)
Parse
Caption
Synthesize
HTML
Extract
Image All Text
Doxsittion Top tewe
Serwen Capture Ned
Image Only
Structunne Page
Mull-lask
Rule-based extraction of image and all text
Screenshot parsed HTML for pre-training only
Commercial-modelee
MultiUT: multi-takk data synthesis with full siructured webpage Uls
image only captioning
Figure 1: MultiUI compared with previous methods. Our proposed MultiUI construction ap- proach synthesizes full structured webpage UIs into multimodal instruction samples of versatile tasks by harnessing powerful LLMs, which leads to more generalizable training samples.
effectively process the textual representations of webpages. By reading the cleaned accessibility tree-a a structured and refined representation of a webpage's HTML and metadata- meaningful instructions that capture both the content and interactions present on the page. These generated instructions are then paired with UI screenshots to train multimodal models, allowing them to learn from both text and visual representations.
generate
To facilitate this, we introduce MultiUU, an open-source dataset containing 7.3 million samples span- ning million websites and various visual understanding tasks. Our pipeline captures key web elements and layout structures using screenshots and enhanced accessibility trees, filtering out irrel- evant data while preserving the core structure of web Uls. Our task taxonomy, which covers three categories and nine tasks, ensures that models trained on MultiUI generalize across wide range of multimodal interactions. Additionally, we introduce variations in device types, aspect ratios, and question formats to further increase the dataset's diversity and enhance model robustness.
Our experiments show that training on MultiUI significantly improves model performance in both Ul-related and general multimodal tasks. Notably, models trained on MultiUl achieved up a 48% improvement on VisualWebBeenc (Liu et al.] (224c) and a 19.1% increase in element accuracy on Mind2Web (Deng al., (023). More surprisingly. we observed that this training generalizes to non-UI domains, resulting in improved performance in document understanding (Mathew et al.) 2021]. OCR (Singh al. 2019 Liu al. 2023c). and chart interpretation (Masry al. 2022) tasks outperforming even models specialized in these areas. These findings underscore the broader utility of web UI data as a powerful resource for improving text-rich visual understanding, enabling models to excel not only in UI tasks but across a diverse range of scenarios, including more complex agent tasks and non-Ul domains.
2 DATASET CONSTRUCTION
In this section, we outline the process of constructing MultiUI. We developed automated data collection pipeline by leveraging accessibility trees" and off-the-shelf LLMs. As illustrated in Fig- ure 2] construct the data collection pipeline through four stages: (1) raw website data scraping, (2) website curation, (3) task extraction from scraped websites, and (4) instruction construction.
2.1 WEBSITE RAW DATA SCRAPING
We begin by constructing a web raw dataset that includes HTML/CSS, high-resolution screenshots. and accessibility trees. The URLs in the "CC-MAIN-2024-10" dump from FineWeb (Penedo al. 2024] are used to render websites via Playwrigh To prevent any potential contamination. removed URLs that also appear in downstream benchmarks, such as VisualWebBench (Liu et al. 2024ch. We employ the accessibility tree (refer to Appendix B for example) as the text represen- tation of a webpage for further utilization by LLMs because it typically provides a more compact structured representation compared to the raw HTML. The accessibility tree focuses on the most important visual elements, such as buttons, links, and headings, which are crucial for understanding the content and funetionality of the page. excludes con-essential elements like those related to styling or purely decorative purposes, which are often present in HTML but do not contribute to the
http://dwwbboamoooooiaaaall/llle tree
http://ghhha.ccommmicco/////yyyh
2

---

Published as a conference paper at ICLR 2025
Website Scraping
Task Extraction
Generalization
COMMON CRAWL
MultiUl: 7:3M Web UI Instructions
GUI Understanding
Visual Understadingg Reasoning
GPT-4o mini
Webpage Caption
Webpago QA
Screenshots
Enbeadded Image QA
Action Prediction
GUl Agen!
Maywright
Ulmm--3700
Image Caption
Llama-3-708
Text Recegnition
Grounding
Earment OCR
Action Grounding
Accessibiiity Tree
Doe'hhart Unddestanding
Heading
Ekenent Grounding
Descriptions
+ Frompts
200 Diverse Prompt Templates
Figure 2: Construction pipeline of MultiUI. The process consists of four main stages: (1) Website Scraping: (2) Website Curation with hama-3-70b-Instrucee (3) Task Extraction utilizing Llama-3- 70b-Instruce GPT-4o mini, and rule-based approaches to generate Web UI tasks across three cat- visual understanding and reasoning, text recognition, and grounding; (4) For each task, generate tasks samples by applying the diverse instruction templates paraphrased by GPT-4o.
egories:
core information. This higher information density in the accessibility tree makes it more efficient and relevant input for language model processing, allowing the model to focus on key content with- data processing,
unnecessary noise. For details websites were crawled. To ensure comprehensive coverage across diverse platforms and window size variations, we rendered websites on two simulated devices (Windows 10 and iPhone 12 Pro) using dynamic window size settings.
Appendix [A] In total, 1.1 million
2.2 WEBSITE CURATION
Despite Fine filtering, our crawled raw data still contains instances inappropriate content and network errors. To address this issue, we employ additional processing step using filter language model. Specifically, we utilize the 1lama-3-70B-Instruu (Dubey et al., 2024) as our filter model. It analyzes the accessibility tree of each website identify problematic content, includ- ing adult material, gambling-related content, violence, discriminatory language, and network errors such "403 Forbidden", "502 Bad Gateway", "Cloudflare blocking". "blank pages", and "404 Not Found" errors. Websites flagged as problematic are subsequently removed from the dataset. The prompt used by filter model can be found in Appendix C After this step, approximately 10% of the raw websites were identified as harmful or invalid and subsequently removed, resulting in a raw dataset million website instances.
2.3
TASK EXTRACTION FROM SCRAPED WEBSITES
Existing approaches using web content in multimodal models have limitations (Figure 1). Tradi- tional rule-based methods for extracting images and surrounding text often introduce noise and lack contextual depth (Zhu et al. 2023; Schuhmann al., 2021 2022). while converting screenshots into simplified HTML structures imposes rigid formats Leee et al. 2023 Gao et al. 2024), limiting generalization across domains. Additionally, methods that rely on GPT- generate captions for
web images (Chen et al. 2023) tend to overlook the rich interaction between text and visuals. Our approach aims to overcome these challenges by synthesizing general multimodal instructions from webpage UIs using text-based LLMs.
To provide multimodality models with robust perception, understanding, grounding, and reasoning capabilities, construct diverse set of tasks, featured by their different focuses of abilities to in- teract with the web. These types tasks are crucial for effective web interaction, they are: (1) visual understanding and reasoning; (2) text recognition; (3) grounding. See Figure for overview of our constructed task samples. All prompts required in the sample construction process are metic- ulously refined through manual tuning, based observations of the generated samples. Refer to
3

---

Published as a conference paper at ICLR 2025
MultiUU
Heading OCR
Element OCR
Webpage Caption
Understand Reason
Question: Extract the main beading from the webpage. Answer: How to Get Bether at Disc Golf [Techniques & Tips)
Question: Please extract the text from the element
Question: Explain the webpage in detail. Answer: webpage appears website that discueses how to get better at disk golf.
Text Recognition
enclosed by the red pociangle. Answer: Best Disc Golf Putters
Visual Grounding
Element Grounding
Webpage QA
Question: Provide the bounding box coordinates of the element described: " "About Us",
Question: What the name the author of article displayed on the website? Answer Peter Howell.
Better Golf [Technigues &
cuordinates should formatted as lieft top, right, bottomn, with each number being foot between 0 andd 1. Answer: [0,624, 0.311, 0.769, 0.439]
Action Prediction
Question Select most suitable website that matches the new page after clicking the element in the red bounding box. A. Twitter Instagram Answer:
C. Youtube
Embedded Image Caption
Action Grounding
Question: Generate description of the image highlighted within the red barder.
HOW TO GET BETTER AT DISS GOLF
Question:
bounding box coorclinates of the element you need to click perform this learn more "perfect throwss". Answer: [0.624, 0.207, 0.769, 0:312)
Answer: person a blue jacket and yellow beanie throws disc low ards.....
Embedded Image QA
poot legh wo Lonta
Question: What the person throwing?
Answer. A disc.
Figure 3:
Task samples from MultiUI. To enhance multimodal models' perception, comprehen- sion, grounding, and reasoning capabilities, we have designed diverse set of nine tasks, emphasiz- ing the critical abilities for text-rich visual understanding scenarios.
Appendix for the complete set of prompts used for task extraction and Appendix for examples of created task samples.
2.3.1 VISUAL UNDERSTANDING AND REASONING TASKS
Our visual understanding and reasoning tasks are designed to improve the model's ability to describe both the overall structure of web pages and the specific visual elements within them (Captioning). meanwhile, enhancing the models' abilities to answer questions (QA), and predict functionalities of elements (Action Prediction) about webpages.
Webpage Captioning: comprehending and summarizing the overall content and structure of web page. The accessibility tree of each webpage serves as a concise textual representation that encodes the page's structural and semantic information. Therefore, we prompt Mama-3-70B-Innsie thesize the structured and informative accessibility tree into coherent and detailed descriptions.
to syn-
Wehpage QA: answering questions with respect to non-image content within webpages. Unlike Webpage Captioning, which provides a high-level summary of the entire page, this task targets de- tailed information extraction and reasoning about particular aspects of the page content. Given the accessibility tree, 1-ama-3-70B-nn the versatility and adaptability of this task, we design a dynamic in-context examples strategy and consider two kinds of styles of output answers. Specifically, a set of five questions from the LLaVA v1.5 instruction-tuning dataset are randomly sampled as in-context examples during each genera- tion. This strategy helps the model draw from a broad range of patterns, reducing redundancy and increasing variability in question types. Furthermore, answers are provided in two distinct formats: detailed conversational responses to accommodate scenarios that require contextual richness, and concise, direct answers for use cases where brevity is preferred. The dual-format answers construc- tion method ensures that the generated QA pairs versatile and adaptable to different user needs.
is prompted to generate question- -nnseer pairs. To improve
Embedded Image Captioning: deseribing embedded images within a web page. While the acces- sibility tree provides a useful overview of the textual and interactive components of webpage, it often falls short when dealing with embedded images. Deseriptions for embedded images are often inadequate, with brief or even missing alt-text being a common issue. To address this issue, we employ the GPT-4o-mini") to generate rich and context-sensitive captions for these embedded im-

---

Published as a conference paper at ICLR 2025
Visual Understanding and Reasoning
Grounding
Text Recognition
Platform
Total
Web Capt.
Img Capt.
Web QA
Img QA
Act. Pred.
Action
Elem.
Head
Elem.
Desktop
150K
526K
1.IM
979K
65K
1.2M
694K
98K
175K
5.0M
Mobile
100K
0
936K
0
34K
613K
488K
74K
41K
2.3M
Total
250K
526K
2.1M
979K
99K
1.8M
1.2M
172K
217K
7.3M
Table 1: Statistics of our dataset MultiUI.
ages. The model is instructed consider not only the visual features of the images but also their surrounding context provided by the accessibility tree, producing captions that are more informative and contextually relevant compared standard alt-text tags.
Embedded Image QA: answering questions with respect to embedded images within webpages. Different from Embedded Image Captioning which provides general descriptions of images in the context of the webpage, Embedded Image QA focuses on answering specific questions about the visual content, requiring more detailed analysis. The Llama-3-70B-Insse to generate QA pairs from corresponding captions. The same dynamic in-context examples strategy and dual-format answers generation method as Webpage QA are employed here.
model is also employed
predicting the outcome of clicking a specific element webpage. Follow- ing Liu et al. (2024c). this task aims to predict the title of the redirected webpage after clicking a specified element, under multiple-choice setting. Negative elements randomly selected from the same webpage as the positive element and the titles redirected sites are obtained through automated interaction using PlayWright
Action Prediction:
2.3.2 OCR TASKS.
Element OCR. In this process, the HTML DOM tree of the crawled webpages is traversed to identify elements with textual descriptions exceeding 20 words, which then utilized create OCR task samples. Each task sample consists of a screenshot of the webpage, with red bounding box highlighting the element targeted for OCR.
Heading OCR. Following Liu et al. (202cc), we incorporate Heading OCR to complement the El- ement OCR and provide a more comprehensive text recognition capability. It focuses on identifying and extracting the textual content of headlines or titles from web pages.
2.3.3 GROUNDING TASKS
Action Grounding: predicting the click position in response specific instruction, such "learn more about perfect throws" as illustrated in Figure 3 Action grounding is crucial for developing web agents capable performing actions autonomously based on user commands. Utilizing our processed accessibility data enhanced by the bounding boxes of elements, 1lama-3-70b-Instree is not only prompted to generate multiple grounding instructions but also provides the corresponding ground-truth bounding boxes.
identifying the coordinates an element based on its textual description. The training examples for this task created by extracting textual descriptions alongside their corresponding bounding boxes from the HTML DOM tree.
Element Grounding:
For both grounding tasks, implement two distinct settings: multi-choice and bounding box gen- cration. In the multi-choice setting. eight candidate bounding boxes are presented within an input sereenshot, and the model must select the correct based on the given element description or instruction. In the bounding box generation setting, the model directly predicts the coordinates of the target element.
2.4 INSTRUCTION TEMPLATE CONSTRUCTION
We further diversify the instruction templates through a prompt variation approach (Appendix J.
We ultimately curated dataset of 7.3 million web Ul-related samples in the form of VQA. covering nine tasks across perception, comprehension, grounding, and reasoning capabilities, which we refer as MultiUI. The statistics are shown in Table ann refer to Appendix for examples.
5

---

Published as a conference paper at ICLR 2025
3 EXPERIMENTAL SETUP
3.1 IMPLEMENTATION DETAILS
Model Architecture We developed UIX using Qwen2-7B-Instruct Yang al. 2024) as the primary LLM backbone. We also use ficuna-7B-vl.5 (Chiang et al., 2023) and Llama-3.1 3--BBB- Instruct (Meta 2024) as backbones to further verify the effectiveness of our dataset. See Appendix for details about our model architectures.
Training Strategy To develop MLLMs with comprchensive GUI knowledge while maintaining robust general multimodal capabilities, propose two-stage training pipeline for our UIX models.
GUI Knowledge Learning. In this stage, we fine-tune the model on 95% of MultiUI dataset to enhance its web/U1-related understanding capabilities. This stage is crucial for devel- oping the model's proficiency interpreting and interacting with GUI environments.
Stage I:
Stage 2: Visual Instruction Tuning. To help the model acquire robust general multimodal pabilities alongside enhanced GUI knowledge, we then continue the fine-tuning process using a combination of general visual instruction dataset (i.e., LLaVA-1.5 data" or LLaVA-NeXT data and the remaining 5% of the MultiUI data.
3.2 BENCHMARKS
We have selected diverse evaluation benchmarks, encompassing a variety of in-domain GUI sce- narios, out-of-domain OCR-related tasks, general multimodal tasks, agent benchmark, sess the models' capabilities in diverse multimodal tasks. See Appendix [F for more details of these benchmarks.
GUI-Related Tasks. We select VisualWebBench (Liu al. 2024c) WebSRC (Chen al. 2021
2020) for GUI understanding. We also include Bbox-version of element and action grounding subtasks in VisualWebBench (Liu al. 2024c), ScreenSpot (Cheng et al. 2024), RefExp (Wichers et al. 2018) for GUI grounding.
ScreenQA-Short (Hsiao et al. 2022), WidgetCap (Li et al.
OCR-Related Tasks.
DocVQA (Mathew et al.]
2021).
ChartQA Masry et al., 2022
TextVQA (Singh et al.
2019), InfoVQA (Mathew 2022
VisualMRC (Tanaka
202
OCRBench (Liu et al.
2023c).
General Multimodal Tasks.
MMMU (Yue et al. 2024a)
MMBench (Liu et al.
2023b) and
VQA-V2 (Goyal et al. (017). We also include RefCOCO+(REC) YYu al. 2016) to evaluate the grounding capability in natural image scenarios.
Agent Task. We employ Mind2Web (Deng et al. 2023 to train and evaluate the agent's capabil- ity of performing complex instructions real-world websites.
We evaluate models against various baselines, including LLaVA-115 series (Liu al. 2024a) and LLaVA-1.6 (NeXT) series (Liu et al. 2024b). To ensure a fair comparison, given the differ- ent backbones of UIX from the original LLaVi VA checkpoints, LLaVA-Vicuna, Llama3. 1, Qwen2, utilizing the same training data as LLaVA. We also compare our models with Pix2Struct (Lee al., (023), S4 (Gao et al. 2024), SeeClick (Cheng al. 2024). Co- gAgent (Hong et al. 2023). and ScreenAl (Baechler et al. 2024). Furthermore, we include GPT-4V and GPT-4o as strong basclines to provide comprehensive evaluation.
re-implemented three baselines:
EXPERIMENTAL RESULTS AND ANALYSIS
4.1 RESULTS GUI-RELATED TASKS
Significant Improvement on GUI Understanding and Grounding Benchmarks: We present performance on GUI understanding and grounding benchmarks in Table 2 Overall, our dataset
v1 mix665k.json

---

Published as a conference paper at ICLR 2025
GUI Understanding
GUI Grounding
Model
Visual WebBench
Web SRC
SQA Short
Widget
VWB
VWB Act-G
SSpol
RefExp
Cap
Ele-G
GPT-4V OpenAA 2023
64.6
0.2
0
16.2
GPT-4o UpenAl
18.3
Gemini 1.5 Pro Reld al. 2024
64.8
Pix2Struct [Lcc
2023
136.7s 7s
S4 (Goo et
61.1+
130.6
SeeCTick
hhnn cf 2al.22000
2024
9.7
53.4
[[ong CogAgent
el al.
2023
28.7
29.3
36,6
47.4
ScreenAl Bacchler ct al.
2024
87.2+
94.8-
156.4
Trained with LLaVA-Ja data
LLaVA-I.5-7B (Liu
2023a
17.0
30.9
42.6
20.0
0.7
0.0
0.6
0.4
L. Taa
19.4
32.5
46.0
10.2
0.0
0.0
0.9
1.1
LLaVA-Vicuna'
23.1
41.5
53.0
38.4
0.0
1.3
1.2
Trained with LLaVA-1.5 1.5 data MultiUI
UIX-Vicuna
71.1
69.5
73.9
66.5
55.5
26.7
44.7
35.8
A over LLAVA-Vicuma
+48.0
+28.0
+20.9
+28.1
+55.5
+26.7
+43.4
+34.6
Trained with LLa VA-NX- data
LLaVA-NeXT-7B
Liu
2023a
36.0
67.2
66.0
35.4
0.2
0.0
0.9
0.4
LLaVA-NeXT-13B
Liu
20258
39.4
71.2
68.3
23.4
0.0
1.0
0.4
0.0
LLaVA-NeXT:34B
Liu al
20258
50.5
83,2
74.0
46.3
1.7
3.0
2.8
3,4
LLVVA-NeX--B
Jiu al.
2024h
42.1
72.8
68.0
49.8
1.0
0.0
1.7
1.1
LLaVA-Llama3.
Lau
2024b
35.3
65.0
65.7
34.2
0.5
0.0
1.3
0.9
LLaVA-Qwen2!
Liu
20246
41.7
72.5
68.6
38.0
1.2
0.0
1.3
1.9
Trained with MultiU++ LLAVA-NeXT data
UIX-Llama3.
74.2
75.3
72.7
55.6
16.2
11.9
22.2
17.9
over LLaVA-Llama3.
+38.9
+10.3
+7.0
+21.4
+16.2
+11.9
+20.9
+17.0
UIX UIX-Qwen2-7B
75.9
82,9
78.8
72.7
66.1
35.6
55,2
43.5
A over LLaVA-Qwen2
+34.2
+10.4
+10.2
+34.7
+64.9
+35.6
+33.9
+41.6
Table 2: Results on GUI understanding and grounding benchmarks. indicates specific fine- tuning on the corresponding training set. denotes our re-implementation with the same backbone model architecture of UIX.
significantly improves model performances over all three backbones (i.e., LLaVA- -Vicna, LLaVA- Llama3.1, and LLaVA-Qwen2) as shown in light cyan rows. When training with LLaVA-1.5 data, the performance improvements on the nine benchmarks are all above 20%, up to 55.5%. The corre- sponding numbers training with LLaVA-1.6 data are 7.0% and 64.9%.
Superior Performance over Larger Models: In contrast, our UIX models, trained on the Mul- tiUI dataset, demonstrate substantial performance improvements across GUI-related tasks. Despite having relatively modest parameter count of 7B/SB, our models outperform baselines with signifi- cantly larger parameter counts, highlighting the efficacy of learning from text-rich web UI informa- tion. For instance, VisualWebBench. UIX-Qwen2 achieves the highest score (75.9), surpassing all other MLLMs, including LLaVA-1.6-34B (50.5) and even GPT-4V (64.6).
Enhanced GUI Grounding Capability: Meanwhile, we observe that, compared with general fig- ure grounding (Table 3 current MI.LMs fail to handle grounding tasks in complex GUI scenarios. Conversely, trained with MultiUI, which encompasses two web grounding tasks, UIX also demon- strates enhanced GUI grounding capability.
4.2 RESULTS GUI AGENT TASKS
To further validate the effectiveness of the learned web knowledge, we evaluated UIX on Mind2Web (Deng et al. 2023), web navigation GUI agent task.
Following previous work (Cheng et al. 2024), we use step-level success rate and element accuracy the metrics. We include SeeClick (Cheng al., (024) and CogAgent (Hong al.) 2023) as baselines, both of which are trained on Wen training set. See more experimental details in Appendix H

---

Published as a conference paper at ICLR 2025
General OCR/ DocQA / ChartQA
General Grounding
Model
Doc VQA
Chart QA
Text VQA
Info
Visual MRC
OCR
RefCOCO+
VQA
Rench
GPT-4V (OpenAl 2023
88.4
78.5
78
75.1
64.5
GPT-4o OpenAl
2024
92.8
77.4
79.2
73.6
Gemini 1.5 Pro [Reid et al. 2024
93.1
78.7
81.0
Claude-3.5 Sonnel
95.2
74.1
74.3
Pix2Struct (Lee ct al
2023
76.6
58.6
40
S4 [Gao et al. 2024
55.0
CogAgent (Hong et al. 2023
81.6
68.4
76.1
44.5
DocOwl-1.. Stt.w-1.--Chaai hat (HD ct al.
2024a
82.2
70.2
68.6
50.7
DocOw12 Hu al. 2024b
80.7
46.4
Trained with LLaVA-1A data
LLaVA-1.5-7B (Liu al.
2023a
28.1
18.1
46.0
25.8
35.3
31.3
50.0
LLaVA-I.5-13B Liu 20253
30.2
18.2
48.7
29.4
38.3
52.1
59.9
LLaVA-Vieuna
46.1
59.6
31.9
39.7
38.1
61.7
Trained with MultiUI LLaVA- 1.5 data
UIX-Vicuna
72.8
24.2
67.0
41.6
43.3
53.4
65.7
A over LLaVA-Vicume
+26.7
+3.0
+7.4
+9.7
+3.6
+15.3
+4.0
Trained with LLa VXT data
LLaVA-NeXT-7BB Liu al.
2023a
74.4
54.8
64.8
37.0
33.3
52.1
77.0
LLaVA-NeXT- 133
Tau al.
2023a
77.5
62.4
67.0
41.5
35.9
55.0
80.8
LLaVA-NeXT-34B Liu ct al.
2023a
83.9
69.4
51.3
37.9
57.2
84.8
LLVVA-NeXT-BB
(Luu
78.2
37.6
29.3
55.2
79.5
LLaVA-Llama3.1
74.7
66.5
64.3
35.7
46.8
54.0
74.8
LLaVA-Qwen2t
76.5
68.5
67.0
41.1
44.1
55.7
75.9
Trained with MultiU1 LLaVA-NeXI data
UIX- UIX-Llama3.1 over LLaVA-Llama3.1
78.0
66.9
65.1
44.2
49,7
58.6
71.7
+3.3
+8.5
+2.9
+4.6
-3.1
UIX-Qwen2
85.3
74.0
72.7
52.2
49.1
66.3
79.1
A over LLaVA-Qwen2
+8.8
+5.5
+5.7
+11.1
+5.0
+10.6
+3.2
Table 3: Results on general OCR/Doc/Chart related QA and grounding benchmarks. Bold text and underlined indicate the best-performing and the second-bes -bst models each group, respectively.
Mind2Web
Model
Cross-Task
Cross- Website
Cross-Domain
Step
Element Acc.
Step SR
Element Acc.
Step SR
Element Acc.
SceClick Cheng 2024
25.51
28.31
16.4
21.4
20.8
23.2
CogAgent Hong et 2023
26.9
30.2
23.4
27.3
28.5
33.1
LLaVA- -wee2
7.5
7.6
10.4
UIX XWeeQ
13.5
9.8
13.8
L.aVA-Qwen2-M22
20.4
24.3
14.3
20.1
16.4
20.0
UIX-Qwen2-M2W
38.2
43.4
31.0
A over LLaVA-Qwen2-MMW
+17.8
+19.1
+16.7
+19.1
+18.5
+20.4
Table 4: Performance a GUI agent task: Mind2Web. indicates numbers taken from (Cheng al. 2024). -M2W means further fine-tuning models on the Mind2Weh training set.
Superior Agent Performance compared with Larger Models: As shown in Table training on MultiUI improves our model's performance by up to 19. 19.1% element accuracy. After further train- ing on the Mind2Web training set, UIX-Qwen2-M2W outperforms other existing GUI agent models significantly, surpassing both SeeClick and CogAgent in step success rate and element accuracy across all three test subsets, while UIX(7B) is smaller than SeeClick(9.6B) and CogAgent(18BB in model size.
4.3 RESULTS ON GENERAL DOCUMENT UNDERSTANDING AND GROUNDING TASKS
Generalization to OCR/Doe/Chart Benchmarks: As illustrated in Table 3 training on MultiUI yields substantial improvements in out-of-domain OCR-related figure comprehension. Notably,
8

---

Published as a conference paper at ICLR 2025
90
+2.8
+1.6
+4.6
-02
Avg. Store
60
GUI Underdandiing
30
Geounding
+0.3
18-
Genert OhWtDoo
40
Geneall Groarding
VWB
Web6RC
SexceSppol
RefExp
DocVQA
CharIQA
Single-Stage
Two-Stage
KMultiUI Samples
Figure 4: Comparison between single-stage and two-stage training strategies.
Figure 5: Effect of scaling up sam- ple size.
outperforms both LLaVA-I.6-34B and DocOwl (Hu et al.) 2024a by. a model specif- ically designed for document understanding, across all evaluated OCR-related benchmarks. This success can be attributed to the diverse nature of MultiUI, which comprises an extensive collection of webpage screenshots, inherently covering wide spectrum of OCR-related and abstract figure comprehension tasks.
UIX-QwenZ
Transfer between Grounding Tasks: Furthermore, the incorporation of web grounding instruc- tion data during training demonstrates significant enhancement in general grounding performance. Vicunaa exhibits improvement of 4.0% RefCOCO+, when compared to models trained exclusively on general instruction data.
4.4 RESULTS ON GENERAL MULTIMODAL TASKS
Robust General Multimodal Capabilities: Here, we conduct an experiment to validate the efficacy our MultiUT in maintaining gen- eral multimodal capabilities. As shown in Ta- ble 5 our UIX model trained with the mix of LLaVA data and MultiUI performs on par with the counterparts trained with only general vi- sual instruction data. This demonstrates that our MultiUI dataset successfully enables mod- els acquire enhanced GUT knowledge alongside robust general multimodal capabilities simulta- neously.
Model
MMMU
MMBench
VQ4-V2
LLaVA-Llama3.1
38.8
72.0
80.0
U1X-Laama3.1
42.3
74.7
80.0
LLaVA-Qwent
44.7
76.5
81.6
UIX- UIX-Qwen2
41.8
77.4
82.1
Table 5: Performance on other general multi- modal benchmarks.
4.5 ABLATIONS
Two-stage training outperforms single-stage training We conducted a comparative experiment to evaluate the effectiveness of the two-stage training mechanism (Section 4.5 As illustrated in Figure 4 the two-stage strategy outperforms the single-stage approach both GUI and general scenarios. This result highlights the efficacy of the two-stage training strategy that achieves robust GUI knowledge while preserving strong general multimodal understanding capabilities.
Model performance improves with more MultiUI data ducted an analysis by gradually increasing the volume of data in MultiUI and evaluating the aggre- gated performance across the four task categories. As illustrated in Figure performance across all task categories generally improves as the data size increases. Specifically, GUI capabilities experi- ence the most substantial improvements with increased data volume. GUI Grounding, in particular, demonstrates a remarkable increase, rising dramatically from approximately 12% to 60%.
To assess the scaling effect, we con-
Different MultiUI tasks enhance unique abilities To study the impact of different task types in MultiUI, we trained models with data each individual task type and evaluated their performances. Figure 6 highlights several key findings from this study. In particular, both QA and Caption sam- ples enhance performance in GUI understanding and OCR-related benchmarks, while OCR samples benefit most in GUI understanding. More detailed analysis can be found in Appendix

---

Published as a conference paper at ICLR 2025
60
75
85
647
80
746
701
45-
750
60
14.0
30
80
793
71.4
75
76.
45
417
011
15
25
70-
30
13
06
70
65
VisualWiebBench
ScreenSpot
DocVQA
ReiCOCO-
Baseline
+QA
+Caption
+CCR
+Grounding
All
Figure Ablation study of training on four task types.
5 RELATED WORK
Web-Based Multimodal Pre-training Dataset Collection Web can be utilized as valuable re- source for collecting multimodal data for MLLM training, For example, the LAION series (Schuh- mann et al. 2021 2022) harvesting billions of image-text pairs from CommonCraw by extracting images with their corresponding alt-text tags from HTML. MMC4 (Zhu et al. 2023) and OBELICS
[Laurençon et al. 2023) focus on an interleaved image-text format that can accommodate multiple images within single instance. However, argue that these datasets do not make full use of web UI information, since they cannot capture the rich layout information and diverse UI elements of webpages. In this paper, we propose that the accessibility tree serves as a well-structured and in- formative textual representation of a webpage, enabling powerful LLMs to be harnessed to generate diverse multimodal instructional data that encompasses perception, understanding. reasoning, and grounding capabilities.
Recently, there have also been some datasets focusing on pre-training on the rich structure infor- mation of webpage, such as Pix2Struct (Lee et al., 2023) which involves parsing screenshots into simplified HTMI., and $4 (Gao et al. 2024) which focus collect multiple types training signals from the web. However, these methods either fall short in a specific task type so must be followed by further fine-tuning downstream tasks, or only focus on tasks in web scenarios and cannot easily generalize to domains beyond web pages and HTML.
MLLMs for GUI-related Multimodal Tasks Graphical user interfaces (GUIs) serve as a pivotal medium for human-computer interactions. Recently, MILMs have made significant strides in UI understanding, focusing on two key areas: information extraction from images and the development of foundation models for UI tasks. For information extraction, deep learning-based Optical Char- acter Recognition (OCR) has advanced text reading capabilities (Back et al., 2019). Concurrently, methods like Pix2Struct (Lee al. 2023) and DocOwl (Hu el. 2024a) have emerged for identify- ing and localizing UI elements within screenshots. Building on these element extraction techniques, several foundation models have demonstrated impressive comprehension abilities. Notable ex- amples include ScreenAl (Baechler et al. 2024). CogAgent (Hong et al. (003)), and Ferret-Ul (You) ict al., 2024). These models can perform tasks such as clement detection, captioning. and inter- action prediction using only screenshot inputs, without relying additional metadata like DOM structures. Concurrently, UGround (Gou et al. 2024) proposes a strong universal visual ground- ing model for GUI agents. Different from these existing works, focus leveraging structured textual information from web UIs to enhance MLLMS text-rich visual understanding capabilities across diverse domains. Our approach utilizes text-based LLMs synthesize general multimodal instructions from webpage accessibility trees, creating a large-scale dataset that improves model performance not only on web UI tasks but also generalizes well non-Ul domains.
CONCLUSION
In this work, tackled the challenge of improving text-rich visual understanding in MLLMs by leveraging wehpage Uls as naturally structured and diverse data source. We introduced the dataset MultiUI, which we demonstrated to significantly improves model across domains. Our experiments also underscore the importance of structured web data as a training resource, which helps MLLMs process and interact with text-rich visual environments more effectively.
"htps://commoncraw..rrgg
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
Junpeng Liu and Wai Lam are supported by grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14200620).
REFERENCES
Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Vic- tor Cärbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. Screenai: vision-language model for ui and infographics understanding, 2024. URL https: org/ abs/
2402. 04615
Youngmin Back, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. Char- acter region awareness for text detection. ference on computer vision and pattern recognition, pp. 9365-9374, 2019.
Proceedings of the IEEF/CVE con-
URL
https://openacces.co www.oeesssseeeef..oooooo content CVPR 2019 10019/htmll htt.t/tll////////.//t Baek Aeegiienwwwreeess___m fer___oooo__________eee_e__e___e Detection CVPR 2019 paper
Character Cepracter_Reiion Awareness
Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi. Jindong Chen, and Blaise Agüera y Arcas. Uibert: Learning generic multimodal representations for understanding. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 1705-1712. International Joint Conferences Artificial Intelligence doi: 10.24963/jjcai.2021/1
Organization, 2021.
URL
ijcai. 225
Main Track.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. CoRR, abs/2311.12793, 2023. doi: 1044550/AXXI.2311.22779 URL 48550/arXiv. 12793 12793
org/1........
Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. WebSRC: dataset for web-based structural reading comprehension. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 4173-4185, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguis- tics. doi: 10.
emm.p ma mwin.a......mmm 343
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Lun-Wei Ku, An- dre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: / wotthelogy.org///e
acl 500.505
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng. Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
//1 wrg.or////.o.. /bl wg///222222222 - 10--0022003---iiii-- 30-vicuna/
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine (eds.), Advances Neural Informa- fion Processing Systems, volume 36, pp. 28091-28114. Curran Associates, Inc., 2023. URL httppp///aaceeeeinn.n.........ep http://rrrflleeeppeeeeeeeeee wtt.f//lls/aaper/222/1.... 55002200aLaaa00000660000eeeeeee2
and Benchmarks.
pdf
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Leuman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21788. 2024. URL htttts:///aatttt.ee wttp://wwwwww/////oo///o/oooo.s wwwp/bb/////.a /407 .21783

---

Published as a conference paper at ICLR 2025
Yuan Gao, Kunyu Shi, Pengkai Zhu, Edouard Belval, Oren Nuriel, Srikar Appalaraju, Shab- nam Ghadar, Zhuowen Tu, Vijay Mahadevan, and Stefano Soatto. Enhancing vision- language pre-training with rich supervisions. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp- 13480-13491, June 2024. URL htttp:://opeacceeeo twwccf..e........ wwtam/oonent//oooo httpm//coneee/////22///////o/// http:////R2222//m2mmmm Enhancing Vision Lanoon-nannageeee eze-trainingg with Rich Supervisions.
CVPR_ 2024 paper
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents,
2024. URL
05243
Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), thecef. com/ catmm/coneett_cvr___e0
July 2017. URL https:ttt////peetle
th____tteee____e____eee 2017 _aperr
Wenyi Hong, Weihan Wang. Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang. Juanzi Li, Bin Xu, Yuxiao Dong. Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2023. URL
org/ wrtpoa//abbb/...././/////
08914
Yu-Chung Hsiao, Fedir Zubach, Maria Wang, et al. Screenqa: Large-scale question-answer pairs over mobile app screenshots. ArXiv preprint, abs/2209.08199 2022. URL https://arxiv:
org/abs/ 2209..2209... 08199
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for oer-free document un- derstanding, arXiv preprint arXiv:2403.12895, 2024a. URL httts:://axxiii org/
2403. 12895
Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docown High-resolution compressing for ocr-free multi-page document understanding, arXiv preprint arXiv:2449.03420, 2024b. URL httpt:t///// /www://wwwww///./// www..arxiv... /rg/
abs/ /eee/249..oo 03420
Jing Yu Koh Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluat- ing multimodal agents on realistic visual web tasks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume Long Papers), pp. 881-905, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/200 11865///1/202..allooonggg org/ 2024 acl.......ooog.........
URL
Hugo Laurençon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, Matthieu Cord, and Vic- tor Sanh. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. In Thirty-sevnnth Conference Neural Information Processing Systems Datasets and Benchmarks net
Track, 2023.
henttts://openreeee wttp:/iwwwweeeeouummmmeeee http://formmmddddsmm SKN IBIZ
IBIZ
Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisensch- los, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2Struct: Screenshot parsing as pretraining for visual language understanding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 18893-18912. PMLR, 23-29 Jul 2023. URL. httpss://pron httee://poccceiinggee mwr www...ppees..... v2 20we22222222222 Lee23g.eeeeeeeeee html
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Gener- ating natural language description for mobile user interface elements. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5495-5510,
12

---

Published as a conference paper at ICLR 2025
Online, 2020. Association for Computational Linguistics. doi: 10.18653/w/20200emmbpp main. wwtp:/oyyy.rg//0oooooo www.:/20..mmmpmmmmma.aa 443.
443. URL
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In
A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 34892-344 34916. Curran Associates, Inc., 2023a. URL /ttppss///pprcceeiiggsssoco ntw.:gssseeuisisp cc.ppppeepcpppppp....ppeeee files/papee/2023/
pdf
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuming. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26296-26306 June 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL https://
flava-vi..itan Lo/p/log/024-011---1111100
llava-next/
Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal LLMs evolved in web page understand- ing and grounding? In First Conference on Language Modeling, 2024c. URL https: 7/ /oopezeeieww.o
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bi,,, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.0628/ 2023b. URL hRttps////axxii
org/ abs/
Yuliang Liu, Zhang Li, Mingxin Huang. Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models, 2023c. (RLhhtts:://
http:///axvvvvroooee wwwp/rr//////aeoeeeee.eeeeeeeeee 2305. 07895
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: findings wtw.://....nnnggnc...a.... acligs-acl.17
URL
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on docu- ment images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021. URL httpps://ieeepploeee wee.eieeeee.rr. org/ wttp:/eeeeeee.tttea///ssss/tetee stamp/stamp
1sp2arnumber942333
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawa- har. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022. URL
therseccccssssseececeee tweccf..e....c com/ content/wAcC222220 htt22/htmm//mm /eetteeeaaaea__aaaa http:///tteeeee.ooooooooooooooaa WACV wtwp:/____22_paerrreaeee
Meta. Introducing llama 3.1: Our most capable models to date, July 2024. URL https://ai.
OpenAl. Gpt-4v(ision) system card, 2023, URL httppss//ppp..emmnttccccooo..
CorpusID:263218033
OpenAl. Gpt-4o system card. 2024.
Corpus CorpusTD:2736621
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL httl/tttss//axxi../
hbtpgg/bb//2500 7557
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat. Julian Schrittwieser. et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055 055330 2024. URL htt.ttttss////ii...... wwtpaaawwwwvaaaaaaaaaaaaaaaaaaae 05530
13

---

Published as a conference paper at ICLR 2025
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs, 2021. URL/tttps:/aaxxi.... wttpvv....r//bb//111....
02114
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In shirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URLhttps: http:///pppeeeee..eemmo wwtp://wwwwweetteounoeo..a
M3Y7 113374vmmsi 4vmsMcY
Amanpreet Singh. Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 20/9, Long Beach, CA, USA, June 16- 20, 2019, pp. 8317-8326. Computer Vision Foundation / IEEE, 2019. doi: 10.1 10.VVR.. 2019.00851. URL http::t/tt// Singh_Towarde VQA_MModeeaaa owt.eeass_heeeceeeeeeeee_.CLPPLL
com cont http:::://oopccccs.ssseeeee wttp://tt.cccccoottt__eCCLLE
9/html/
html
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehension on document images. In ThiirtyFiftth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 13878-13888 13888, AAAI Press, 2021. URL http:::/ts///jtt aaai.
org/index. index.
AAAI httppp/pwwwaaaaaaaaaeeeeeeeaaaee
Nevan Wichers, Dilek Hakkani-Tüne and Jindong Chen. Resolving referring expressions in images with labeled elements. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 800- 806, 2018. doi: (0.1100/LT.2001.86633 (document/ (ocuument/633955 /6699118
URL
ieee, org/
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal Ilms, 2024.
URL hhttl::t////aat http://axvvvv./. or..orgg/aa...... http//b//22/ 2404. 05719
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 69-85, Cham, 2016. Springer International Publishing. ISBN 978-3-319 URL https: //1ink www.sprriiieee com/ chapter/10. 978-3-319-46475-5
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin. Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Pruceedings of the JEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR), pp. 9556-9567, June 2024a. URL Tue__MMU_A_ Mww.sccppppnneMMMitmtooo..aa www.taimdal_unneeestttdddaaa and Reasoning Benchmark for___rr__________
Massive_Muil
Xiang Yue, Tianyu Zheng. Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Ming Yin, Botao Yu, Ge Zhang, et al. Mmmu-pro: more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXhv:2409.02873, 2024b. URL httpt::tp///attt.///
org/abs/2400 02813
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation
14

---

Published as a conference paper at ICLR 2025
of large multimodal models. arXiv preprint arXiv:2407.122721 2024. URL https://arxiv
org/ 10s//ab//22 /////bbb////.// 2
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A real- istic web environment for building autonomous agents. In The Twelfth International Confer- ence on Learning Representations, 2024. URL bttps://ppeneevee..ttt net. forum?id- oKn.966tLLx
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang. and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. In A. Oh, T. Naumann, A. Globerson. K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Infor- mation Processing Systems, volume 36, pp. 8958-8974. Curran Associates, Inc., 2023. filesi Beleee/paperr222///552 ccbec77cts11136c3337255eeeeob000 Paper Datasets, and Benchmarks
URL hutpess/pproceeding..eerrr wwwphn/sss..uu.s...c.....s..oo
pdf
15

---

Published as a conference paper at ICLR 2025
Table of Contents in Appendix
Screenshot Cropping & Accessibility Tree Refinement
17
Example of Our Processed Accessibility Tree
18
Prompts For Task Extraction
19
Example Training Data by Task
24
Training Setup
27
F Details of Evaluated Benchmarks
28
G Evaluation Details and Full Results
30
Mind2Web Experiment Details
32
Ablation Task Types
33
Prompt Variation
34
Comparison with Other Datasets
35
16

---

Published as a conference paper at ICLR 2025
A SCREENSHOT CROPPING & ACCESSIBILITY TREE REFINEMENT
The original full-page webpage screenshots may be quite long (e.g. height much larger than width), to produce screenshots that are suitable to be input into an image-based model with image size or resolution requirements, we crop the full-page screenshots. Because webpage screenshots are typically structured vertically, horizontal cuts preserve the logical flow of text and sections. Thus, we crop the screenshots borizontally by selecting a random height while keeping the width fixed as the original one. Specifically, a height-width ratio is sampled from uniform distribution [min height width ratio, max height width ratio), where the ranges are [0.5, 1.5) and [1.5, 2.5] for Windows 10 and iPhone 12 Pro, respectively. These ranges are determined by considering screenshot sizes in existing GUI datasets (e.g., VisualWebBench, ScreenSpot).
Required by grounding tasks samples, which need element coordinates, we extend the accessibility tree used in Zhou et al. (2024) by adding the bounding boxes of each element into the original accessibility tree. Figure illuutrates an example of the processed accessibility tree webpage. The accessibility tree is comprised of several lines, each detailing a UI element. Each schema includes: 1) The UT element ID, 2) The UI element type, 3) the text content (e.g., OCR text, icon name), and 4) The bounding box coordinates.
17

---

Published as a conference paper at ICLR 2025
B EXAMPLE OF OUR PROCESSED ACCESSIBILITY TREE
1001 focts
[]] Root Element 'Masala chai madelcines with brown sugar
sugar honey
und honey 1001 Tea Facts' focused: True
[6] navigation 'Primary' [0.0, 0.0. 1.0, 0.055]
Sign and Get didusia
[62] link "1001 Tea Facts' [0.047, 0,012, 0.299, 0.053]
Crlvared
[89] image '1001 Tea Facts [0.047, 0.012, 0.299, 0.055]
cocsent dae senpidee srts
[92] link 'HOME' [0.299, 0.018, 0.378, 0.048]
[93] link "BI [0.378,
0.452, 0.048)
[31] heading 'Masala chai madeleines with brown sugar
and honey' [0.135, 0.082, 0.865, 0.127]
[147] image "Masala chai madeleines with brown sugar
and honey' [0.031. 0.155, 0.625, 0.551]
[118] HenderAsNNonLammmaatt [0.031, 0.558, 0.625, 0.684]
menullcuue cansine beversingg comora auttilaiiion
when aeen cecon to ber of prodesslrtee
[202] time [0.031, 0.559, 0.131, 0.576]
purt
product posseet
[303] StaticText 'October 14, 2015' [0.031, 0.559, 0.131,
Search
0.576]
[205] StaticText "by [0.134, 0.559, 0.152, 0.576]
Wieeccaa ald araa su cullantton auttaan subbic in att ongetfis
may hemerarl seccnseveent mura inte rapo
[207] link 'Dvvd Kartuzinski [0.152, 0,559, 0.263, 0.576]
stteghhneee
scale dificiuu
[309] StaticText [0.031, 0.581, 0.039, 0.601]
thes and afferer voye eith cede ateppe repedddee
[310] StaticText 'Note: teafacss com contains paid
likers with
advertising and contextual affiliate links where
Webpage
Processed Accessibility Tree
Figure 7: Example of processed accessibility tree. (Left) The webpage screenshot; (Right) The corresponding accessibility tree. Each line represents an element the webpage, formatted as <element-id> <element-type>> <embedded text>> <bounding- box coordinates>>
18

---

Published as a conference paper at ICLR 2025
C
PROMPTS FOR TASK EXTRACTION
Prompt for Website Curation.
You are given a truncated accessibility tree of webpage, which derived by crawling program:
{axtree]
You need to answer the following two questions: whether the crawl is successful (c.g., 403 forbidden, 502 bad gateway, blocked by Cloudflare, blank page. 404 not found are considered as unsuccessful etc. )? Does the webpage contain some harmful content (e.g., adult content, gambling, violence, discrimination)
Your output only consists of two lines, cach line is your final answer to one question. Only output "YES" or "NO" cach line and do not generate any other content. Note that the truncation does not necessarily mean the crawl is not successful.
19

---

Published as a conference paper at ICLR 2025
Prompt for Constructing Webpage Captioning Samples.
You are an Al visual assistant that can analyze a single screenshot of a webpage. You receive the meta description extracted from its HTML content and the accessibility tree of the webpage.
The accessibility tree consists of several lines, each describing an element. Each schema contains: 1) The UI element IDs, 2) The UI element types, 3) The OCR text (when applicable) or the element descriptions (e.g. captioning. or the icon name), and 4) The bounding box coordinates [left, top, right, bottom]. as [xl, yl. x2. y2] with floating numbers ranging from 0 to 1. These values correspond the left, top, right, and bottom. Indentation are used to indicate the hierarchical structure between the elements.
Using the provided meta description and accessibility tree, describe the webpage in detailed manner.
Instead of directly mentioning the bounding box coordinates, utilize this data explain the webpage using natural language. Include details like element counts of specific element type, position of Ul elements, relative position between the UI elements.
Important Notes about your caption:
(1) When using the information from the meta description and the accessibility tree, directly explain webpage, and not mention that the information source the meta description the accessibility tree (e-g.- "Based on the provided meta deseription and accessibility tree, can descrihe the webpage as follows"). (2) When you mention some text paragraph, mention the summarized content instead of just saying "there articles text".
(3) Every when you want express "below", "right". "left". "following", "aaove" and other words representing relative position, you must carefully compare the bounding coordinates of elements you want to mention, because the orders within the accessibility tree do not mean the relative position, c.g, element "A" appearing after another element "B" within the accessibility tree may be on top of the other element "B", in term of bounding box coordinates.
other words representing absolute position, you must determine its position based on the reference points: top left [0, O, 0, 0]. top right [1.0, O, 1.0, 0]. bottom left [0, 1.0, 0, 1.0], and bottom right [1.0, 1.0. 1.0, 1.0].
(3) Every time when you want to express "top". "right", "left", "boltom", "above"
(4) No need discuss overall design, e.g.. "The overall layout clean and organized, with clear headings
and concise text").
(5) You cannot say there are no images on the page.
(6) Do not mention line breaks and separator line.
20

---

Published as a conference paper at ICLR 2025
Prompt for constructing Webpage QA Samples.
You are given the following webpage, describe in words. Follow the following guidance to think step by step before generating five questions and their answers.
(Wehpage Mete Description) The meta description of the current web page is "description". Based on both the web page description and your understanding, think carefully what the website can be used to do.
(Webpage Accessibility Tree) You given the following accessibility tree, Each schema contains: 1) The UI element IDs, 2) The element types, 3) The OCR text (when applicable) or The element descriptions (c.g. captioning, or the icon name), 4) The bounding box coordinates [left, top, right, bottom], as [x1, yl, x2, y2] with floating numbers ranging from 0 to 1. These values correspond to the left, top, right, and bottom. Indentation are used to indicate the hicrarchical structure between the elements.
Note: The appearing orders of elements in the accessibility tree do represent their relative vertical positions. For instance, even if element "A" comes after element "B" the accessibility tree, might still appear above element "B" based on their bounding box coordinates (yl. y2 of "A" values are smaller than those of "B", so "A" is above "B" vertically). You must determine their vertical positions by carefully comparing their yl, y2 coordinates instead of infering their positions based the orders within the accessibility tree. axtree
(Potential Questions) Some example questions on other websites are shown below for your reference. {question demo}
(Answers) For each question, you should generate two kinds answers: short answer and detailed answer. Ask questions whose short answers consist of fewer than 10 words, and the answer should be as short as possible.
The detailed answer consists of thinking reasoning process of obtaining the final answer and should be as detailed as possible. (Notes)
1) Your questions should hard as possible, and need deep understanding about the webpage and reasoning ability.
2) You must ask objective questions, not subjective ones.
3) Your questions should be diverse as possible.
4) not mention in answers that you answer questions based on accessbility tree, instead, answer you
are seeing the webpage screenshot.
5) The given meta description is only for your better understanding about the webpage, so should never
ask any questions specific to the meta description.
6) You can only generate questions that be answered immediately hased the current wehpage, do
ask those that need your background knowledge.
7) You should never ask questions about the advertisements or the whether--ccept-ccokii--e section of
the webpage. if any.
8) Do not user element IDs refer element.
not ask questions that you cannot answer based on the given webpage accessibility tree. 10) Do not ask any questions about the main heading of the webpage, like "what is the title the webpage?
"WWat is the title of the main heading?"
(Final Output) Your output should be several lines of json with each line being an question-answer pair, and do not generate any other content. You only speak ISON. not write text that isn't JSON.
{"quesstoon": <the question content, string data type enclosed in double quotes>, "answer": <short answer. string data type enclosed in double quotes>, @etailed_ansswr"" <detailed answer, string data type enclosed in double quotes>)
21

---

Published as a conference paper at ICLR 2025
Prompt for constructing Embedded Image QA Samples.
(Caption)
[The start of the caption]
{caption)
[The end of caption]
(Potential Questions)
Some example questions other websites shown below for your reference. {question.demo)
(Answers)
For each question, you should generate two kinds of answers: short answer and detailed answer. Ask questions whose short answers consist of fewer than 10 words, and the answer should be short possible. The detailed answer consists of thinking or reasoning process of obtaining final answer and should be as detailed as possible.
(Notes)
1) Your questions should hard possible, and need deep understanding about webpage and reasoning ability.
2) You must ask objective questions, not subjective ones. 3) Your questions should be as diverse as possible.
4) Do not mention in answers that you answer questions based on the caption, instead, answer as if you seeing the webpage screenshot.
5) You can only generate questions can answered immediately based current webpage,
ask those that need your background knowledge.
6) You should never ask questions about the advertisements or the whecher-acoet--cooicoooo section of
the webpage, any.
not ask questions that you cannot answer based on the caption.
8) not any questions about the main heading webpage, like "what the title the webpage?
"WWat the title of the main heading?"
(Final Output)
Your output should be several of json with each line being an question answer pair, and do not generate any other content. You only speak JSON, Do not write text that isn't JSON. {"questioon" <the question content, string data type enclosed in double quotes>, "answer": <short answer. string data type enclosed in double quotes>, "detailed_answer" <detailed answer, string data type enclosed in double quotes>]} }}
22

---

Published as a conference paper at ICLR 2025
Prompt for constructing Action Grounding Samples.
The screenshot below shows the webpage you sec. Follow the following guidance to think step by step before generating several executable instructions on it.
(Wehpage Mete Description)
The meta description of the current web page is "[deseription]" Based on hoth the web page description and your understanding, think carefully what the website can be used to do.
(Webpage Accessibility Tree)
You are given the following accessibility tree, Each schema contains: 1) The element IDs, 2) The element types, 3) The OCR text (when applicable) or The element descriptions (c.g. captioning, or the icon name), 4) The bounding box coordinates (left, top, right, bottom), quantized and normalized between 0 and 999.
Indentation are used to indicate hierarchical structure between elements. (axtree)
(Potential Instructions and Target Bounding Boxes)
Think what instructions would be taked by human above interactive elements on the current webpage. Describe the instruction (i.e., action description) using natural expression, and output the corresponding bounding box (exactly same as in the accessibility tree). Some example instructions on other wehsites are shown helow.
1) search in the forum
go personal homepage
3) create a new issus
4) view orders
5) zoom in the map
check my postbox
Notes:
1) The instructions above are only for reference and the generated instructions must be executable on the
current web page and as diverse possible.
Do not generate action description about accepting decline cookie policy
3) Because generated action descriptions are used to test the capability other multi modal models,
your action description should not be too detailed and simple, even include the detailed position of the target element. E.g., click the red log-in button at bottom-rigg side
4) Your generated action description instruction must only correspond in order not to lead ambiguous
element within the screenshot,
5) Your generated action description instruction must corresponds to interactive element within screenshot, e.g., button, input field, link, etc. instead of those not interactive, e.g., StaticText. 6) Do not user element IDs to refer an element.
Do not generate instructions targeted the root element named "Root Element".
(Final Answer)
Finally, conclude your answer with the format below strictly, comprising one to five single-step executable instructions/ action descriptions on the current webpage. Note that you output a line whose content is be generated and you don't have to force yourself generate with "[Invalid]", and not generate any other
"Invalid]" if you find proper instructions one. Your answer should be several lines json or content. You only speak JSON. Do not write text that isn' JSON. Output format:
(""instruction": <the action description / instruction, string data type enclosed in double quotes>, "bbox": <the corresponding bounding hox coordinate copy in the accessibility tree, list integers>
23

---

Published as a conference paper at ICLR 2025
D EXAMPLE TRAINING DATA BY TASK
Webpage Captioning
Prompt: Describe all the visual and textual components of the webpage comprchensively.
webpuge appears to a product company page, with prominent logo displayed at the top left comner, repeated five times. Below the logo, there is navigation menu with seven links: "PRODUCTS" "ABOUT US". "CLIENTS", "NEWS BLOG", "CLIENT LOGIN", and "English", which small flag icon next to These links are positioned horizontally, taking up most of the top scction of the page. In the middle of the page, there is large heuding that reads "Al Driven Data for Market Leading Intelligence", followed by a subheading that explains how the company, dataxet:nama, helps customers compete in the media marketplace by reimagining monitoring and listening data. This text is centered on the page and takes up significant amount of vertical space. There no prominent image or graphic on the page, aside from the logo and the small flag icon next to the "English" link. The overall layout is focused on presenting the company's 's message and navigation options.
Response:
Al Driven Data for Market
Leading Intelligence
Figure 8: Example of training data for webpage captioning.
Embedded Image Captioning
Prompt: Give an in-depth description of the picture inside the red marked area.
The image depicts stunning, gathic-style chapel, likely King's College Chapel the University Cambridge, illuminated during the night. The chapel stands tall with its intricate stone architecture, large pointed windows, and spires reaching towards a sky filled with textured clouds, dramatically lit by the The mounlight and artificial lighting create serene yet slightly dominated by grussy field trees, which ftame structure and add to the overall picturesque and timeless foel of the scene.
Response:
Wedding trip part anrivalion in England
umbiunce. foreground
Figure 9; Example of training data for embedded image captioning.
24

---

Published as a conference paper at ICLR 2025
Webpage QA
What is name the company that designed the website? Look at the screenshot and respond with word short phrase.
Prompt:
www. www.candsrmaieeendeee www.eeedddddooooooooooooooo pcoo...
STAY CONNECTED JIR
Response: makead design
Sohnoon Tachng Al fights Reseved Velbatrs makest design
Figure 10: Example of training data for webpage QA.
Embedded Image QA
What the phone number for customer inquiries? Using the image as a reference, give elaborate response the question.
Prompt:
ONEWEST INTEENET SERvice FERVIDER Use Prometional Cade 630 Call O8
Two Months FREE!
DSL Dial-Up
CLICK MORE
Response:
The phone number' 1-866-ONEWESTT is provided at the bottom of the advertisement for customers to contact for inquiries support.
Figure 11: Example training data for embedded image QA.
Action Prediction
Consider the screenshot of webpage with bounding box select the webpage description that best describes the new page that appears after clicking the element inside the red box. Here are the candidates:
Prompt:
A. Make it with idleWilde Quality custom apparel solutions for businesses
Our Work idleWilde Printing
Muke with Wilee Custom Screen Printing Embroidery
EMBROIDERY
D. Contact Us idlcWilde Printing Co.
E. Get A Quote idleWilde Printing Co.
F. Ordering idleWilde Printing Co.
Make with idleWilde- Custom Screen Printing-
SCTUP FEDS
Screen Printed T-Shirts
H. idleeilde Printing Co. Custom T-Shirts Apparel
Response: B
Screen Printing
Figure 12: Example of training data for action prediction.
25

---

Published as a conference paper at ICLR 2025
Element OCR
Prompt: Analyze the screenshot of the webpage and extract the text from the UT element that inside the red bounding box.
ligety
Response: Quigley said "20 ceach and every ticket holder will be offered rescating options and should contact their original point of purchase for further details".
Quigley und 'rect and evyy bicket holder will afficee rusecing options and seculld contact thee"
onginal paint of parceww for further detale"
Mondaa 2uth Fubruary
Figure 13: Example of training data for element OCR.
Heading OCR
Please locate and retrieve the main header text of the webpage.
Prompt:
Making Business Fun
Response: Making Business Fun
BUSINESS
MANAGEMENT
Companies Adkartages
Figure 14: Example of training data for heading OCR.
Element Grounding
Prompt: Based on the element
CATEBOREE
description: "A5 Sportback", identify the UI element and provide its bounding box coordinates. Use four float numbers between and 1, [left, top, right, bottom].
widds sufr
1005
fatk aeaeeaiie, leeiing reporicd
puanii inienuuoo
Response: [0.706, 0.643, 0.781, 0.657]
Figure 15: Example of training data for element grounding.
Action Grounding
Prompt: Determine the bounding box coordinates of the clickable region to execute the instruction: "Read latest news". The coordinates should be four float numbers between and 1, denoted as [left, top, right, bottom).
dr dragnews
Response: [0.706, 0.643, 0.781, 0.657]
Figure 16: Example of training data for action grounding.
26

---

Published as a conference paper at ICLR 2025
E TRAINING SETUP
To leverage the high-resolution images collected in Section we adopted dynamic high-resolution strategy inspired by LLaVA-NeXT (Liu et al. 2024b). The process involves dividing the input im- age into patches matching the vision encoder's s original resolution, encoding these patches indepen- dently, and combining them into a single feature map. We also incorporate downsampled version of the entire image to provide global context. This combined representation is then processed by the LLM, which allows us scale the input to arbitrary resolutions while maintaining data efficiency.
Model
LLM
Vision Encoder
Max Res.
Training Data
UIX- MIX-Vicuna
Vicuna-7B-v1.5
CLIP
672 672
LLaVA 1.5 MultiUI
UIX-Llama3.1 1omma-3.--B-Insttee
CLIP
672 x 672
LLaVA NeXT + MultiUI
UIX-Qwent
Qwen2-7B-Instin
Siglip
768 x 768
LLaVA NeXT MultiUI
Table 6: Details of three models developed in this paper.
27

---

Published as a conference paper at ICLR 2025
F DETAILS OF EVALUATED BENCHMARKS
GUI Understanding Benchmarks
VisualWebBench (Liu et al. 2024c) is multimodal benchmark specifically designed to evaluate the performance of Multimodal Large Language Models (MLLMs) in web-related tasks. Un- like existing benchmarks, it focuses on capturing the unique aspects of web pages and measures fine-grained abilities such as OCR, understanding, and grounding, providing a comprehensive evaluation of MILMMs in the web domain.
WebSRC (Chen et al., 2021) is designed to evaluate models structural reading comprehen- sion tasks involving web pages, where both text understanding and structural comprehension are required.
ScreenQA (Hsiao et al. 2022) bridges the gap between component-level and high-level task un- derstanding in mobile app screen content. By annotating 86K question-answer pairs over the RICO dataset, this benchmark offers a comprehensive evaluation of screen reading comprehen- sion in various application scenarios.
WidgetCap (Li et al. 2020) evaluates the ability to generate natural language descriptions for mobile elements, a task critical for accessibility and enhancing language-basee interaction.
GUI Grounding
ScreenSpot (Cheng et 2024) serves as a comprehensive GUI grounding benchmark, covering mobile, desktop, and web environments. By encompassing multiple device types, it enables thorough assessment of cross plassfora GUI grounding capabilities.
RefExp (Bai et al., 202T) is designed to assess the model's ability predict the component referred to by natural language expression, given app screenshot. In order to simulate more realistic scenario, we evaluate models under the setting of predicting coordinates directly instead of retrieving component from a set of candidate components.
GUT Agent Benchmark
Mind2Web (Deng et al. 2023 is the first dataset for training and evaluating generalist web agents perform complex tasks on real-world websites. includes over 2,000 tasks from 137 websites across 31 domains, offering diverse interactions for building robust web agents.
General OCR / DocQA ChartQA Benchmarks
DoA (Mathew et al. 2021 is a benchmark for evaluating models' ability to answer questions based on document images. This dataset challenges models comprehend and extract relevant information from diverse document formats such as letters, forms, and reports.
ChartVQA (Masry al. 2022 presents unique evaluation scenario where models are tested on their ability to answer complex reasoning questions about data visualized in charts. This bench- mark is crucial for evaluating model's capability to perform logical and arithmetic operations, as well as to reference visual features within the charts.
Tex (Singh et al. 2019) focuses on questions that require reading and understanding text within images. evaluates model's ability to handle text-based reasoning, which is essential for tasks involving visually impaired users and real-world scenarios.
Info (Mathew et al,, 2022) focuses on the automatic understanding of infographic images through Visual Question Answering (VQA). This benchmark requires models integrate rea- soning over textual content, graphical elements, and data visualizations, which emphasizes both elementary reasoning and arithmetic skills.
VisualMRC (Tanka et al. 2021) focuses on evaluating the machine's ability to comprehend the visual layout and textual content within document images to answer questions. It requires model to be able to read and reason about multiple pieces of text and non-text data images and to generate abstractive answers.
28

---

Published as a conference paper at ICLR 2025
OCRBench (Liu et al. 2023c) encompasses 29 datasets and enables thorough assessment across a wide range of text-related visual tasks, such as Text Recognition, Scene Text- Text-Centric Visual Question Answering, Ceeument-Orientee VQA, Key Information Extraction, and Handwritten Mathematical Expression Recognition.
General-domain Tasks
MMMU (Yue et al. 2024a) is comprehensive benchmark designed to assess multimodal models on complex, college-level tasks actoss six core disciplines, including Art Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering.
MMBench (Liu et al. 2023b) is bilingual benchmark for evaluating the multimodal capabilities large vision-language models (VLMs). features a diverse set of carefully crafted evaluation questions and employs rigorous quality control measures, enabling precise assessments of model performance both English and Chinese contexts.
VQAv2 (Goyal al. 2017) introduces a balanced dataset that pairs each question with two similar images leading different answers. The findings show that sfate-of-the-art VQA models struggle with this dataset, underscoring the need for models to better utilize visual information rather than relying language priors.
RefCOCO+ et al. 2016) focuses on generating and comprehending natural language referring expressions for objects images, particularly by improving the use of visual context.
29

---

Published as a conference paper at ICLR 2025
G EVALUATION DETAILS AND FULL RESULTS
In this section, we describe the evaluation details and present the full experimental results of both baselines and our UIX models.
We employ LMMs-Eval (Zhang et al. which official evaluation code The shows the employed evaluation metrics for bench- marks. andd Table show results of baselines and our UIX models.
2024) for all evaluations except for VebbenWebBench, for
Benchmarks
Metric
Split
VisualWebBench
Aggregated Score SQuAD-FI FFI
test
WebSRC
validation test
ScreenQA-short
SQuAD-F1
WidgetCap
CIDEr
test
Element Ground (VWB)
Accuracy (IoU>0.5)
test
Action Ground (VWB)
Accuracy (I5U>0.5)
ScreenSpot
Accuracy (IoU>0.5)
test
RefExp
Accuracy (I0U>00.5)
DocVQA
ANLS
validation test validation
ChartQA
Relaxed Accuracy
TextVQA
Exact Match
InfoVQA
ANLS
validation
VisualMRC
ROUGE-L
test
OCRBench
Accuracy (%)
test
RefCOCO+ (REC)
Accuracy (IoU>0.5)
validation
Table 7: Evaluation metrics for all benchmarks.
LLaVA-
LLaVA- 1,5-13B
LLaVA- 1.6-7B
LLaVA- 1.-13B
LLaVA- 1.6-34B
LLaVA- NeXT-8B
1.5-7B
VisualWebBench
17.0
19.4
36.0
39.4
50.5
42.1
WebSRC
30.9
32.5
67.2
71.2
83.2
72.8
ScreenQA-short
42.6
46
66
68.3
74
68
WidgetCap
20
10.2
35.4
23.4
46.3
49.8
Element Ground (VWB)
0.73
0
0.24
0
1.7
0.97
Action Ground (VWB)
0
0
0
0.99
3
0
ScreenSpot
0.6
0.9
0,9
0.4
2.8
1.7
RefExp
0.4
1.1
0.4
0
3.4
1.1
Doc
28.1
30.2
74.4
77.5
83,9
78.2
ChartQA
18.1
18.2
54.8
62.4
68.6
69.2
TextVQA
46
48.7
64.8
67
69.4
65.3
InfoVQA
25.8
29.4
37
41.5
51.3
37.6
VisualMRC
35.3
38.3
33.3
35.9
37.9
29.3
OCRBench
31.3
33.6
52.1
55
57.2
55.2
RefCOCO+
50
59.9
77
80.8
84.8
79.5
MMMU
36.3
35.4
36.3
35
49.3
40.3
MMBench
64.2
68.5
67.1
69.2
78.1
72.2
VQAv2
76.1
77.8
79.9
80.6
81.8
80.7
Table 8: Full experimental results of baselines.
30

---

Published as a conference paper at ICLR 2025
LLaVA- Vicuna
UIX- Vicuna
LLaVA- Llama3.1
UIX-
LLaVA- Qwen2
UIX-
Llama3.1
Qwen2
VebBeeWeb
23.1
71.1
35.3
74,2
41.7
75.9
WebSRC
41.5
69.5
65.0
75.3
72.5
82.9
ScreenQA-short
53.0
73.9
65.7
72.7
68.6
78.8
WidgetCap
38.4
66.5
34.2
55.6
38.0
72.7
Element Ground (VWB)
0.0
55.5
0.5
16.7
1.2
66.1
Action Ground (VWB)
0.0
26.7
0.0
11.9
0.0
35.6
ScreenSpot
1.3
44.7
1.3
22.2
1.3
55.2
RefExp
1.2
35.8
0.9
17.9
1.9
43.5
Doc VQA
46.1
7288
74.7
78.0
76.5
85.3
ChartQA
21.2
24.2
66.5
66.9
68.5
74.0
TexiVQA
59.6
67.0
64.3
65.1
67.0
72.7
InfoVQA
31.9
41.6
35.7
44.2
41.1
52.2
VisualMRC
39.7
43.3
46.8
49.7
44.1
49.1
OCRBench
38.1
53.4
54.0
58.6
55.7
66.3
RefCOCO+
61.7
65.7
74.8
71.7
75.9
79.1
MMMU
34.7
33.6
38.8
42.3
44.7
41.8
MMBench
66.1
66.9
72.0
74.7
76.5
77.4
VQAv2
78.5
79.8
80.0
80.0
81.6
82.1
Table 9: Full experimental results of our models compared to three different backbones.
31

---

Published as a conference paper at ICLR 2025
H MIND2WEB EXPERIMENT DETAILS
In the experiments on Mind2Weh, we give the models a raw screenshot, a task description, and previous actions inputs. Models are then prompted directly predict the next action.
Screenshots from Mind2Web's original observation capture the entire page, which can go as long as over 8:1 in neight-to-width ratio. Observations of this length disconnect from human observations. Similar to SeeClick (Cheng et al. 2024), we randomly crop around the ground truth element, with random window size at least as big those used in SeeClick.
We evaluated CogAgent-Chat Chat in the same setting as baseline, as CogAgent-Chat is the recom- mended version of CogAgent for agent application. We used the exact prompt in Hong et al.) (2023) that is used for agent operation with grounding. To translate model output into GUI action, we con- vert the model's predicted bounding box into point of operation on the web page by taking the center point of the bounding box. Here is an example prompt we used:
Prompt example for CogAgent
What's the proper procedure for "Add to cart one Private Vehicle Pass for Yosemite National Park for April 30, for License Plate number 12345 and Zip Code 94587
Previous Actions:
Idivl All About Passes -> CLICK
linput] -> TYPE: Yosemite
[link] 1 Yosemite National Park id: 74296 -> CLICK
[img] Private Vehicle Pass -> CLICK
[texthox] zip Code (Required) TYPE: 94587"
(with grounding).
32

---

Published as a conference paper at ICLR 2025
I ABLATION ON TASK TYPES
The tasks defined in Section are categorized into QA, Caption, OCR, and Grounding. To eval- uate the impact of different types of training samples on downstream performance, we conducted a comprehensive ablation study. For instance, we trained a captioning model using combination of Caption samples from the MultiUI dataset and LLaVA data, and analyzed its performance on various downstream benchmarks. Figure 6 presents the results of this study, highlighting several key findings. Specifically, both QA and Caption samples significantly enhance performance in GUI understanding (Visual WebBench) and OCR-related benchmarks (DoeVQA), while OCR samples primarily benefit tasks in GUI understanding (VisualWebBendee that require basic text recognition. Notably, relying solely on grounding samples derived from web data can adversely affect perfor- mance on the RefCOCO+ benchmark, as it involves natural scene images rather than webpage- images. Overall, incorporating all sample types yields the most balanced performance across all four benchmark categories.
33

---

Published as a conference paper at ICLR 2025
J PROMPT VARIATION
To improve the generalizability of the generated conversation samples, we diversify the instruction templates for each task. First, create detailed task description and initial example templates. Then, using GPT-4o, we generate 200 varied prompt templates based on the task descriptions and hroandprovided demonstrations.
34

---

Published as a conference paper at ICLR 2025
K
COMPARISON WITH OTHER DATASETS
Datasets
Task Types
#Samples
SeeClick
Grounding, Captioning
1M
Ferret-Ul
QA, Grounding, Captioning
841K
Mind2Web
Web agent
2K
MultiUI
QA, OCR, Grounding, Captioning, Action Prediction
7.3M
Table 10: Comparision among GUI-related datasets.
35


Reviews:
[{"review_id": "s8PAS2Dh6P", "reviewer": "Reviewer_K6jg", "summary": "This paper introduces MultiUI, a large-scale dataset of website UIs annotated for diverse UI-related tasks, including captioning, visual question answering, OCR, and UI interaction grounding. MultiUI leverages web screenshots and accessibility trees as key components of its data pipeline, with annotations synthesized using powerful LLMs to support these specific UI tasks. The second part of the paper explores training Vision-Language Models (VLMs) on this new dataset and evaluates their performance on GUI understanding and grounding benchmarks.", "questions": "- How did the authors filter websites for making sure it does not contain harmful or adult content? They provide some detail on LLM based filtering on section 2.2, but this can be insufficient. Did you consider human inspection to enhance this process?\n- How does this dataset compare with other datasets in the field? A statistical comparison on the quantity and quality of samples is missing. Also, in the experiments section, we are missing a comparison on how VLMs finetuned on other dataset perform on the proposed benchmarks, in order to see the benefits of the proposed data in performance terms.\n- How did authors solve hallucination when creating this dataset synthetically using LLMs?\n- Have authors analyzed the train/test overlap, to avoid data contamination from train to test?\n- Have authors considered finetuning other VLMs than Llava?\nHow does image resolution impact performance on the proposed benchmarks? Are there additional patterns observed in the experiments when comparing different Llava variants? A discussion of these insights would be appreciated.\n\n(see the weaknesses section for additional questions)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is easy to follow and provides substantial figures for improving its understanding.\n- The methodology for defining the data collection pipeline seems valid and the qualitative examples show that the samples are relevant.\n- The proposed dataset provides substantial annotations relevant for the UI understanding field. It is a large-scale dataset, and can be relevant for the VLM community.\n- Experimental results of training Llava-based VLMs on the proposed data shows that models performance improve with respect to models not trained on it.", "weaknesses": "- The paper lacks a comprehensive comparison with other datasets in the field, in terms of number of samples, and types of annotations and tasks they can perform. Some examples are SeeClick, Ferret-UI, Mind2Web, or WebArena, among others. \n- The selection of baselines seems limited, mostly relying on Llava. Most of the results on other baselines and architecture appear empty (referring to tables 2 and 3). Authors need to thoroughly evaluate closed models like GPT4o or Claude 3.5 in the proposed benchmarks in order to offer a more strong comparison, alongside powerful open-source options like MolMo, Gemini, Llama3.2.\n- Authors only present finetuning experiments on Llava-based models, lacking performing experiments on other baseline architectures to show the benefits of the proposed dataset.\n- While the paper presents contributions in annotating tasks across visual understanding and reasoning, OCR, and UI grounding, the novelty of its approach is limited compared to recent advancements in the field, such as SeeClick, Ferret-UI, Mind2Web, and WebArena. The primary distinction appears to be in the scale of annotation rather than introducing fundamentally new methodologies or task definitions.\n- Authors claim that the pipeline solves hallucination in the annotation process, a phenomena observed in previous works, but the paper does not describe how exactly this hallucination prevention is done.\n- The experiments section is missing a batch of experiments analyzing what is the performance of models when trained on other similar competitor datasets in the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k6CHH5UCw3", "reviewer": "Reviewer_exHC", "summary": "This paper addresses the challenge of improving text-rich visual understanding in multimodal large language models, focusing on the ability to interpret both textual and visual elements. MultiUI aims to improve MLLM performance in document understanding, GUI comprehension, and grounding, enhancing generalization across web and other domains. Experimental results show that models trained on MultiUI significantly outperform baselines in tasks such as GUI understanding, OCR, and grounding.", "questions": "Recommended to clarify whether any human validation was conducted on the MultiUI dataset, including the QA tasks generated within the dataset.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The introduction of MultiUI as a large, diverse dataset specifically designed to enhance multimodal understanding using structured web UI data.\n2. The paper demonstrates the dataset's impact, showing substantial performance gains over existing baselines across various multimodal tasks, emphasizing its importance for text-rich visual understanding.", "weaknesses": "1. There's a lack of documentation, making the dataset hard to navigate and providing no usage guide for the code. The authors should improve accessibility and include a clear code guide.\n\n2. In the construction pipeline of MultiUI, while the use of Llama and LLaVA may be reasonable for certain tasks, relying on them for higher-level tasks like question answering raises concerns about reliability. Using these models for complex tasks in the dataset risks introducing biases and undermining the robustness of the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YUvjDlm4Zz", "reviewer": "Reviewer_8zAm", "summary": "This work introduces MultiUI, a dataset of 7.3 million VQA examples spanning 9 types of tasks covering topics of perception, comprehension, grounding, and reasoning of webpage UIs, rendered on both desktop and mobile platforms.\n\nSpecifically, the authors collect data leveraging information from the websites both directly visible from the rendered view as well as the originating accessibility tree and DOM structure and generate task data using a mixture of rules and various foundation models:\n1. Webpage captioning: LLama 3 70B summarizes the website based on the accessibility tree.\n2. Webpage QA: Llama 3 70B generates questions and answers for a given website based on the accessibility tree.\n3. Embedded image captioning: GPT-4o-mini generates a caption of an image on the website, including context provided by the accessibility tree.\n4. Embedded image QA: Llama 3 70B generates questions and answers for an embedded image, based on the generated caption.\n5. Action prediction: PlayWright is used to interact with elements on the website to generate multi-choice Q&A pairs. The task is predicting the title of a target page when interacting with an element on the original website.\n6. Element OCR: the task is to predict the text in an area highlighted with a red bounding box (set-of-mark prompting) in the rendered view. Relevant elements are selected based on minimum text length in the DOM.\n7: Heading OCR:  the task is to predict the title of the website.\n8: Action Grounding: Llama 3 70B generates request for click targets to execute a certain action, based on information from the accessibility tree. Data is generated for both multiple-choice and direct answer settings.\n9: Element Grounding: coordinates and a textual descriptions are extracted from a DOM tree, with the task to predict the coordinates based on the description. Again both multi-choice and direct answer settings are considered.\n\nThe show the efficacy or the collected data to improve performance on GUI understanding and grounding, the authors train multimodal LLMs using various LLM backbones and show consistent and significant improvements over models trained on LLaVA 1.5 or LLaVA-NeXT data mixtures when incorporating their proposed MultiUI data.", "questions": "* The ablation on task type (section 4.7) shows that RefCOCO+ performance deteriorates slightly when adding MultiUI Grounding data, but seems to meaningfully improve when adding other data, particularly QA data. The discussion attributes the reduction in performance to domain shift but doesn't offer suggestions why MultiUI as a whole may still benefit RefCOCO+. Is there some understanding why this may be the case?\n* ScreenSpot results are not reported for e.g. SeeClick or CogAgent, but as far as I can tell they are reported, e.g. here: https://github.com/njucckevin/SeeClick Can they be added to table 2?\n* In section 3.1 the high-resolution strategy is briefly introduced. However, the exact tiling setup is not iterated on, including in appendix F. Appendix F lists a \"maximum resolution\" but I presume this is the resolution of an individual tile? Since the exact tiling strategy has significant impact on some of the reported benchmarks (such as DocVQA), I'd recommend to add a few more details to clarify the implementation and facilitate reproducibility in appendix F.\n* For test sets like MMMU, RefCOCO+, etc. that have multiple splits that are somewhat widely used, please clarify which sets are used (perhaps in appendix G).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* The collected dataset appears relevant to the burgeoning field of LLM based UI understanding and control.\n* The method introduced to collect the dataset is described clearly and the identified sub-tasks are relevant and well motivated.\n* The models fine tuned on the introduced data achieve strong results across benchmarks, which supports the utility of the data collected.", "weaknesses": "* The description of the grounding task data generation (2.3.3) could be more specific. The description of how Llama 3 is used is not very clear to me (\"[the model] is not only prompted to generate multiple grounding instructios to predict the bounding box of a given element but also provides the corresponding ground-truth bounding boxes\"). For element grounding the extraction from the DOM tree is not described. Is the \"element description\" simply the text corresponding to an element (as per figure 15)?\n* While mentioned in the baseline section (3.3) and measured on general OCR and grounding benchmarks, GPT-4o is not considered for GUI understanding and grounding (table 2). Given that GPT-4o shows significantly stronger results on the general benchmarks shown in table 3, it would appear to be a stronger / more relevant baseline to also consider in table 2. This is particularly noteworthy since section 4.1 discusses GPT-4V as the \"most advanced MLLM\" in the comparison.\n* Minor: figure 4 and 6 rely on color, hard to follow on a black & white print.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hg0UtNLHDr", "reviewer": "Reviewer_oGMe", "summary": "This paper presents MultiUI, a new dataset that leverages webpage user interfaces as a structured and diverse multimodal data source for training large language models to understand text-rich visual content. The dataset consists of 7.3 million samples, categorized across various tasks such as visual understanding, text recognition, and GUI grounding. The authors address challenges of noise, hallucination, and generalization limitations in existing multimodal data sources. The paper introduces a two-stage training strategy, achieving significant gains in GUI-related tasks and general multimodal understanding. MultiUI demonstrates potential for improving MLLMs' capabilities in both GUI-specific and broader text-rich visual tasks, highlighting structured web data as a valuable training resource.", "questions": "In the data construction process, how do you ensure the quality of the data points?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. **Useful Dataset on a Good Problem**: The paper tackles the challenging and impactful task of text-rich visual understanding by introducing MultiUI, a large, well-structured dataset derived from webpage UIs. This dataset fills a critical gap by leveraging the naturally organized and text-embedded nature of UIs, enhancing MLLMs’ abilities to interpret complex, multimodal information in real-world scenarios.\n\n2. **Comprehensive Experiment Results**: The authors present thorough experiments that validate MultiUI’s effectiveness across GUI-related and general multimodal benchmarks, demonstrating significant performance improvements and robust generalization capabilities. The results solidly support MultiUI’s utility and showcase its potential in both domain-specific and cross-domain applications.", "weaknesses": "One notable weakness of the paper is the limited discussion around the quality of the generated data and labels within MultiUI. While the dataset construction relies on accessibility trees and LLM-generated instructions to filter and label data, the paper lacks in-depth analysis or validation of data quality, particularly concerning the accuracy, consistency, and noise levels in generated labels. Without a rigorous assessment or benchmarks comparing the quality of MultiUI labels against human-labeled or gold-standard datasets, it is difficult to gauge the dataset’s reliability for downstream tasks, potentially impacting the generalizability and robustness of models trained on it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)