Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms
Authors: Zhangheng LI
Keen You
Haotian Zhang
Di Feng
Harsh Agrawal
Xiujun Li
Mohana Prasad Sathya Moorthy
Jeffrey Nichols
Yinfei Yang
Zhe Gan
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal LLM
UI Understanding
Primary area: foundation or frontier models, including LLMs
TL;DR: A multimodal large language model (MLLM) designed for universal UI understanding across diverse platforms, including iPhone, Android, iPad, Webpages, and AppleTV.

Abstract: Building a generalist model for user interface (UI) understanding is challenging due to various foundational issues, such as platform diversity, resolution variation, and data limitation. In this paper, we introduce Ferret-UI 2, a multimodal large language model (MLLM) designed for universal UI understanding across a wide range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV. Building on the foundation of Ferret-UI, Ferret-UI 2 introduces three key innovations: support for multiple platform types, high-resolution perception through adaptive scaling, and advanced task training data generation powered by GPT-4o with set-of-mark visual prompting. These advancements enable Ferret-UI 2 to perform complex, user-centered interactions, making it highly versatile and adaptable for the expanding diversity of platform ecosystems. Extensive empirical experiments on referring, grounding, user-centric advanced tasks (comprising 9 subtasks $\times$ 5 platforms), GUIDE next-action prediction dataset, and GUI-World multi-platform benchmark demonstrate that Ferret-UI 2 significantly outperforms Ferret-UI, and also shows strong cross-platform transfer capabilities.

Published as a conference paper at ICLR 2025
FERRET-UI 2: MASTERING UNIVERSAL USER IN- TERFACE UNDERSTANDING ACROSS PLATFORMS
Zhangheng Li! Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li,
Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, Zhe Gan
Apple
Schariiuueeaas {yinfely, zhe. gan] com
ABSTRACT
Building generalist model for user interface (UT) understanding is challenging due to various foundational issues, such as platform diversity, resolution variation, and data limitation. In this paper, we introduce Ferret-UI 2, a multimodal large language model (MLLM) designed for universal UI understanding across a wide range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV. Building on the foundation of Ferret-UI, Ferret-Ul introduces three key inno- vations: support for multiple platform types, high-resolution perception through adaptive scaling, and advanced task training data generation powered by GPT- 4o with set-of-mark visual prompting- These advancements enable Ferret-UI 2 to perform complex, user-centered interactions, making it highly versatile and adaptable for the expanding diversity of platform ecosystems. Extensive empiri- cal experiments on referring, grounding, user-centric advanced tasks (comprising 9 subtasks x 5 platforms), GUIDE next-action prediction dataset, and GUI-World multi-platton platform benchmark demonstrate that Ferret-UP UI 2 significantly outperforms Ferret-Ul UI, and also shows strong cross plass-rm transfer capabilities.
INTRODUCTION
User interfaces (UIs) are central human-computer interaction, shaping how users interact with digital systems. The complexity of UIs has evolved with the proliferation of platforms such as smartphones, tablets, web platforms, and smart TVs. Despite this increasing diversity, many current approaches to UI understanding and interaction (Hong et al., 2023 Wang et al. 2024b Kapoor et al.) 2024), particularly those in multi-platie malti-platffom ecosystems, face limitations.
One prominent effort in this space is Ferret-Ul (You et al.) 2024), which has advanced the field of referring and grounding UIs. However, though taking an any-resolution approach (Liu et al. 2024a), Ferret-Ul is constrained by fixed grounding resolution (i.e.. 336 336x672 and 672x336), and focuses on single-type platforms (i.e., mobile devices including iPhone and Android), limiting its applicability in the context of today's 's highly diverse platform landscape. For example, illus- trated in Figure []] one notable difference among these four exemplified platforms is resolution, the native resolution of iPhone differs from that of iPad, Web UI, and also AppleTTV, directly applying Ferret-U UI across these platforms presents significant challenges.. Another major challenge is the lack of platform-specific, high-quality data, given different platforms. Though Ferret-UT's approach to training data generation can be extended to these platforms, it primarily relies on text-based GPT-4 prompting, where bounding boxes are represented in purely textual format. This absence of visual input and spatial relationships between UI elements diminishes the quality of training data, which in turn limits the performance and effectiveness of the resulting model.
To address these limitations, we introduce Ferret-U12 2, a multimodal large language model (MLLM) designed to understand diverse UI screens and respond to user intent through single-step interactions across multiple platforms. Building on the foundation of Ferret-UI (You et al.) 2024). Ferret-UI 2 significantly enhances UI perception and user interaction capabilities via three key advancements: (i) multi-platfonn support, (ii) dynamic high-resolution image encoding, and (iit) high-quality mul- timodal training data generation.
University of Texas Austin. Work done during internship at Apple.

---

Published as a conference paper at ICLR 2025
Deging
Boxt 919
Booz
Box'
Box
Box
Box3
BaxO
What kind of UI widget in [Box0]?
What does the text in [Roed] indicates?
This UI widget is toggle.
The text that reads "10-DAY FORECAST" indicates the shows
What is the cument state of the appeamace toggle and how switch it?
forecast
days.
What the carrest tonaeratuue in Cupertino, and what is the weather condition?
The Steme switch Mode, you tap on radio [Bex2] nest to "Dark", which currently unchecked.
[Box]]. To
The currert temperature in Cupertino is 911 [Blox]], and the weather condition "Mostly Sunny"(Ban2)
How would you interact with the brightees slider to brightess?
the
The curzent beighmeess lovel slighty above mid Jevel.
You can scroll horizonzally on the hourly forecast row [Box3] so weather conditions foe later hours.
To decrease the hrightness, drag the slider [Boc]] bowards left. where the brighmess icon
FOR ALL MANKIND
EHER Box1
Box3
tv
f
A
What does the test in [Box0] say?
What is [Roxh]
The text "Cumplate return coline take it Apple Stere"
doe "Ap Stoce" Apple's devices.
What does the large "For All Mankind" hanner suggest about the of
explore ootor options AirTag FineWeven Key Ring?
bunner suggests that "Pur All Mankind" feetured show content the Apple TV app. The hunncr acts as penmotional cunteré, ansouraging
Top click the dots [Boc]] below the image the AirTig FineWoven Kay Ring.
watch explors the show.
are
screen?
Where open Apple TV app?
The prices are AirTag for $29.00 [Box2] AirTag 4- for 309 [Box3] AirTag FineWoven Key Ring- Rlackberry $05.0 (Ros4)..
To open the Apple TV app, top select the black tile with the Apple TV [Box2].
Real examples of a single Ferret-Ul 2 model interacting with four different platforms (iPhone, iPad, Webpage, and AppleTV) for UI understanding. Refer to Appendix Hor more exam- ples including multi-step interactions.
Figure 1:
2

---

Published as a conference paper at ICLR 2025
First, Ferret-UI 2 extends its compatibility beyond mobile platforms (iPhone and Android), incor- porating additional platforms like tablets, webpages, and smart TVs. Figure illustrates visual examples of Ferret-UI 2 interacting with users across four typical screen types. This multi-platfonn support enables broader applicability and allows the system to seamlessly scale across a variety of user environments.
Second, Ferret-UI 2 supports high-resolution image encoding via the any-resolution method (Liu al. 2024a| Zhang et al. 2024c). However, going beyond that, we introduce an enhanced adaptive gridding approach to maintain perception capabilities at the original resolution of the UI screenshot, ensuring more accurate recognition of visual elements. By leveraging human-collected bounding box annotations, we enhance referring and grounding precision, which allows for a more detailed understanding of UI components and their relationships.
Third, Ferret- UI 2 leverages high-quality training data for both elementary and advanced tasks. For elementary tasks, we convert simple referring and grounding data into conversations, allowing the model to develop a fundamental understanding of diverse UI screens. For advanced tasks, which focus on user-centered, free-form conversations, we replace the text-based GPT-4 prompting (where bounding boxes are described only in text) with GPT-4o using set-of-mark visual prompting (Yang) et al. 2023) for training data generation. This approach enhances spatial understanding of UI el- ements, resulting in higher-quality training data. Additionally, unlike previous methods that use straightforward instructions such as "click on [bbox location]". Ferret-UI 2 performs single-step user-centered interactions. For example, when given command like "please confirm submission", the system understands and executes the intended action, rather than simply following mechanical click instructions. Overall, our contributions are summarized as follows.
We present Ferret-UI 2, multimodal LLM that sets itself apart from previous efforts by support- ing a broader range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV. We upgrade Ferret-Ul UI across multiple fronts, including better instruction-tunin data for model train- ing, high-resolution image encoding for enhanced performance, and new referring and grounding benchmarks tailored for different platforms.
We demonstrate that Ferret-U1 2 advances the UI referring and grounding performance on dif- ferent platforms. On three categories of tasks (referring, grounding, and user-centric advanced tasks, comprising 9 subtasks platforms), Ferret-Ul outperforms Ferret-UI, and also shows competitive performance compared to GPT-4o. Besides, Ferret-Ul 2 also exhibits strong transfer capabilities across platforms. Finally, Ferret-UI- 2 achieved strong performance recent bench- marks like GUIDE (Chawla et al.) 2024) and GUI- World (Chen et al., (022a).
2 RELATED WORK
UI agents have garnered significant attention in recent research, particularly in multimodal models that seek to automate complex UI tasks across diverse platforms. Many works have advanced the field by tackling specific challenges related to single-platform and multi-platfomm UI understanding, interaction, and automation.
Single-Platform UI Agents. Single-platform UI agents focus automating tasks specific device ecosystem, such as Android, iOS, desktop environments, or webpages. On the mobile side, DigiRL (Bai al., 2024). AppAgent V2 (Li et al. 2024c), AutoDroid (Wen al. 2024]) and MobileFlow (Nong et al. 2024) proposed Android agents largeting on human-like interactions. For web-based agents, systems like WebShop Yao al. 2022), WebArena (Zhou al. 2023), LASER (Ma et al. 2023). WebAgent (Gur et al. 2023), Auto WebGLe (La et 2024), Web (He) al. al.] (024) and Agent-E (Abuelsaad et al. (024) explored agents that navigate and perform tasks within web environments, while MindSearch agent Chen et al. (2024b) focused on Al engine for web search. AssisIGUI (Gao et al., (023), OS-Copilot (Wu et al. 2024), SYNAPSE (Zheng et al.) 2023) and UFO (Zhang et al. (2024a) explored the more complex computer OS interaction. These efforts have significantly improved task-specific automation, although their single-plation nature limits plass-pl flexibility.
UI Agents. Multi-platform UI agents have emerged to address the growing com- plexity of device ecosystems. supporting variety of devices and platforms, including mobile, web, and desktop environments. [Zheng et al.] (2024a); Cheng et al. (2024) demonstrated GPT-4V as a generalist agent when grounded. Recent works like CogAgent (Hong et al. 2023) support UT navi-
3

---

Published as a conference paper at ICLR 2025
Multi-platform Uniformed Data
Elementary Taeks
Referring:
Grounding:
.CR
-Wigeet listng
Screen- wide
Widget Label Mapping
Invalid Data
-Widget
-Find test
ciace ficsrion, widget
Filtering
Tappablity
Self- Annotated
prediction
Advanced Tasks
Comprahanaive Description
Surmmaiiz overall functionality
UU'
Picture- only OCR
Set-of-Mark (SoM)
Visual Prompting
Divide
describe their local functionality
Multi-round
generation prompts
Caption-based -asedaaeddd grouncling
Grounded content description
Widget
Welb
Datasot
GPT-4o
Multi-round Interaction QA
LLyyuu hiiraccy recognition
collection:
User- centecenteeed Interaction
screenshats, widget
Interactivity understanding
bounding hoxes and labels
Task-orienied neat step prediction
Figure 2: Illustration of the Core-set data generation pipeline.
gation on both PC webpages and Android devices. Mobile-Agent V2 (Wang et al. 2024a) features Harmony OS and Android OS for non-English and English scenarios. Ferret-U1 You et al. 2024) focuses on mobile UI understanding for both Android and iPhone screenshots using multimodal LLMs (McKinzic et al., 2024 Zhang et al., 2024b)), with a focus on referring and grounding capa- bilities. These agents aim to perform more complex, user-intent-based interactions across multiple device types, paving the way for truly generalist multimodal agents.
UI-Agent Benchmarks. The evaluation of UI agents requires specialized benchmarks to test var- ious aspects of UI interaction, including task execution, navigation, and interaction understanding. Rico (Deka et al. (017)) remains foundational dataset for mobile app interaction, while Mobile-Env (Zhang et al. 2023), AndroidEnv (Toyama et al. 2021), AndroidWorld (Rawles et al.) 2024a). An-
droid in-the Wild (Rawles et al. 2024b), AndroidContro (Li et al. 2024b), and AMEX (Chai al.
2024) provide benchmarks for mobile device control. Windows Agent Arena (Bonatti et al.
2024)
introduces benchmark focusing on PC Windows environment. OSWorld (Xie al. 2024 takes
broader approach by providing benchmarks for agents in real computer environments, including Ubuntu, MacOS, and Windows. Web-based interaction benchmarks include WebSRC (Chen al.,
202 Mind2Web (Deng et al.] 2024). and WebCanvas (Pan et al., (024) focusing structural
reading comprehension and task execution in web environments. OmniACT (Kapoor et al. 2024) supports both desktop and web interfaces. More recently, MobilcAgentBench (Wang al. 2024c) and Vibuelhc (Liu et al., 2024b) have introduced taxonomies designed to evaluate the per- formance of multimodal agents across both mobile and web interfaces. VisualAgentBench (Liu et al., 2024c) expands this with a focus on multimodal LLMS as visual foundation agents. GUI Odyssey (Lu et al. 2024) provides benchmarks for cross-app navigation. GUI-World (Chen al.
2024a) pioneer in covering multi-platform benchmarking, while CRAB (Xu et al.) (024) further cross-environment tasks for GUI agents. These benchmarks contribute growing need for
unified, multi pllti--p evaluations that can assess UI agents' adaptability, precision, and efficiency.
Compared to the aforementioned works, Ferret-UI 2 the first target universal UI understanding across diverse platforms, including smartphones, tablets, web interfaces, and smart TVs. It focuses on foundational capabilities like fine- grainrd referring, grounding, and reasoning, aiming create a generalist agent for versatile UI navigation.
FERRET-UL UI 2
In this section, first describe how curate training datasets from the raw data annotations [Section[3.1 and then describe the model architecture (Section B.2).
3.1 DATASET CONSTRUCTION
We construct our own dataset in order to train a strong multi-platfone UI understanding model. A flow diagram of our complete dataset generation pipeline is shown in Figure
4

---

Published as a conference paper at ICLR 2025
Table I: A summary of datasets across various platforms used train Ferret-UI The screenshot
resolution statistics are shown in Appendix
Training Data
Platform
#Images (k)
Task Types
iPhone
112
Elementary Tasks
Android
61
(Referring, Grounding),
Core-set
iPad
19
Advanced Tasks (Comprehensive Description, Multi-Roune Perception QA and Interaction QA)
Web
321
AppleTV
16
GUIDE
Web
51
Next Action Prediction
GroundUI-18k
Web
18
Simple Interaction
Spotlight
Android
66
Screen2Word, Widget Caption, Taperception
Raw Annotation Collection. The primary dataset used for training Ferret-UI 2 is combination of data from various platform types, including iPhone, Android, iPad, Webpage, and AppleTVV. The data collection process varies depending on the platform type:
iPhone, iPad, and AppleTV: We use human-collected iPhone, iPad, and AppleTV data under diverse usage scenarios and numan-annotater widget bounding box coordinates and labels. To save annotation costs, we do not collect text annotations; instead, text bounding boxes are replaced by screen-wiid wide detedetected text and bounding boxes using an OCR confidence threshold of 0.5.
The web data is derived from the WebUI dataset (Wu et al. 2023). Bounding boxes of all types of UI widgets and text annotations for non-picture widgets directly parsed from the source HTML view hierarchy tree, providing high-quality annotations. For picture widgets, we further use OCR detect texts contained in the pictures.
Webpage:
The Android data for screenshots, bounding boxes, and text annotations is transformed from the RICO dataset (Deka et al. 2017). Similar to the WebUI dataset, we also perform picture- only OCR to complete the missing text annotations in picture widgets.
Android:
Use of this data for the purpose of this paper was approved by our organization's 's legal team.
For the collected data, we perform data filtering including: (i) filter out or narrow down out- of-bound bounding boxes and remove empty screenshots with remaining bounding boxes after box filtering; (ii) since we do not intend to add multilingual support for the Ferret-UI model, screenshots with more than 5% non-ASCII characters in the text annotations also removed.
Despite the different types of label spaces from various sources, we filter out bounding boxes asso- ciated with less relevant labels (e.g., UI types) and uniformly map the remaining labels to common label space containing 13 classes: 'Checkbox", 'Button', 'Container', 'Dialog', "Icon', 'PageCon- trol', 'Picture', "SegmentedControlf 'Slider', "TabBar', "Text', "TextField", and "Toggle', and obtain a multi-p poltt--plttfom uniformed dataset with raw UI widget annotations. We provide the original label statistics and their converted labels in Appendix
We name the above-collected screenshot dataset the Core-set, which we will use to construct the ele- mentary and advanced task data. Besides, we also employ third-party training datasets enrich our data souree and avoid overfitting our predefined tasks. A complete statistics of the training dataset of Ferret-UI 2 is summarized Table indicating that the dataset distribution is very unbalanced across different platforms. In particular, the number of iPad and AppleTTV screenshots is signifi- cantly smaller than that of other platforms. To address this, we (i) assign different loss weights to different platforms during training, and (ii) generate all three types of advanced tasks for each ex- ample iPad and TVpleTV platforms and generate only type of advanced task for each example of other platforms.
Compared to the training dataset of Ferret-UI, which relies on model-detected bounding boxes, Ferret- UI 2's training dataset predominantly utilizes either human-collected annotations or bounding boxes directly parsed from the source HTML, resulting in a significant improvement in annotation quality, evidenced later in quantitative evaluation in Section 4.2
5

---

Published as a conference paper at ICLR 2025
moy
Tusk type: Multi-round Interaction QA
Settings
Display & Brightness
Conversations:
User:
<image>
How switch dark mode this screen?
Model:
You cun switch to dark mode by tapping the "Dark" option [<WIDGET 34>] and then selecting circle [<WIDGET_ undereath it.
Ish
User:
What the "True Tonc' option do?
Model:
Automatic
The "True Tone option (<WIDGET 21>] automatically udapts the iPhone display based
ambient lighting conditions to make colors appear consistent [<WIDGET 39> <WIDGETT _3>]]
User:
How adjust brightness?
Model:
Thue Tone
You can adjust the brightness using the slider (<WIDGET 4>] under the 'BRIGHTNESS'
cowatecalyeeee enape thf disste haved undent venes))
section [<WIDGET 235]..
User:
Is there a way to loggle the automatic brightness?
Night Shiit
Model:
Yes, you can toggle automatic brightness switching the toggle [<WIDGET_
'Automatic" [<WIDGET [1>]].
Auto Auto-Lock
20 Seconds
User:
8M
se to Wake"
Where
Wake" Teature?
Model:
The "Raise to Wake" feature is currently on, can top on the toggle [<WIDGET_SS _55+] to
Jest Size
switch off.
Hald
Figure 3: Example of set-of mark visual prompting Yang et al. 2023) (left) and one of its generated advanced task training examples (right).
Task Data Generation. For task data generation, we follow the paradigm of Ferret-UT data con- struction, which includes both elementary tasks and advanced tasks.
Elementary tasks (Figure consist of 3 referring tasks and 3 grounding tasks. Specifically, referring tasks include (i) OCR: recognizing the text given text bounding box, (fi) widget classification: predict the UI type of the elements, and (dit) tapperbility: predict whether the selected widget is tappable for interaction; meanwhile, grounding tasks include (i) widget listing: list all the widgets in the screen, (ii) find text: find the location of given text, and (iii) find widget: find the widget given the widget description.
For advanced tasks, we prompt GPT-4o with bounding box annotations of given screenshot and require GPT-4o to generate QA tasks related to the UI widgets in the screenshot. Unlike Ferret- UI, which mainly focuses spatial descriptions due to the limitation of using textual prompts without image information (i.e., screenshots) for bounding box annotations, Ferret-Ul leverages GPT-4o generate advanced task data that covers a variety of aspects of UI understanding. This is possible because GPT-4o demonstrates an improved ability to comprehend the spatial relationships between UI widgets when provided with the screenshot input. Specifically, we prompt GPT-44o to generate 3 types of advanced tasks (shown in Figure 2 including: (i) comprehensive description: describe global and local functionalities of the screen, (ii) mudti-round perception QA: multi-round question answering regarding the UI perception capabilities, and (iii) multi-round interaction QA: multi-round question answering regarding the single-step and user-centric UI interactions based on the current screen status. More detailed requirements and prompts for GPT-4o when generating advanced tasks are provided in Appendix
We empirically find it hard for GPT-4o to find the location of referred UI widgets with the original screenshot as input (i.e,, bad grounding capability). To address this, we use Set-of-Mark (SoM) visual prompting (Yang et al. 2023) when generating multi-round perception and interaction QA training samples. One example of SoM prompting and its generated data sample is shown in Fig- ure 3 where each UI widget is marked with comee-style bounding box and unique number tag for easy identification. Furthermore, the same class of UI widgets have the same color for visual prompting to help GPT-4o better differentiate the bounding boxes of spatially close nested wid- gets. Please refer to Figure for visual prompts on other platforms, and Appendix D for additional examples of generated data for advanced tasks.

---

Published as a conference paper at ICLR 2025
can tap on [980, 785, 1011, 796].
Large Language Model
Flatten
T
Flatten
Global Embedding
Patch Embeddings
Visual Sampler
Decore alcrkccam
P PIRO
CLIP Encoder
CLIP Encoder
PIRO
How do pre-order iPhone
iPhnn
Adaptive Gridding
Figure 4: Overview of the Ferret-U1 2 model architecture, which allows for seamless UI understand- ing and user-centered single-step interactions with high-resolution support.
Algorithm I: Adaptive N -ggridding
Require: Original resolution: h, grid size: 336 336, size limit N
Ensure: Optimal gridding size Nw and Nn. (Nw,, N, E INT)
N Waree' N hoss 0, Abeat f- 00, Nwo + 336 ho + 336
2: for Nw = 1 to N do
D Traverse all grid configuarations
3: for N₂, 1 to N
N do No Nn
4:
Auseect
Nng
Nua
Ns. N
Nwg
Noo
Get aspect ratio change
N MorNs-Nwug xNNagl
5:
Apixel +
D Get relative pixel change for resizing
Nug x Nip
6:
if Abost Auspoct A Apixel then
7:
(Narea Nhmaa (Nw, N₂)
8:
Abest Aspeca pixel
9:
end end for
11:
end for
return (Nu Nowa)
As aforementioned in T addition to the tasks generated on the Core-set, we augment training data with additional third-party datasets, including GroundU1-18k (Zheng et al. 2024b), GUIDE (Chawla et al. 2024) and Spotlight (Li & Li 2023
3.2 MODEL ARCHITECTURE
As shown Figure 4 the model architecture of Ferret-Ul directly builds upon Ferret-Ul (You et al. 2024), which uses the Any-Resolution (AnyRes) method (Liu et al. (222a)) to enhance refer- ring and grounding, enabling the encoder to capture diverse image resolutions.
Specifically, the CLIP image encoder first extracts both global (derived from the low-resolution overview image) and local features (corresponding to high-resolution sub-images) from the UI screenshot. Then, these image features are flattened and sent into the LLM. The Visual Sampler identifies and selects the relevant UI regions based on user instructions. The model then outputs grounded descriptions for perception or interaction with the UI elements.
Adaptive Gridding. Local image features are extracted by calculating the optimal grid size using proposed adaptive N-gridding mechanism, then resizing and encoding the visual features of

---

Published as a conference paper at ICLR 2025
Table 2: Results on our constructed benchmarks for elementary and advanced tasks, as well as the GUIDE benchmark (Chawla et al. 2024). Results on elementary and advanced tasks are averaged over all platforms, including iPhone, Android, iPad, Webpage, and AppleTTV. Each platform includes 6 elementary tasks and 3 advanced tasks. SeeClick model (Cheng et al. (0224) trained on their original data is compared. (t) tasks that require referring, GPT-4o is equipped with set-of-mark (SoM) prompting by adding red rectangular box to screenshots for the referred widget. Note that SoM visual prompting is not used for Ferret-UI and U11
Elementary
Advanced
GUIDE Bench
Model
Backbone
Refer
Ground
GPT-4o Score
Multi-ToU
BertScore
IoU
Ferret-Ul
Vicuna-Bi 13B
64.15
57.22
45.81
18.75
41.15
26.91
Gemma-28
75.20
78.13
80.25
40.51
83.71
51.13
Femet-U1
LJama3-8B
80.28
82.79
89.73
41.15
91.37
55.78
Vicuna- 13B
81.34
81.31
86.25
41.71
88.81
54.71
SceClick Cheng al. 2024
QWen-VL-9.0h
51.58
62.82
67.49
21.56
54.70
39.51
GPT-4
56.47
12.14
77.73
7.06
75.31
9.64
GPT-4o + Promppt
87.91
84.33
7.36
each grid. This is a key model innovation compared to Ferret-U1. As shown in Algorithm 1 the optimal gridding size Nw and N₂, determined when the gridding and resizing based on (NV No.NNN, N₂) lead to minimal aspect ratio change times the relative pixel number change, under the constraint Nw + N₂ S N, where is the size limit. With the size limit N, the total number of grids is upper bounded by - Compared to the AnyRes module that has unbounded cost, the key differences of automatically finds the optimal gridding configuration, i.e., least resolution distortion (aspect ratio change and pixel number change) within predefined inference cost limit that N N₂ N2 which is both information-preservin and efficient for local encoding. With adaptive gridding, Ferret- UI 2 understands UI screens and provides user-centered interactions with an optimal configuration at any resolution given the inference cost limit specified as N.
adaptive N -rgdddiin
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Training Data. The training datasets are summarized Table [] which can be divided into two categories: (i) datasets constructed by our own, which include elementary task data and ad- vanced task data across all platforms as introduced in Section 3.1 and (ii) public datasets including GroundU1-18k (Zheng al. 2024b), simple user-centered interaction dataset webpage screen- shots, GUIDE (Chawla et al. 2024). a next-action prediction dataset on webpage screenshots, and Spotlight (Li 2023), ,, Android UI understanding and interaction dataset.
Model. Following Ferret-UI YYou et al. 2024), Ferret-Ul 2 uses CLIP ViT-L/14 model as the
image encoder; for the LLM backbone, besides Vicuna- (Chiang et al. 2023) used in the original Ferret-UI, we also tried 2 additional L.MMs at mobile scales, including Gemma-2B (Team) (et al., 2024) and Llama3-8B (Dubey et al.) 2024). As dynamic high-resolution image encoding. we set the size limit N to 8, so that the maximal grid number is 16 for adaptive gridding.
Evaluation. At high level, model evaluation falls into two broad categories: (i) benchmarks we constructed, and (ii) public benchmarks. For our benchmarks, we created a total of 45, including elementary tasks and 3 advanced tasks per platform type, across 5 platforms. For elementary tasks, we follow the evaluation metrics outlined by You et al. (2023). For advanced tasks, we use GPT- 4o to score generated answers for a given screenshot and user query, visually prompting GPT-4o with red rectangular bounding box. Advanced tasks are tested using GPT-44o evaluation score and multi-loU, The multi-loU is calculated by first matching predicted bounding boxes with ground truth bounding boxes and then calculating the average IoU of each pair of bounding boxes (IoU = 0 if no match). Furthermore, we conduct a next-action prediction test given previous action history on the GUIDE benchmark (Chawla et al. 2024), and evaluate the semantic similarity w.n.t. the reference answer and grounding Intersection-over Union (IoU). Additionally, we evaluate our model on the recently released GUI-World benchmark (Chen et al. 2024a) on the supported platforms following
8

---

Published as a conference paper at ICLR 2025
Table Zero-shot performance of Ferret-UI 2 the GUI-World benchmark (Chnn et
2024a
GPT-4 Score
Model
iOS
Android
Webpage
Average
MiniGPT4Video Ataallah et al.
2024
1.501
1.342
1.521
1.455
VideoChat2 Li et al.
2024a
2.169
2.119
2.221
2.170
Chat-Univi Jin
20/24
2.337
2,390
2.349
2.359
GUI-Vid Chen ct al.
2024a
2.773
2.572
2.957
2.767
QWen- VL-MAX
2023
2.779
2.309
2.656
2,580
SeeClick Cheng et al.
(2024)
2,614
2,650
2.848
2.704
Ferret-Ul You et
2024
2.713
2.791
2.411
2.638
Ferret-U12
2,881
2,954
3,013
2,948
Gemini-Pro 1.5 Reid et al.
2024
3.213
3,220
3,452
3.295
GP
3,558
3.561
3,740
3.619
Table 4: Zero-shot cross-platform transfer results of Ferret-UI 2. For simplicity, train and test the model only using data corresponding to the elementary tasks.
Test Referring
Test Grounding
Training
iPhone
iPad
AppleTV
Web
Android
iPhone
AppleTV
Web
Android
iPhone
86.3
68.1
31.2
45.3
71.2
84.1
65.2
43.1
51.7
63.1
iPad
67.5
80.2
40.7
51.5
63.3
64.5
82.1
32.1
38.5
53.8
AppleTV
29.1
45.1
79.3
54.2
36.4
33.7
41.2
81.6
52.1
29.7
59.2
57.4
41.2
85.5
41.7
54.0
51.2
46.5
87.5
45.9
Android
72.5
51.2
86.2
66.7
29.7
44.1
83.9
the original GPT-4 evaluation protocol in Chen et al. (2024a). which does not include evaluating the grounding capability for interaction-relate tasks.
4.2 EXPERIMENT RESULTS
Main results. Our main results are summarized in Table 2 which shows the comparative perfor- mance of different models on our constructed elementary and advanced tasks, as well as the GUIDE benchmark (Chawla et al.) 2024). Note, that for each data entry corresponding the elementary and advanced tasks, it is an average across all platforms. The detailed results on each platform are provided in Table 6 Appendix Below, we highlight few observations.
Ferret-UI 2, powered by Llama-3-8B, delivers the best results across most metrics. achieves the highest GPT-4o score on advanced tasks, with a notable 89.73, surpassing Ferret-Ul by 43.92 points and GPT-4o by 12.0 points. Notably, Ferret-U12 with Llama-3 1amma-3-8B also achieves the highest IoU score on the GUIDE benchmark with 55.78, indicating superior grounding capability.
Ferret-UI 2, equipped with Vicuna-13B. also performs well, e.g., achieving a strong Multi-loU score of 41.71 advanced tasks. Despite being six times smaller, Ferret-UI 2 with Gemma-2B delivers competitive performance across the board.
In contrast, GPT-4o struggles with fine-graines UI understanding, shown by its low referring (56.47) and grounding (12. 14) scores the elementary tasks. Its Multi-loU and loU scores in advanced tasks and the GUIDE benchmark are also low.
Overall, the results demonstrate the versatility of Ferret-UI handling UI understanding tasks across different platforms.
Results on GUI-World. of the box, further test model on the recently released GUI-World benchmark (Chen et al.) 2024a). Results are summarized in Table 3 Clearly, Ferret-UP 2 does not overfit the training data and can generalize well to the test data in the wild. Notably, Ferret-U12 outperforms GUI-Vid (Chen cet al. 2024a). a model developed in the Wor-d paper, on supported platforms including iOS, Android, and Webpages.
To further demonstrate the zero-shot performance of using Ferret-Ul out

---

Published as a conference paper at ICLR 2025
Table 5: Ablation results of the architecture and dataset improvements of Ferret-UI 2 w.r.t. Ferret- Ul-anyRes (You et al. 2024), i.e., the high-resolution version of Ferret-Ul equipped with the any- resolution module. iPhone v1 refers to the dataset on the iPhone platform originally used by Ferret- UI. while iPhone v2 is the data used by Ferret-UI 2. Models are evaluated the advanced tasks.
iPhone vl
iPhone v2
Training Data
Model
GPT-4o Score
Multi-loU
GPT-4o Score
Multi-loU
91.3
36.89
68.3
27.13
iPhone vl
Ferret-U1
93.7 (+2.4)
37.12 (+0.23)
70.2 (+1.9)
28.21 (+1.08)
Ferret-Ul-anyKee
86.2
35.89
85.97
39.81
iPhone
Ferret-U12
88.1 (+1.9)
36.43 (+0.54)
89.7 (+3.73)
41.73 (+1.92)
4.3 ABLATION STUDY
Cross-Platform Transferability. In Table we evaluate the zero-shot platform (domain) transfer capability of the Ferret-U1 2 model by training it on elementary tasks from one platform and testing it on other platforms. These results provide insights into how well the model generalizes across different platforms. We observe similar performance patterns across tasks. Specifically,
iPhone transfers well to iPad and Android platforms both tasks, achieving least 68.1 and 71.2 scores on referring tasks and 65.2 and 63.1 scores on grounding tasks due to its diverse screenshot contents (because of the large screenshot number) and similar resolutions and aspect ratios with other two platforms. iPad and Android also transfer fairly well to iPhone at around 65 scores.
AvpleTV and Web do not transfer very well to the mobile domains, including iPhone, Android, and iPad, achieving the highest referring score of 59.2 and grounding score of 54.0, possibly be- cause they are mostly landscape screenshots, which is in contrast the mostly portrait screenshots in mobile platforms. Models trained on other domains all achieve poor performance on the Ap- pleTVV test domain, with the highest score of around 40 both kinds of tasks, which is reasonable due to a large domain gap in terms of Apple'TV's content distribution compared to other domains.
The results suggest that (i) iPhone, iPad and Android have similar content distribution, which helps them generalize to each other; (ii) models trained on more diverse contents (e.g., around 100k iPhone data) can generalize better to other platforms; (iii) platforms with similar resolution and aspect ratios may transfer better to each other; and (iv) good transferability among some of these platforms contributes to the cross-platfone plossfolatoom performance of UI
Ablation on Architecture and Dataset Improvements. Table 5 presents a comparison between the Ferret-UI and Ferret-UI model trained on different versions of the iPhone dataset. Models are evaluated on the test set corresponding to advanced tasks. The results indicate that both the adaptive N N-gridding and improved dataset (iPhone v2) contribute to performance gains.
Specifically, when evaluated on the iPhone v1 test set, Ferret-UI 2 shows slight improvement over Ferret-UI, with the GPT-4o score increased from 91.3 to 93.7 and the Multi-loU score increased from 36.89 to 37.12. However, the improvements are more pronounced on the iPhone v2 dataset. Here, Ferret-Ur 2 achieves GPT-4o score of 89.7, outperforming Ferret-UT's 85.97, along with substantial Multi-loU score boost from 39.81 to 41.73. These results suggest that while both architecture and dataset enhancements contribute to overall performance, the new dataset plays a more significant role in driving improvements, particularly more challenging tasks.
5 CONCLUSIONS
In this paper, we presented Ferret-Ul 2, multimodal large language model designed to improve UI understanding and interaction across diverse platforms. With multi-platform support, high- resolution image encoding with adaptive gridding, and improved data generation, Ferret-U1 2 outper- forms Ferret-UI on all tested benchmarks. The model demonstrates strong zero-shot transferability across platforms, establishing Ferret-Ul UI? 2 as solid foundation for universal UI understanding. Fu- ture work will focus on incorporating additional platform types and building a generalist agent for universal UI navigation.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
The use of UI understanding data for this paper was approved by our organization's legal team. We have strictly adhered ethical standards, ensuring that no private or sensitive information has been used or compromised.
ACKNOWLEDGEMENTS
The authors would like to thank Forrest Huang, Zhen Yang, Hаoxuan You, Amanda Swearngin, Alexander Toshev and Yang Zhao for valuable guidance, suggestions, and feedback.
REFERENCES
Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. Agent-e: From autonomous web navigation to foundational design principles in agen- tic systems. arXiv preprint arXiv:2407.13033 2024.
Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal Ilms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.0344 034133, 2024.
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Di- girl: Training inlthe-wild device-control agents with autonomous reinforcement learning. arXiv preprint ar arXiv:2406. 11896, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint 2338 /2966, 2023.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Justin Wa- gle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, et Windows agent arena: Evaluating multi-modal os agents at scale. arXiv preprint arXiv:2409.08266. 2024.
Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. Amex: Android multi inllii-nnotttionn expo dataset for mobile gui agents. arXiv preprint arXiv:2407.144 2024.
Rajat Chawla, Adarsh Jha, Muskaan Kumar, Mukunda NS, and Ishaan Bhola. Guide: Graphical user interface data for execution. arXiv preprint arXiv:2404. 16048, 2024.
Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, et al. Gui-world: A dataset for gui-oriented multimodal 11m- based agents.
arXiv preprint arXiv:2406. 10819, 2024a.
Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. Websre: A dataset for web-based structural reading comprehension. arXiv preprint 10Xi::2001.09655. 2021.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao. Mindsearch: Mimicking human minds elicits deep searcher. arXiv preprint arXiv:2407.20183, 2024b.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiy- ong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401. 10935, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuma: An open-source chatbot impresssing gpt-4 with 90%* chatgpt quality, March 2023. URL hetps:
7/1 mnsyssorrg/bllg//223333333

---

Published as a conference paper at ICLR 2025
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: mobile app dataset for building data-driven design ap- plications. In Proceedings of the 30th anmual ACM symposium user interface software and technology, pp. 845-854, 2017.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. NeurlPS, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 2rXi::2477.2 1783, 2024.
Difei Gao, Lei Ji. Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, et al. Assistgui: Task-oriented desktop graphical user interface automation. arXiv preprint arXiv:2312. 13 108, 2023.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and pro- gram synthesis. arXiv preprint arXiv:2307. 220.::230...88666 2023.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.139) 2024.
Wenyi Hong, Weihan Wang. Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint (rXxv:2312.089/41 2023.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In CVPR, 2024.
Raghay Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. arXiv preprint arXiv:2402.17553 2024.
Hanyu Lai, Xiao Liu, lat Long long, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: Bootstrap and reinforce large lan- guage model-based web navigating agent. arXiv preprint arXiv:2404.0368 2024.
Gang Li and Yang Li. Spotlight: Mobile ui understanding using vision-language models with focus, 2023.
Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024a.
Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale computer control agents. arXiv preprint arXiv:2406.03679, 2024b.
Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Appagent v2: Advanced agent for flexible mobile interactions. arXiv preprint arXiv:2408. 11824, 2024c.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://
llvaavv... .ivvavvl.giae thub. i wtt.o/lll///e./
Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal Ilms evolved web page understanding and grounding? arXiv preprint 4444552404.05555 2024b.
12

---

Published as a conference paper at ICLR 2025
Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long. Yifan Xu, Xixuan Song. Shudan Zhang. Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06327, 2024c.
Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation mobile devices. arXiv preprint arXiv:2406.080 08451, 2024.
Kaixin Ma, Hongming Zhang, Hongwei Wang. Xiaoman Pan, Wenhao Yu, and Dong Yu. Laser: Lim agent with state-space exploration for web navigation. arXiv preprint arXiv:2309.08172, 2023.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng. Floris Weers, et al. Mml: Methods, analysis & insights from multimodal IIm pre-training. arXiv preprint arXiv:2403.09611 2024.
Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, and Wenhao Xu. Mobileflow: A multimodal Ilm for mobile gui agent. arXiv preprint axXiv2440.04434, 2024.
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. Webcanvas: Benchmarking web agents in online environ- ments. arXiv preprini arXiv:2406. 12373, 2024.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint S.lv:2405.14577. 2024a.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. NeurlPS, 2024b.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillierap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055500 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00 001 18, 2024.
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. arXiv preprint arXiv:2 130.1331,, 2021.
Junyang Wang. Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv preprint arXiv:2406.01014 2024a.
Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Ji- tao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception, 2024b.
Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. Mobilcagentbench: An efficient and user-friendly benchmark for mobile 11m agents. arXiv preprint arXiv:2406.08184 2024c.
Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang. Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Lim-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pp. 543 557, 2024.
Jason Wu, Siyan Wang, Siman Shen, Yi-Hao Peng, Jeffrey Nichols, and Jeffrey Bigham. Webui: A dataset for enhancing visual ui understanding with web semantics. ACM Conference on Human Factors in Computing Systems (CHI), 2023.
13

---

Published as a conference paper at ICLR 2025
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-imprrvemenn. arXiv preprint 47Xhv:240..07456, 2024.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Stheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972 2024.
Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, et Crab: Cross-environment agent benchmark for multimodal language model agents. arXiv preprint arXiv:2407001511, 2024.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310.1 11411, 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. NeurlPS, 2022.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity, 2023.
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearggin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms. arXiv preprint 14X7:22404.0579, 2024.
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. Ufo: A ui-focused agent for windows os interaction. arXiv preprint arXiv:2402.07939, 2024a.
Danyang Zhang, Hongshen Xu, Zihan Zhao, Lu Chen, Ruisheng Cao, and Kai Yu. Mobile-env: evaluation platform and benchmark for Ilm-gui interaction. arXiv preprint arXiv:2305.088 144444 2023.
Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, Mml. 5: Methods, analysis insights from multimodal Ilm fine-tuning. arXiv preprint arXiv:2409. 20566, 2024b.
Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu-Jui Fu, William Yang Wang. Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint arwhv:2404.0797.. 2024c.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is generalist web agent, if grounded. arXiv preprini arXiv:2401.011 4411111144 2024a.
Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exe exxemmlla prompting with memory for computer control. In ICLR, 2023.
Longtao Zheng, Zhiyuan Huang, Zhenghai Xue, Xinrun Wang. Bo An, and Shuicheng Yan. Agentstudio: A toolkit for building general virtual agents. arXiv preprint urXiv:2403.179113 2024b.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena; A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307. 13854, 2023.
14

---

Published as a conference paper at ICLR 2025
A DETAILED RESULTS ON ELEMENTARY AND ADVANCED TASKS
O the performance on 9 subtasks x 5 platforms of the Ferret-UI 2 model. We empirically find that OCR, widget listing, and comprehensive description are relatively difficult tasks compared to other tasks within the same task type.
Table 6: The performance breakdown of Ferret-U1 2 with Llama-3-8h backbone. Note that we only report GPT-4o evaluation scores and omit the multi-IoU scores for advanced tasks for simplicity.
Test Domain
Task type
Task
iPhone
iPad
AppleTV
Web
Android
OCR
75.3
69.3
74.3
80.5
74.5
Refer
Widget Classify
79.1
78.7
82.5
83.6
82.0
Tapperbility
89.2
86.0
84.3
85.6
Widget Listing
76.7
74.9
71.6
79.7
76.8
Ground
Find Text
85.2
84.0
82.2
90.2
82.2
Find Widget
88.6
86.8
87.7
87.8
86.4
Comprehensive
89.0
86.5
84.1
83.7
85.6
Advanced Tasks
Perception
94.1
87.5
86.6
93.2
92.1
Interaction
93.5
92.7
86.1
96.4
94.7
B GROUNDING PERFORMANCE ON SCREENSPOT BENCHMARK
We evaluate Ferret-Ul on the ScreenSpot (Cheng al. 2024) benchmark for grounding perfor- mance on unseen data. Results are shown in Table Ferret-U1 achieves 54.0% average accuracy, outperforming CogAgent (Hong et al. 2023) and SeeClick (Cheng et al., 2024). and in particu- lar, achieving good performance on supported Mobile and Web platforms while achieving fair performance on the unseen Desktop platform.
Table 7: Performance of Ferret-UI 2 on the ScreenSpot benchmark (Cheng et al. 2024 I/W refers to (con/Widget data type. Note that the data distribution of ScreenSpot is unseen during Ferret-U UT 2 training. The results indicate that Ferret-UI 2 achieve superior grounding accuracy on Mobile and Web screenshots, while underperforms other two models on the unseen Desktop screenshots, which means Ferret-UI 2 transfers well to unseen data in supported platforms.
Mobile
Desktop
Web
Model
Model Size
Avg
Text
I/W
Text
I/W
Text
I/W
CogAgent
8B
67.0%
24.0%
74.2%
20.0%
70.4%
28.6%
47.4%
SeeClick
9.6B
78.0%
52.0%
72.2%
30.0% %
55.7%
32.5%
53.4%
Ferret-Ul UI 2
8B
80.3 %
55.4%
52.1%
21.7%
81.2%
33.5%
54.0%
C DETAILED GPT-40 PROMPTS FOR GENERATING ADVANCED TASK DATA
In this section, we elaborate on how we prompt GPT-4o to generate training data for advanced tasks. In partieular, have the following requirements when generating each type of advanced task.
1. Comprehensive Description: Provide one-sentence description of the overall functionality of the UI page shown in the screenshot. Then, describe the screenshot in detail by dividing into several areas/groups and explaining the functionality of each area/group.
2. Round Perception QA:
15

---

Published as a conference paper at ICLR 2025
Table 8: Statictics of different multi-platform GUI understanding datasets.
Dataset
Sample
Platforms
Tasks
OmniAct
9,802
Desktop, Web
Code Generation
OS-World
369
Desktop, Web
General Control
AITW
715,142
Appsoid(Apps+ Web)
Navigation
GUI-world
12,379
iPhone, Android, Weh, XR, Desktop
GUI understanding, instruction following
Elementary: Referring, Grounding
Ferret-UP UI
123,702
iPhone, Android
Advanced: Function/detailed description,
pereeption and interaction QA
Elementary: Referring, Grounding
2
529,411
iPhone, Android, iPad, AVpleTV, Web
Advanced: Comprehensive description, perception and interaction QA
Basic Perceptual Understanding: Interpret the content of referred widgets, ground the positions of the widgets based on the widget descriptions, and describe the functionality and status (enabled, disabled, selected, hovered) of each widget.
Contextual Awareness: Understand the context in which UI widgets are presented, includ- ing the relationships between different UI components and how they contribute to the overall user experience.
(c) Layout and Hierarchy Recognition: Recognize how widgets are grouped and nested within the UI layout.
Multi-Round Interaction QA:
Generate instructions for user-centered interaction. For example, "Please help me confirm submission" instead of "please click on [Box0] button"".
(b) Identify and interact with buttons, links, icons, scrollbars, and toggles, and recognize wid- gets that cannot be interacted with, such static text and background images. Identify and interact with input fields, dropdowns, checkboxes, and radio buttons, well as their current states (e.g., checked, unchecked, filled, empty) in the context of specific interaction tasks.
(d) Predict the current step for potential navigation paths based on currently visible widgets like tabs, menus, and breadcrumbs controls.
D ADVANCED TASK EXAMPLES
Comprehensive Description
This UI page for adjusting settings related downloads, language, news, and notifications.
OVERALL:
DETAIL:
Top Navigation Bar: Contains menu options such as Home, Account, Settings (currently selected), and Downloads, allowing users to navigate between different sections.
Games Section: Features checkbox labeled "Download over WiFi only," which is cur- rently checked, indicating that downloads are restricted to WiFi connections.
Language Section: Offers a dropdown menu to select the language, currently set to En- glish, with pink arrow indicating
options.
Title Screen Section: Includes checkbox for "Show Sago News." which is checked, meaning news updates will displayed on the title screen.
Notifications Section: Contains checkbox for "Display notifications." which is unchecked, allowing the user to enable or disable notifications.
Close Button: Located in the top right corner, represented by an 'X', used to close the settings page.
16

---

Published as a conference paper at ICLR 2025
Multi-Round Perception QA
Q: What does the "SERVEWARE" text represent?
A: The text "SERVEWARE" [718, 72, 948, 108] is likely category label indicating that the items displayed are related to serving ware.
Q: Can you identify any product names and their corresponding prices?
A: Yes, the product "ARV BROLLOP" [35, 847, 233, 907] priced $12.99 [10, 112,
32] and "IKEA 365+" [609, 787, 809, 823] is priced at $2.99 [1174, 1336, 1627, 2119].
Q: What the function of the shopping carl icon?
A: The shopping cart icon [630, 2157, 690, 2217] is typically used to view the items in your
cart proceed checkout.
Q: How is the product rating displayed?
A: The product rating is displayed with star symbols and numerical rating. For example,
"ARV BRÖLLOP" has a rating of 4.9 next to the stars [[38, 956, 100, 986), [248, 1004, 294, 1030]]-
Q: What does the "Filter" button do?
A: The "Filter" button [1564, 170, 1630, 198] is likely used to apply specific criteria to narrow down the product search results.
Q: Are there any navigation elements on the screen?
A: Yes, there seems to be a navigation bar the bottom with icons for different sections,
like a house icon for home [611, 1, 268, 1057, 714] and pencil icon [1320, 2157, 1380, 2217] which might be for editing or creating a list.
Multi-Round Interaction QA
Q: How can I refresh the content shown?
A: You can tap on the refresh icon [272, 736, 424, 882) to reload the content.
Q: What does the "General" tag refer to?
A: The "General" tag [1074, 900, 1443, 926] categorizes the content related general events, like the Caltech Beaver Name Reveal.
Q: How can I view more recent activities?
A: You can tap the "Recent" section label [90, 598, 202, 630] to view more recent activities.
Q: What is the logo displayed this page?
A: The logo at the top of the page is the SCIAC Network logo [198, 56, 444, 128] represent-
ing the network conducting the event.
Q: How do check the institution hosting the event?
A: The event is hosted by the California Institute of Technology, shown in the text [11, 499, 483, 542] below the event details.
E DATASET COMPARISON
In this section, we compare the dataset of Ferret-UI 2 with previous multi-platform datasets as shown in Table 8 In particular, outline the data generation pipeline of Ferret-UI (You al. 2024h in Figure 6 Through a comparison of Figure 2 and 6 we highlight the following key differences between Ul 2 and Ferret-Ul:
support: The training data of Ferret-UI 2 consists of 5 platforms compared to platforms of Ferret-UI.
17

---

Published as a conference paper at ICLR 2025
FIVE GUYS PLUMBING WARREN
586-382-9378 1110
UPEATO)
Plumbing Service Warren MI
2 Solls
On1
internedile stee Vicece
30
Adtanooo Midinn
Figure 5: Examples of visual prompting using GPT-4o generate task data for Multi-Roune Percep- tion QA and Mulli-Rounm Interaction QA. Each UI widget is annotated with a corner-style bounding box, where only the corners of the widget are highlighted by small lines, leaving the rest of the box open. This minimalistic bounding style is accompanied by a unique number tag placed near one of the corners, making it easy to identify and reference specific UI widgets for further interaction or perception analysis.
Elementary Tasks
Dual-platform Data
Referrin::
Grounding:
Screen- wide
Invalid Data Filtering
OCR
WWdgett littin
Widget
Find trot
Icn recogntion
Find loon
Pseudo Annotations
widget
Widget Label Mapping
Advanced Tasks
GPT-4
Function Inference
-Smmmarzz overall UI functionaliyy
Detailed description
UT
Descr ibe spotial al relations of UI el elements
Android
RICO Dataset
Query with textual prampts
Multi-round QA
Percepttio interaction
Figure 6: Illustration of the data generation pipeline of Ferret-UI You et 2024
Raw annotation qualities: The majority part of bounding boxes, labels, and on-screen text anno- tations of Ferret-Ul 2 are either extracted from source data or annotated by humans, while these annotations of Ferret-UI are all generated from model detection.
Bounding box prompting: When generating advanced tasks, Ferret-UI 2 uses GPT-4+SSMM visual prompting for bounding boxes, while Ferret-Ul uses purely textual prompting containing coordinates, bbox labels, and text annotations.
18

---

Published as a conference paper at ICLR 2025
Advanced task quality: Due to the constraint of textual prompting, the Ferret-Ul pipeline not perceive the visual content of UI elements, thus limiting the quality and content diversity of generated advanced tasks.
F RESOLUTION STATISTICS
In this section, present the resolution statisties for images collected from various platforms, categorized by device types and resolutions, as summarized in Table 9
Table 9: Resolution statistics by device type.
Device Type
Resolution
Number of Images
828x
83,250
1125x2436
6,055
iPhone
1792x828
4,686
2126x1125
104
2224x1668
4,829
iPad
1668 x2224
14,312
1242x2208
19
AppleTTV
1920x 1080
16,152
1280x720
53,500
1366x768
53,500
1536x 864
53,500
WebUI
1920 1080
53,500
2048 x2332
53,500
1170x2532
53,500
x40
14,092
1080 1920
52, 102
Android
1920 1080
55
960x
12
G LABEL STATISTICS AND MAPPING RESULTS OF ORIGINAL DATA ACROSS PLATFORMS
We demonstrate the label statistics of original data across platforms and their mapping results into a uniform label space for better joint training as shown in Figures 7 8 and I0 The mapping results are obtained via GPT-4 suggestions and additional human reviews. Note that the "Other" label after mapping indicates the widget information will be deprecated.
H ADDITIONAL INFERENCE EXAMPLES
In Figure show additional qualitative inference examples of Ferret-Ul 2 different platforms. Additionally, in Figure [5 we show an example of Ferret-UI 2 performing multi-step interactions on real-time webpages tollowing GUIDE-style QAs.
19

---

Published as a conference paper at ICLR 2025
Text
43.4%
Converted Label
Checkoox
icon
PageControl
Picture
16.6%
0.6%
SegmentedContral
0.0%%
Slider
3. 3.0%
ToxFled
Text
9.4%
SegmenCortol
TextFiield
11.89
Toggle
Picuree
TebBulia
Figure 7: The statistics of original labels (wedge) and converted labels (legend) on iOS (iPhone iPad) platforms. Each color represents one converted label, and each wedge represents one original label from source data.
38 tie
Converted Label
Image
Button
Checkbox
17.2%
Container
Dialog
Icon
Other
PageContral
Baskgroundtrage
6
Checebox
Picture
7%%
999
Drower
Tab
Advetiiemett
Text
10 70.7
Textfield
13.0%
Too
Toggle
Red Butoo
Figure 8: The statistics of original labels (wedge) and converted labels (legend) on Android plat- form. Each color represents one converted label, and each wedge represents one original label from source data.
20

---

Published as a conference paper at ICLR 2025
8%
Converted Label
Checkbox
Dialog
icon
Other
PageControl
54.7%
Teot
31.75 31.7%
Slider
SegmentedControl
Picure
Text
Toggle
Figure 9: The statistics of original labels (wedge) and converted labels (legend) on AppleTV plat- form. Each color represents one converted label, and each wedge represents one original label from source data.
Ustitm
Ing
Bullon
21.1%
Converted Label
Button
42. 42.3%
Checkbax
Dialog
Picture
Slider
Heeding
Tab
Text
Paragrah
TextField
Taggla
LineBroak
Eection
Figure 10: The statistics of original labels (wedge) and converted labels (legend) on webpage plat- form. Each color represents one converted label, and each wedge represents one original label from source data.
21

---

Published as a conference paper at ICLR 2025
Starch
Beminner
BoxD
Box1
Frone
Boxr
What the romnc and purpose of the the scrozn?
What does the arta (Bool) indicate?
ap "Remindens" [Box0], purpose
The contains
loggle curcntly
in help serrs marage bode and tu-ko lioe.
enahled, allowing the user to snoone the alamn for tew minates when rings.
the ratings and details app?
options
for sotting a asstom
The spp has 4 5-dae raing [Bos basod on 451K
[Box2], &t categuriaed undes Productiviy
The usr can tap on "Sound" option [Bos.] (camenily set to "Hilbide" [Boac]) to access a litt of available sorncs and select a difficene alomm tone.
[Rooc] and designed for users old (Booo))
interactive is prividal oponing app?
How change the time of the abem?
blue buton [Box3] allows luunch .pp
The user can change the alarm time by acmiling thrrugh the hour and minute pickers [[Box3), [Bord] located in the miltle the
Figure 11: Real examples of the Ferret-UI 2 model interacting with iPhone.
Apple
TV
Tum
Gec Rooduag
WALL
purchase
piare beizg lighlighted the raap and Iccarat
To purchaue opon "Ray This you
The baing hightighted Apple Beleevee Square
[Ruol]) which hasted is Bellevue Washingtn [Boo!]]
How toee the morie in another app?
Whst infermmtion provided the sume's noviows?
Top the 'Open. In..* brnoo [Ilos]] to ose other app4
carmenty Upen averaee poctive rating
catings
platforma where you can vipw perchass the movie,
How cmn connect stoe directy phone?
recouutinn movic?
stow, top the "CSil betoon (Baos)) This will ninats phone call to the Apple Belloves Siparn skore on you can spaak store regroentative.
The movie RALL-E ls available 4B resotution. as indeated by the 4K label [Boo2].
Figure 12: Real examples of the Ferret-Ul model interacting with iPad.
22

---

Published as a conference paper at ICLR 2025
BoxO
DAILY TEST i
BoxO
Prodoe
Box2
What the sigrifcance of the text [Bord)?
What does the text [BoaD] suggest?
The text "Duly Ves"" likely retees a cecuring or schasalod ses the user which seggesn the app may provil: daily leaminy lesting
"My EYE CARE on eye care, possibly providing uerns with woul festines munitor, maintain, or impeowe their eye heaild.
suggests this
application is
What dhe furetiorality the "Quiouss" uptins
The "Quisces" option [Box 11 allows users to take tants quizzes, reluted daily mestioned top screen.
What parpose the "Taining Plans" opticu?
The "Triining Plans" option [Boxi] allows the user and mannge traiting listly relaned eye cuze.
progrums
be purpose the "Statiikkss option?
the tarigate to aquat their sertings foo
The "Statirtees option [Rosc] litoly displays the user's progres performseee within sppusuch gaix sccers, time speat loaning, or completkon sales.
The navigate adjus settings by urping on the 'Setinns" uption (Box2), This will likely take them to settings page where they can modify their preferences app crefiguratios.
Figure 13: Real examples of the Ferret-UI 2 model interacting with Android.
Game Center
FNL VINTAGE
Bax2
Box1
Kau Aicaei Duutees
FASHION NEWS LIVE
FNL VINTAGE
What (Bost) will
area [Booo)
top- leaderhord ines indicates gume Teloport provides cocess the Leaderbourdh.
The indicatos that ENL Vintage Traiker provides show
watch
spisstra, you go device. where episodes available viewing.
your
What activity shows Teloport Pro?
How do caplorc other show traikee??
activiy
you started plying
Telopoct ninutes qgo [Boc]],
under
To exploes other tralees, sorill through Show Trailers section [Box]] at the bottums of the screen.
the Receer Activity in Game' section.
How can I ewitch smother pene from this screnn?
Hclp me selus the home
sereen. the page, simply tap Home [Bosz], will take you
To sointt to dicineor game, you cun on the game name activity lis (e.g. Gravity Spices QuizBong ([Box2), [Box]]].
where
you can movigate other conient.
Figure 14: Real examples of the Ferret-Ul 2 model interacting with AppleTV.
23

---

Published as a conference paper at ICLR 2025
Apple Support
if forget apple accomn passeond, Wist should cn this screen?
BoxO
sheeld
"Pregt Apple
Acarut peswond"
butin (Bos0).
Search for more topics
Searc Seppar
If you forgot your Apple Account password
Herl' rese sacowedd and
casees
dewel sip
if Surpet apole recount peasooed,
whst should I nest on this screen?
Civen the previoes actions are
Boxd
1. Tap on the "Korgot Apple Accoen:
paswwedd batien.
Agire nen Accoust You wath same anwilaareess
Itaedd on prrvices actors and the cument screen, you shruld on the "Rewtting your passwerd the web' butten [Boxl]]
Reset your Apple Account password on your iPhone
Gertigg
Sie-i Secarty ChangePannce
Apple Account
fonget spple accous poowwsed,
vhst should do zest cn
screm!
pecsista
T.p "Eooget Apple
Tap
previsus
you
Forgot Password?
Apple Account
forgel spole accourd
passwool,
Reset your password
phons number tha use thyour accoutto
cese Aorptten passoved form underm se acdle
Top the "Renctingg paawore wel' baiton.
Top the "Reset Passward" button.
VHAAS
sesetting your Apple pesowaed, you
Typo migined impur [[]kes [Blos]]],
BoxG
Figure 15: An example of the Ferret-UI model performing multi-step interactions on real-time webpages following GUIDE-style QAs.
24


Reviews:
[{"review_id": "8U0IvTCuwj", "reviewer": "Reviewer_zXTq", "summary": "The authors present a model to recognize widgets on device screens, an evolution of a previously-published (in arxiv) model by Apple. The model performs better than the previous model, and slightly better than GPT-4o.", "questions": "1. What do you think are the scientific contributions of this paper? How much of it is just good engineering of tested ideas?\n2. How is this model going to be made available to researchers and developers?\n3. What would you consider a \"usable\" performance of a model like Ferret-UI?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and fairly easy to understand. It provides a clear comparison with the previous version, what simplifies the reader's task of understanding what is novel.\n\nThe proposed model seems to perform significantly better than the previous one, in spite of greater generality. Its performance is comparable, if not slightly better, than GPT-4o.", "weaknesses": "Since the authors fail to disclose whether the model is going to be available, and how, it is hard to evaluate the impact and relevance for the community of the work described here. \n\nMost importantly, since most of the improvements are engineering-related, the main take-away of the paper is that Ferret-UI, a paper which was never peer-reviewed, can be improved. In other words, there is not much novelty here, beyond knowing it can be done, since the methods used are quite straight-forward.\n\nNotice that the performance of the model is probably still too low to make it useful. It fails elementary and ground tasks at the rate of 1 and 5, that is, in most screens it will be incorrect in relation to at least one widget. This is probably too much for any serious attempt to connect it to an agent, at least in a safe way. In fact, a more interesting and useful benchmark is how successful such models are in actually interacting with a screen, but this is beyond this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FAXk6zdR3V", "reviewer": "Reviewer_EWX6", "summary": "Edit: Updated Review based on Feedback from Associate Program Chairs\n\nThis paper introduces Ferret-UI One, a multimodal large language model designed for universal UI understanding across multiple platforms including iPhone, Android, iPad, Web, and AppleTV. The key innovations are threefold: multi-platform support through unified data collection and processing, high-resolution perception through an adaptive N-gridding mechanism, and advanced task training data generation using GPT-4o with set-of-mark visual prompting. The authors demonstrate the model's effectiveness through comprehensive evaluations on elementary tasks, advanced tasks, and public benchmarks like GUIDE and GUI-World.", "questions": "2. Can you provide detailed examples of failure cases, especially for low-resource platforms like AppleTV?\n3. How does the quality of GPT-4o generated training data vary across platforms, and are there systematic differences in generation quality?\n4. What are the inference time comparisons between platforms for Ferret-UI 1 ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper's technical contributions are well-motivated. The adaptive N-gridding mechanism provides an elegant solution to handling varying resolutions while maintaining bounded computational costs, with clear mathematical formulation in Algorithm 1. The set-of-mark visual prompting approach for GPT-4o data generation represents an acceptable way to improve spatial understanding in training data generation, addressing a key limitation of previous text-only prompting methods.\nThe authors conduct extensive experiments across multiple platforms and task types, providing strong baseline comparisons including GPT-4o and previous state-of-the-art models. The ablation studies examining cross-platform transfer capabilities and architectural components offer valuable insights into the model's behavior. Particularly noteworthy is the model's strong performance even with smaller variants like Gemma-2B, suggesting practical applicability.\nThe paper addresses a genuine need in the field for universal UI understanding across diverse platforms. The demonstrated zero-shot generalization capabilities, especially in the GUI-World benchmark results, indicate robust real-world applicability.", "weaknesses": "A significant concern is the severe data imbalance across platforms, as evident in Table 1. The web platform dominates with 321k images, followed by iPhone with 112k images, while iPad and AppleTV have only 19k and 16k images respectively. Though the authors acknowledge this limitation and mention mitigation strategies like loss weighting and targeted task generation, the effectiveness of these approaches isn't thoroughly analyzed or quantified. To mitigate this, the answer is in balancing the data used across devices. \nThe methodological analysis has several gaps. The paper lacks a detailed examination of failure cases or systematic error analysis. There's limited discussion of computational requirements and inference speed comparisons across different platforms and resolutions. (for model it'll be specific, but in UI design, speed is a major element and latency numbers there would be helpful). Inference speed and overall latency in milliseconds is be a good metric. Additionally, potential biases in GPT-4o generated training data aren't thoroughly explored (although I consider it to be out of scope of this paper), which could impact the model's generalization capabilities. \nThe experimental design could be more rigorous. The cross-platform transfer results presented in Table 4 would benefit from error bars / error spread measures. Wherever appropriate, provide confidence intervals / standard deviations and running the experiment multiple times to ensure reliability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nfyOdqul1t", "reviewer": "Reviewer_u2Px", "summary": "The paper introduces an MLLM-based GUI Agent aimed at enhancing user interface (UI) understanding across a wide range of platforms. These platforms include mobile devices (iPhone, Android), tablets (iPad), web pages, and smart TVs (AppleTV). The paper builds on previous work, Ferret-UI, proving its efficacy on referring, grounding, and advanced user-centric tasks, showing significant improvements over the previous Ferret-UI model and other benchmarks like GUIDE and GUI-World.", "questions": "See Weakness Above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- **Multi-platform Support**: One of the paper's key strengths is the successful integration of multiple platforms (iOS, Android, web, smart TVs), making the model more versatile in various user environments. The authors also discuss the transfer learning effects of each platform.\n- **Advanced Task Training with GPT-4o**: The use of GPT-4o with visual prompting for training data generation significantly enhances the model’s performance, particularly on complex tasks requiring spatial understanding of UI components.\n- **Strong Empirical Results**: The results presented in the paper indicate that Ferret-UI One outperforms its predecessor, Ferret-UI, across several metrics, including grounding, referring, and advanced task handling, particularly on the GUIDE and GUI-World benchmarks.\n- **Zero-shot Transfer**: The model demonstrates robust zero-shot performance, indicating that it generalizes well across different platforms without overfitting to training data, a key advantage for building scalable UI systems.", "weaknesses": "- **Extra Probing for Models Potential**: This paper mainly follows the pipeline of GUI-world, proposing a multi-platform GUI agent by collecting the dataset first and annotating it with SOTA MLLM GPT-4o. Although its performance is SOTA in GUI-World and GUIDE benchmark, the whole paper is not that exciting to me.  I think your model's experiment results are expected to be superior to Ferret-UI because Ferret-UI One is trained on more high-quality instruction tuning and task-specific data. The authors are suggested to probe more potential capability of their models such as the grounding plugin for general MLLM-based GUI Agents in benchmarks such as OS-world or VisualAgentBench. \n- **Evaluation and Analysis of Synthetic Dataset**: The authors are suggested to evaluate whether GPT-4o can perform the annotation task well by providing some quantitative results. Moreover, a comprehensive comparison to previous GUI datasets and analysis of your dataset is needed.\n- **Case Study**: Show more case studies in the appendix and provide failure cases of Ferret-UI One in GUIDE and GUI-World in grounding and navigation tasks for a more comprehensive review of your proposed model, as well as an analysis of why your model fails. A more detailed discussion of future direction is also suggested on topics of building more general agents across various platforms or how to improve the grounding or navigating capabilities of current models or agents.\n- **Additional Experiments**:  I think authors can include more comparative baselines to show the strength of your synthetic data generation pipeline. The authors can consider SeeClick and SeeClick-V on your proposed benchmark, two strong baseline models in REC, grounding, and navigation tasks because these models are all recently released and show high potential in GUI grounding tasks. Another option is to evaluate your model in popular GUI grounding benchmark ScreenSpot proposed in SeeClick.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)