OmniParser for Pure Vision Based GUI Agent

## Abstract

Abstract The recent success of large vision language models shows great potential in driving the agent system operating on user interfaces. However, we argue that the power multimodal models like GPT-4V as a general agent on multiple operating systems across different applications is largely underestimated due to the lack of a robust screen parsing technique capable of: 1) reliably identifying interactable icons within the user interface, and 2) understanding the semantics of various elements in a screenshot and accurately associate the intended action with the corresponding region on the screen. To fill these gaps, we introduce OmniParser , a comprehensive method for parsing user interface screenshots into structured elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface.
We first curated an interactable icon detection dataset using popular webpages and an icon description dataset. These datasets were utilized to fine-tune specialized models: a detection model to parse interactable regions on the screen and a caption model to extract the functional semantics of the detected elements. OmniParser significantly improves GPT-4V’s performance on ScreenSpot benchmark. And on Mind2Web and AITW benchmark, OmniParser with screenshot only input outperforms the GPT-4V baselines requiring additional information outside of screenshot.

## 1 Introduction

Large language models have shown great success in their understanding and reasoning capabilities. More recent works have explored the use of large vision-language models (VLMs) as agents to perform complex tasks on the user interface (UI) with the aim of completing tedious tasks to replace human efforts [, , , , , , , , ]. Despite the promising results, there remains a significant gap between current state-of-the-arts and creating widely usable agents that can work across multiple platforms, e.g. Windows/MacOS, IOS/Android and multiple applications (Web broswer Office365, PhotoShop, Adobe), with most previous work focusing on limiting applications or platforms.

While large multimodal models like GPT-4V and other models trained on UI data [, , ] have demonstrated abilities to understand basic elements of the UI screenshot, action grounding remains one of the key challenges in converting the actions predicted by LLMs to the actual actions on screen in terms of keyboard/mouse movement or API call []. It has been noted that GPT-4V is unable to produce the exact x-y coordinate of the button location, Set-of-Mark prompting [] proposes to overlay a group of bounding boxes each with unique numeric IDs on to the original image, as a visual prompt sent to the GPT-4V model. By applying set-of-marks prompting, GPT-4V is able to ground the action into a specific bounding box which has ground truth location instead of a specific xy coordinate value, which greatly improves the robustness of the action grounding []. However, the current solutions using SoM relies on parsed HTML information to extract bounding boxes for actionable elements such as buttons, which limits its usage to web browsing tasks. We aim to build a general approach that works on a variety of platforms and applications.

In this work, we argue that previous pure vision-based screen parsing techniques are not satisfactory, which lead to significant underestimation of GPT-4V model’s understanding capabilities. And a reliable vision-based screen parsing method that works well on general user interface is a key to improve the robustness of the agentic workflow on various operating systems and applications. We present OmniParser, a general screen parsing tool to extract information from UI screenshot into structured bounding box and labels which enhances GPT-4V’s performance in action prediction in a variety of user tasks.

We list our contributions as follows:

- •
We curate a interactable region detection dataset using bounding boxes extracted from DOM tree of popular webpages.
- •
We propose OmniParser, a pure vision-based user interface screen parsing method that combines multiple finetuned models for better screen understanding and easier grounded action generation.
- •
We evaluate our approach on ScreenSpot, Mind2Web and AITW benchmark, and demonstrated a significant improvement from the original GPT-4V baseline without requiring additional input other than screenshot.

## 2 Related Works

### 2.1 UI Screen Understanding

There has been a line of modeling works focusing on detailed understanding of UI screens, such as Screen2Words [], UI-BERT [], WidgetCaptioning [], Action-BERT []. These works demonstrated effective usage of multimodal models for extracting semantics of user screen. But these models rely on additional information such as view hierarchy, or are trained for visual question answering tasks or screen summary tasks.

There are also a couple publicly available dataset that on UI screen understanding. Most notably the Rico dataset [], which contains more than 66k unique UI screens and its view hierarchies. Later [] auguments Rico by providing 500k human annotations on the original 66k RICO screens identifying various icons based on their shapes and semantics, and associations between selected general UI elements (like icons, form fields, radio buttons, text inputs) and their text labels. Same on mobile platform, PixelHelp [] provides a dataset that contains UI elements of screen spanning across 88 common tasks. In the same paper they also released RicoSCA which is a cleaned version of Rico. For the web and general OS domain, there are several works such Mind2Web [], MiniWob++[], Visual-WebArena [, ], and OS-World [] that provide simulated environment, but does not provide dataset explicitly for general screen understanding tasks such as interactable icon detection on real world websites.

To address the absence of a large-scale, general web UI understanding dataset, and to keep pace with the rapid evolution of UI design, we curated an icon detection dataset using the DOM information from popular URLs avaialbe on the Web. This dataset features the up-to-date design of icons and buttons, with their bounding boxes retrieved from the DOM tree, providing ground truth locations.

### 2.2 Autonomous GUI Agent

Recently there has been a lot of works on designing autonomous GUI agent to perform tasks in place of human users. One line of work is to train an end-to-end model to directly predict the next action, representative works include: Pixel2Act [], WebGUM[] in web domain, Ferret [], CogAgent [], and Fuyu [] in Mobile domain. Another line of works involve leveraging existing multimodal models such as GPT-4V to perform user tasks. Representative works include MindAct agent [], SeeAct agent [] in web domain and agents in [, , ] for mobile domain.
These work often leverages the DOM information in web browser, or the view hierarchies in mobile apps to get the ground truth position of interactable elements of the screen, and use Set-Of-Marks[] to overlay the bounding boxes on top of the screenshot then feed into the vision-language models. However, ground truth information of interactable elements may not always be available when the goal is to build a general agent for cross-platforms and cross-applications tasks. Therefore, we focus on providing a systematic approach for getting structured elements from general user screens.

## 3 Methods

A complex task can usually be broken down into several steps of actions. Each step requires the model’s (e.g. GPT-4V) ability to: 1) understand the UI screen in the current step, i.e. analyzing what is the screen content overall, what are the functions of detected icons that are labeled with numeric ID, and 2) predict what is the next action on the current screen that is likely to help complete the whole task. Instead of trying to accomplish the two goals in one call, we found it beneficial to extract some of the information such as semantics in the screen parsing stage, to alleviate the burden of GPT-4V so that it can leverages more information from the parsed screen and focus more on the action prediction.

Hence we propose OmniParser, which integrates the outputs from a finetuned interactable icon detection model, a finetuned icon description model, and an OCR module. This combination produces a structured, DOM-like representation of the UI and a screenshot overlaid with bounding boxes for potential interactable elements. We discuss each component of the OmniParser in more details for the rest of the section.

### 3.1 Interactable Region Detection

Identifying interactable regions from the UI screen is a crucial step to reason about what actions should be performed given a user tasks. Instead of directly prompting GPT-4V to predict the xy coordinate value of the screen that it should operate on, we follow previous works to use the Set-of-Marks approach [] to overlay bounding boxes of interactable icons on top of UI screenshot, and ask GPT-4V to generate the bounding box ID to perform action on. However, different from [, ] which uses the ground truth button location retrieved from DOM tree in web browswer, and [] which uses labeled bounding boxes in the AITW dataset [], we finetune a detection model to extract interactable icons/buttons.

Specifically, we curate a dataset of interactable icon detection dataset, containing 67k unique screenshot images, each labeled with bounding boxes of interactable icons derived from DOM tree. We first took a 100k uniform sample of popular publicly availabe urls on the web [], and collect bounding boxes of interactable regions of the webpage from the DOM tree of each urls. Some examples of the webpage and the interactable regions are shown in 2.

Apart from interactable region detection, we also have a OCR module to extract bounding boxes of texts. Then we merge the bounding boxes from OCR detection module and icon detection module while removing the boxes that have high overlap (we use over 90% as a threshold). For every bounding box, we label it with a unique ID next to it using a simple algorithm to minimizing the overlap between numeric labels and other bounding boxes.

### 3.2 Incorporating Local Semantics of Functionality

We found in a lot of cases where only inputting the UI screenshot overlayed with bounding boxes and associated IDs can be misleading to GPT-4V. We argue the limitation stems from GPT-4V’s constrained ability to simultaneously perform the composite tasks of identifying each icon’s semantic information and predicting the next action on a specific icon box. This has also been observed by several other works [, ]. To address this issue, we incorporate the local semantics of functionality into the prompt, i.e. for each icons detected by the interactable region detection model, we use a finetuned model to generate description of functionality to the icons, and for each text boxes, we use the detected texts and its label.

We perform more detailed analysis for this topic in section 4.1. To the best of our knowledge, there is no public model that is specifically trained for up-to-date UI icon description, and is suitable for our purpose to provide fast and accurate local semantics for the UI screenshot. Therefore we curate a dataset of  7k icon-description pairs using GPT-4o, and finetune a BLIP-v2 model [] on this dataset. Details of dataset and training can be found in Appendix 7.1. After finetuning, we found the model is much more reliable in its description to common app icons. Examples can be seen in figure 4. And in figure 3, we show it is helpful to incorporate the semantics of local bounding boxes in the form of text prompt along with the UI screenshot visual prompt.

Figure: Figure 1: Examples of parsed screenshot image and local semantics by OmniParser. The inputs to OmniParse are user task and UI screenshot, from which it will produce: 1) parsed screenshot image with bounding boxes and numeric IDs overlayed, and 2) local semantics contains both text extracted and icon description.
Refer to caption: extracted/5767271/flow_merged_0.png

Figure: Figure 2: Examples from the Interactable Region Detection dataset. The bounding boxes are based on the interactable region extracted from the DOM tree of the webpage.
Refer to caption: extracted/5767271/100k_2000.png

## 4 Experiments and Results

We conduct experiments on several benchmarks to demonstrate the effectiveness of OmniParser. We start by a motivating experiments showing that current GPT-4V model with set of mark prompting [] is prone to incorrectly assigning label ID to the referred bounding boxes. Then we evaluate on Seeclick benchmark and Mind2Web to further showcase OmniParser with local semantics can improve the GPT-4V’s performance on real user tasks on different platforms and applications.

### 4.1 Evaluation on SeeAssign Task

To test the ability of correctly predicting the label ID given the description of the bounding boxes for GPT-4v models, We handcrafted a dataset SeeAssign that contains 112 tasks consisting of samples from 3 different platforms: Mobile, Desktop and Web Browser. Each task includes a concise task description and a screenshot image. The task descriptions are manually created and we make sure each task refers to one of the detected bounding boxes, e.g. ’click on ’settings”, ’click on the minimize button’. During evaluation, GPT-4V is prompted to predict the bounding box ID associated to it. Detailed prompt are specified in Appendix.
The task screenshot images are sampled from the ScreenSpot [] benchmark, where they are labeled with set of marks using OmniParser. The tasks are further divided into 3 sub-categories by difficulty: easy (less than 10 bounding boxes), medium (10-40 bounding boxes) and hard (more than 40 bounding boxes).

From table 1, we see that GPT-4V often mistakenly assign the numeric ID to the table especially when there are a lot of bounding boxes over the screen. And by adding local semantics including texts within the boxes and short descriptions of the detected icons, GPT-4V’s ability of correctly assigning the icon improves from 0.705 to 0.938.

From figure 3, we see that without the description of the referred icon in the task, GPT-4V often fails to link the icon required in the task and the ground truth icon ID in the SoM labeled screenshot, which leads to hallucination in the response. With fine-grain local semantics added in the text prompt, it makes it much easier for GPT-4V to find the correct icon ID for the referred icon.

Figure: Figure 3: Examples from the SeeAssign evaluation. We can see that fine-grain local semantics improves the GPT-4V’s ability to assign correct labels to the referred icon.
Refer to caption: extracted/5767271/seeassign_eg.png

**Table 1: Comparison of GPT-4V with and without local semantics**
|  | Easy | Medium | Hard | Overall |
| --- | --- | --- | --- | --- |
| GPT-4V w.o. local semantics | 0.913 | 0.692 | 0.620 | 0.705 |
| GPT-4V w. local semantics | 1.00 | 0.949 | 0.900 | 0.938 |

### 4.2 Evaluation on ScreenSpot

ScreenSpot dataset [] is a benchmark dataset that includes over 600 interface
screenshots from mobile (iOS, Android), desktop
(macOS, Windows), and web platforms. The task instructions are manually created so that each instruction corresponds to an actionable
elements on the UI screen. We first evaluate the performance of omniparser using the this benchmark. In table 2, we can see across the 3 different platforms: Mobile, Desktop and Web, omniparser significantly improves the GPT-4V baseline. Noticeably, omniparser’s performance even surpasses models that are specifically finetuned on GUI dataset including SeeClick, CogAgent and Fuyu by a large margin.
We also note that incorporating the local semantics (omniparser w. LS in the table) further improves the overall performance. This corroborates with the finds in section 4.1 that incorporating local semantics of the UI screenshot in text format, i.e. adding OCR text and descriptions of the icon bounding boxes further helps GPT-4V to accurately identify the correct element to operate on.
Furthermore, our findings indicate that the interactable region detection (ID) model we finetuned improves overall accuracy by an additional 4.3% compared to using the raw Grounding DINO model. This underscores the importance of accurately detecting interactable elements for the success of UI tasks. Overall, the results demonstrate that the UI screen understanding capability of GPT-4V is significantly underestimated and can be greatly enhanced with more accurate interactable elements detection and the incorporation of functional local semantics.

**Table 2: Comparison of different approaches on ScreenSpot Benchmark. LS is short for local semantics of functionality, GD is short for Grounding DINO, and ID is short for the interactable region detection model we finetune.**
| Methods | Model Size | Mobile | Desktop | Web | Average |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Text | Icon/Widget | Text | Icon/Widget | Text | Icon/Widget |  |
| Fuyu | 8B | 41.0% | 1.3% | 33.0% | 3.6% | 33.9% | 4.4% | 19.5% |
| CogAgent | 18B | 67.0% | 24.0% | 74.2% | 20.0% | 70.4% | 28.6% | 47.4% |
| SeeClick | 9.6B | 78.0% | 52.0% | 72.2% | 30.0% | 55.7% | 32.5% | 53.4% |
| MiniGPT-v2 | 7B | 8.4% | 6.6% | 6.2% | 2.9% | 6.5% | 3.4% | 5.7% |
| Qwen-VL | 9.6B | 9.5% | 4.8% | 5.7% | 5.0% | 3.5% | 2.4% | 5.2% |
| GPT-4V | - | 22.6% | 24.5% | 20.2% | 11.8% | 9.2% | 8.8% | 16.2% |
| OmniParser (w.o. LS, w. GD) | - | 92.7% | 49.4% | 64.9% | 26.3% | 77.3% | 39.7% | 58.38% |
| OmniParser (w. LS + GD) | - | 94.8% | 53.7% | 89.3% | 44.9% | 83.0% | 45.1% | 68.7% |
| OmniParser (w. LS + ID) | - | 93.9% | 57.0% | 91.3% | 63.6% | 81.3 | 51.0% | 73.0% |

### 4.3 Evaluation on Mind2Web

In order to test how OmniParser is helpful to the web navigation secnario, We evaluate on [] benchmark. There are 3 different categories of task in the test set: Cross-Domain, Cross-Website, and Cross-Tasks.
We used a cleaned version of Mind2Web tests set processed from the raw HTML dump which eliminates a small number of samples that has incorrect bounding boxes. In total we have 867, 167, 242 tasks in the test set from Cross-Domain, Cross-Website, and Cross-Tasks category respectively. During evaluation, we feed both the parsed screen results and the action history as text prompt, and SOM labeled screenshot to GPT-4V similar to the prompting strategy in [, ]. Following the original paper, we perform offline evaluation focusing on the element accuracy, Operation F1 and step success rate averaged across the task.

In the first section of the table (row 1-3), We report numbers from a set of open source VL models as it appears in [, ]. Here CogAgent and Qwen-VL are not finetuned on the Mind2Web training set. More detailed information about model settings can be found in the Appendix7.4.

In the second section of the table (row 4-9) we report numbers from Mind2web paper [] and SeeAct []
paper. In this section, all of the approaches use the HTML elements selected by a finetuned element proposal model on Mind2Web training set which produces top 50 relevant elements on the HTML page based on the user task. Additionally, GPT-4V+SOM and GPT-4V+textual choices corresponds to the SeeAct with image annotation, and textual choices grounding methods respectively. In GPT-4V+SOM, the set of mark (SOM) boxes are selected from the element proposal model, and are labeled with the ground truth location extracted from HTML. In contrast, GPT-4V+textual uses DOM information of the selected relevant elements directly in the text prompt, rather than overlaying bounding boxes on top of screenshot. The better performance of textual choice corroborates with the experiment results in 4.1.

In the last section (row 10-11), we report numbers from OmniParser. We observe GPT-4V augumented with local semantics of icon functionality and the finetuned interactable region detection model (w. LS + ID) performs better than the model with raw grounding DINO model (w. LS + GD) in all of the categories.

Further, without using parsed HTML information, OmniParser is able to outperform GPT-4’s performance that uses HTML in every sub-category by a large margin, suggesting the substantial benefit of the screen parsing results provided by OmniParser.
Additionally, OmniParser outperforms the GPT-4V+SOM by a large margin. Compared to GPT-4V+textual choices, OmniParser significantly outperforms in Cross-Website and Cross-Domain category (+4.1% and +5.2%), while underperforming (-0.8%) slightly in the Cross-Task category, which indicates that OmniParser provides higher quality information compared ground truth element information from DOM and top-k relevant elemnt proposal used by the GPT-4V+textual choices set-up, and make the GPT-4V easier to make a accurate action prediction.
Lastly, OmniParser with GPT-4V significantly outperform all the other trained models using only UI screenshot such as SeeClick and Qwen-VL.

**Table 3: Comparison of different methods across various categories on Mind2Web benchmark.**
| Methods | Input Types | Cross-Website | Cross-Domain | Cross-Task |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | HTML free | image | Ele.Acc | Op.F1 | Step SR | Ele.Acc | Op.F1 | Step SR | Ele.Acc | Op.F1 | Step SR |
| CogAgent | ✓ | ✓ | 18.4 | 42.2 | 13.4 | 20.6 | 42.0 | 15.5 | 22.4 | 53.0 | 17.6 |
| Qwen-VL | ✓ | ✓ | 13.2 | 83.5 | 9.2 | 14.1 | 84.3 | 12.0 | 14.1 | 84.3 | 12.0 |
| SeeClick | ✓ | ✓ | 21.4 | 80.6 | 16.4 | 23.2 | 84.8 | 20.8 | 28.3 | 87.0 | 25.5 |
| MindAct (gen) | × | × | 13.9 | 44.7 | 11.0 | 14.2 | 44.7 | 11.9 | 14.2 | 44.7 | 11.9 |
| MindAct | × | × | 42.0 | 65.2 | 38.9 | 42.1 | 66.5 | 39.6 | 42.1 | 66.5 | 39.6 |
| GPT-3.5-Turbo | × | × | 19.3 | 48.8 | 16.2 | 21.6 | 52.8 | 18.6 | 21.6 | 52.8 | 18.6 |
| GPT-4 | × | × | 35.8 | 51.1 | 30.1 | 37.1 | 46.5 | 26.4 | 41.6 | 60.6 | 36.2 |
| GPT-4V+som | × | ✓ | - | - | 32.7 | - | - | 23.7 | - | - | 20.3 |
| GPT-4V+textual choice | × | ✓ | 38.0 | 67.8 | 32.4 | 42.4 | 69.3 | 36.8 | 46.4 | 73.4 | 40.2 |
| OmniParser (w. LS + GD) | ✓ | ✓ | 41.5 | 83.2 | 36.1 | 44.9 | 80.6 | 36.8 | 42.3 | 86.7 | 38.7 |
| OmniParser (w. LS + ID) | ✓ | ✓ | 41.0 | 84.8 | 36.5 | 45.5 | 85.7 | 42.0 | 42.4 | 87.6 | 39.4 |

### 4.4 Evaluation on AITW

In additional to evaluation on multi-step web browsing tasks, we assess OmniParser on the mobile navigating benchmark AITW [], which contains 30k instructions and 715k trajectories. We use the same train/test split as in [] based on instructions, which retain only one trajectory for each instructions and no intersection between train and test. For a fair comparison, we only use their test split for evaluation and discard the train set as our method does not require finetuing.

In table 4, we report the GPT-4V baseline in [] paper, which corresponds to the best performing set up (GPT-4V ZS+history) that uses UI elements detected by IconNet [] through set-of-marks prompting [] for each screenshot at every step of the evaluation. The detected UI elements consist of either OCR-detected text or an icon class label, which is one of the 96 possible icon types identified by IconNet. Additionally, action history is also incorporated at each step’s prompt as well. We used the exact same prompt format in [] except the results from the IconNet model is replaced with the output of the finetuned interactable region detection (ID) model.
Interestingly, we found that the ID model can generalize well to mobile screen. By replacing the IconNet with the interactable region detection (ID) model we finetuned on the collected webpages, and incorporating local semantics of icon functionality (LS), we find OmniParser delivers significantly improved performance across most sub-categories, and a 4.7% increase in the overall score compared to the best performing GPT-4V + history baseline.

**Table 4: Comparison of different methods across various tasks and overall performance in AITW benchmark.**
| Methods | Modality | General | Install | GoogleApps | Single | WebShopping | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ChatGPT-CoT | Text | 5.9 | 4.4 | 10.5 | 9.4 | 8.4 | 7.7 |
| PaLM2-CoT | Text | - | - | - | - | - | 39.6 |
| GPT-4V image-only | Image | 41.7 | 42.6 | 49.8 | 72.8 | 45.7 | 50.5 |
| GPT-4V + history | Image | 43.0 | 46.1 | 49.2 | 78.3 | 48.2 | 53.0 |
| OmniParser (w. LS + ID) | Image | 48.3 | 57.8 | 51.6 | 77.4 | 52.9 | 57.7 |

## 5 Discussions

In this section, we discuss a couple of common failure cases of OmniParser with examples and potential approach to improve.

Repeated Icons/Texts From analysis of the the GPT-4V’s response log, we found that GPT-4V often fails to make the correct prediction when the results of the OmniParser contains multiple repeated icons/texts, which will lead to failure if the user task requires clicking on one of the buttons. This is illustrated by the figure 7 (Left) in the Appendix. A potential solution to this is to add finer grain descriptions to the repeated elements in the UI screenshot, so that the GPT-4V is aware of the existence of repeated elements and take it into account when predicting next action.

Corase Prediction of Bounding Boxes One common failure case of OmniParser is that it fails to detect the bounding boxes with correct granularity. In figure 7 (Right), the task is to click on the text ’MORE’. The OCR module of OmniParser detects text bounding box 8 which encompass the desired text. But since it uses center of the box as predicted click point, it falls outside of the ground truth bounding box. This is essentially due to the fact that the OCR module we use does not have a notion of which text region are hyperlink and clickable. Hence we plan to train a model that combines OCR and interactable region detection into one module so that it can better detect the clickable text/hyperlinks.

Icon Misinterpretation We found that in some cases the icon with similar shape can have different meanings depending on the UI screenshot. For example, in figure 8, the task is to find button related to ’More information’, where the ground truth is to click the three dots icon in the upper right part of the screenshot. OmniParser successfully detects all the relevant bounding boxes, but the icon description model interpret it as: "a loading or buffering indicator". We think this is due to the fact that the icon description model is only able to see each icon cropped from image, while not able to see the whole picture during both training and inference. So without knowing the full context of the image, a symbol of three dots can indeed mean loading buffer in other scenarios. A potential fix to this is to train an icon description model that is aware of the full context of the image.

## 6 Conclusion

In this report, We propose OmniParser, a general vision only approach that parse UI screenshots into structured elements. OmniParser encompasses two finetuned models: an icon detection model and a functional description models. To train them, we curated an interactable region detection dataset using popular webpages, and an icon functional description dataset. We demonstrate that with the parsed results, the performance of GPT-4V is greatly improved on ScreenSpot benchmarks. It achieves better performance compared to GPT-4V agent that uses HTML extracted information on Mind2Web, and outperforms GPT-4V augmented with specialized Android icon detection model on AITW benchmark. We hope OmniParser can serve as a general and easy-to-use tool that has the capability to parse general user screen across both PC and mobile platforms without any dependency on extra information such as HTML and view hierarchy in Android.

## Acknowledgement

We would like to thank Corby Rosset and authors of ClueWeb22 for providing the seed urls for which we use to collect data to finetune the interactable region detection model. The data collection pipeline adapted AutoGen’s multimodal websurfer code for extracting interatable elements in DOM, for which we thank Adam Fourney. We also thank Dillon DuPont for providing the processed version of mind2web benchmark.

## 7 Appendix

### 7.1 Details of Icon-Description Dataset

In figure 4, we see that the original BLIP-2 model tend to focus on describing shapes and colors of app icons, while struggling to recognize the semantics of the icon. This motivates us to finetune this model on an icon description dataset. For the dataset, we use the result of parsed icon bounding boxes inferenced by the interactable icon detection model on the ScreenSpot dataset since it contains screenshots on both mobile and PC. For the description, we ask GPT-4o whether the object presented in the parsed bounding box is an app icon. If GPT-4o decides the image is an icon, it outputs one-sentence description of the icon about the potential functionality. And if not, GPT-4o will output ’this is not an icon’, while still including this in the dataset. In the end, we collected 7185 icon-description pairs for finetuning.

We finetune BLIP-2 model for 1 epoch on the generated dataset with constant learning rate of $1e^{-5}$, no weight decay and Adam optimizer. We show a few of the qualitative examples of finetuned model vs the original model in figure 4.

Figure: Figure 4: Example comparisons of icon description model using BLIP-2 (Left) and its finetuned version (Right). Original BLIP-2 model tend to focus on describing shapes and colors of app icons. After finetuning on the functionality semantics dataset, the model is able to show understanding of semantics of some common app icons.
Refer to caption: extracted/5767271/finetune_icon.png

### 7.2 Training details of Interactable Icon Region Detection Model

As introduced in 3.1, we train a YOLOv8 model on the interactable icon region detection dataset. We collect in total of 66990 samples where we split 95% (63641) for training, and 5% (3349) for validation. We train for 20 epochs with batch size of 256, learning rate of $1e^{-3}$, and the Adam optimizer on 4 GPUs. We show the training curve in figure 5.

Figure: Figure 5: Training curves of interactable icon region detection model.
Refer to caption: extracted/5767271/yolo_train_curve.png

### 7.3 Details of SeeAssign Evaluation

#### 7.3.1 Prompt Used for GPT-4V

GPT-4V without local semantics:

GPT-4V with local semantics:

### 7.4 Details of Mind2Web Evaluation

Here we list more details of each baseline in table 3.
SeeClick, QWen-VL SeeClick is a finetuned version of Qwen-VL on the Mind2Web training set and we report both of their numbers in their paper.
CogAgent CogAgent number is taken from the SEEAct paper [], where they report cogagent-chat-hf checkpoint that is not fine-tuned on Mind2Web for experiments.
MindAct(Gen), MindAct, GPT-3.5-Turbo, GPT-4 The numbers for these baseline are taken from the Mind2Web [] paper, where they use HTML information to augument the corresponding web agent.
GPT-4V+som This model corresponds to the image annotation grounding method in SeeAct paper, where the som boxes extracted from the selelcted HTML elements are provided to GPT-4V to make action prediction.
GPT-4V+textual choice This corresponds to the best performing scenario in SeeAct paper (except the Oracle), that uses the selected HTML elments information in a multi-choice question format as input to the GPT-4V agent.

#### 7.4.1 Qualitative Examples

We list a few more examples to demonstrate local semantics of icon function description helps GPT-4V make better action prediction in figure 6.

Figure: Figure 6: More examples of local semantics of icon functionality help with GPT-4V in grounding actions
Refer to caption: extracted/5767271/seeassign_eg2.png

Figure: Figure 7: Analysis of failure cases. All the bounding boxes are labeled by which relies only on the screenshot. Left: There are in total 7 similar enable button for 7 different alarm times in the parsed screenshot. And the correct Icon ID corresponding to alarm 7:30 is 27. GPT-4V fails to make the correct prediction. Right: The ground truth region to click is the text ’MORE’ inside bounding box 8. We can see that the OCR fails to detect the text ’MORE’ in bold, and only detects the bounding box 8, which encompasses ’MORE’. Since the predicts the click point as the center of the box, so it the predicted click point falls outside of the ground truth region, which leads to failure in this task.
Refer to caption: extracted/5767271/fail_case1.png

Figure: Figure 8: Analysis of failure cases. The task is to find button related to ’More information’, and the ground truth is to click the three dots icon in the upper right part of the screenshot. The the icon functional description model does not take into account the context of this page and interpret it as: "a loading or buffering indicator" which causes the failure.
Refer to caption: extracted/5767271/fail_case3.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: - What is the OCR module used in the Omniparser?

- Since each icon requires a description generated by BLIP-2, could this contribute to increased model latency?

- What are the performance levels of the separate icon detection and description modules, and which one currently presents a bottleneck? Additionally, it seems that no validation set is included for the icon description at this stage. It is quite hard to control when to stop training. 

- Since GPT-4V can perform icon description directly, would it be feasible to test its performance on description tasks using GPT-4V?
soundness: 3
strengths: - The paper proposes an alternative method for parsing GUI elements in screenshots by using an icon detection model combined with a description model to generate comprehensive information about the elements.
- The collected dataset on icon detection and description is expected to be highly beneficial for the community.
- The experiments are rich and thorough. And the benchmark performance shows promising results.
summary: This paper introduces OMNIPARSER, a method designed to enhance the action-generating capabilities of multimodal models like GPT-4V when interacting with user interfaces. Specifically, the paper collects an interactable icon detection dataset with 67,000 unique screenshot images and DOM annotations. Then, the YOLO-v8 model is fine-tuned to detect icons within a screenshot. Additionally, an icon description dataset with 7,185 icon descriptions is built, and BLIP v2 is fine-tuned on it to output icon functionalities. Consequently, given a screenshot, OMNIPARSER can detect UI elements through the proposed icon detection, caption, and OCR models. It can further incorporate GPT-4V to generate GUI actions. Experiments on Mind2Web and AITW demonstrate promising improvements over the baselines.
weaknesses: 1. Since the paper mainly uses existing YOLO-v8 and BLIP v2 models, its primary contribution lies in the proposed icon-detection and description datasets. However, many details regarding dataset construction and data statistics are missing.

   - The authors mention that the main focus is on collecting an "INTERACTABLE REGION DETECTION" dataset. However, how are elements deemed interactive? The DOM does not directly provide metadata indicating interactivity, only element types. How was this implemented, and does this process have a risk of misclassification?
   
   - For the icon description dataset, the authors used GPT-4o for labeling. However, GPT-4o itself has limitations in icon recognition. What mechanisms did the authors employ to ensure data quality, and what is the accuracy of the generated descriptions?
   
   - The authors mention annotating 7,185 icon-description data points. Are all these icons unique? Could the authors provide data distribution examples, including icon types?

2. The authors mention merging DOM and OCR results by calculating overlap and merging bounding boxes with high overlap (>90%). For web applications, does this approach introduce substantial redundancy? For instance, in the bottom-left image of Figure 2, the bounding box for "Contact Us" is likely derived from the DOM and should generate a larger box, while OCR might produce a tighter one with lower overlap. More examples demonstrating the effectiveness of the merging strategy would be helpful.

3. Some comparisons appear unfair. For instance, using OCR predictions for grounding in ScreenSpot will likely outperform existing MLLMs, but this does not necessarily highlight the model’s contribution. Similarly, in results on other benchmarks, it is difficult to determine whether OCR results or icon detection contribute more. Testing GPT-4V + OCR to demonstrate any performance boost would be beneficial.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. If GPT-4V can locate objects without relying on SOM in the future, would this pipeline still be effective?

2. Would better results be achieved by training an end-to-end model directly with constructed data, rather than relying on additional fine-tuned models?
soundness: 3
strengths: 1. The authors conducted comprehensive evaluations on multiple benchmark datasets, with OmniParser achieving state-of-the-art results across all datasets.

2. OmniParser does not rely on additional information like DOM or view hierarchy, making it more generalizable.

3. The authors performed extensive ablation studies validating the effectiveness of the proposed ID and IS modules, and verified the robustness of the entire framework on open-source Llama-3.2-V and Phi-3.5-V models.

4. The authors released their code and models, making it easy for researchers to reproduce the results.
summary: The paper introduces OMNIPARSER, a universal UI interface parsing methodology that addresses the challenges of reliably identifying interactive elements in user interfaces. By developing a dataset based on popular webpage DOM trees for interactive area detection and an icon-description dataset, the approach trains task-specific models to understand the semantics of elements within screenshots and accurately map intended actions to corresponding screen regions. OMNIPARSER, through its integration of multiple fine-tuned models, achieves enhanced screen understanding capabilities and demonstrates significant improvements across benchmarks including ScreenSpot, Mind2Web, and AITW, validating its seamless compatibility with existing vision-language models.
weaknesses: 1. OmniParser's strong performance heavily relies on the powerful backbone model (GPT4-V), and switching to open-source models would significantly decrease its performance.

2. The entire pipeline is not end-to-end, which increases its complexity and inference latency.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. typos: l198 - l199: ‘click on ‘settings”, ‘click on the minimize button’ → “click on settings”, “click on the minimize button”
2. Related questions from the weakness part.
3. Are the authors going to release the following? (1) the icon detection and description dataset, (2) the trained models, (3) code for data collection.
soundness: 3
strengths: 1. This paper is well written and easy to follow. The demonstration of the GUI framework is clear.
2. The proposed strategy, which involves extracting the position of interactable elements along with their function descriptions, is both flexible and effective. It significantly enhances GPT-4V’s agentic capabilities, particularly the GUI grounding capabilities of the entire framework.
3. The data curation considers the position and function description of interactable elements, which would be beneficial to the research community if it were to be open-sourced.
summary: This paper introduces OmniParser, a vision-based GUI agent designed to enhance large vision-language models (such as GPT-4V) by effectively parsing user interface (UI) screenshots. OmniParser comprises several key components: an OCR module for extracting text elements, an icon detection model for identifying interactable icons, a captioning model to describe the functions of detected elements, and a generalist large vision-language model for interpreting the parsed information to make reasoning and action decisions. The authors curated datasets to train the icon detection and captioning models, fine-tuning them to improve their performance. They tested OmniParser on benchmarks such as ScreenSpot, Mind2Web, and AITW, demonstrating that the proposed framework significantly enhances the capabilities of vision-based agents over baseline models by enabling them to process screenshots without relying on textual inputs.
weaknesses: 1. The novelty and innovation in this work are limited. The primary approach involves training existing modules for various purposes, such as icon detection and captioning, and then integrating these modules to construct a GUI framework. This does not meet the criteria for ICLR.

2. Missing details.
    
    - 2.1 The authors collected data for interactable region detection from web pages. However, in Figure 2, most examples contain text elements, with very few icons shown. If most of the interactable elements from web pages are texts, how does the model generalize to icons from other domains, such as mobile and desktop interfaces? Additionally, what are the categories of interactable elements and their distributions in the training data?

    - 2.2. The authors curate the icon description from the ScreenSpot dataset, since the data scale of ScreenSpot is not large, how do the authors guarantee the model’s generalization to other datasets, benchmarks or real-world applications?

     - 2.3 Evaluation from the cost perspective: Since OmniParser combines different modules for OCR, element detection, element description, and action decision, the inference time could be longer when compared to end-to-end frameworks, such as SeeClick[1], or more advanced VLMs (e.g., Qwen2-VL) trained with SeeClick data. Additionally, OmniParser utilizes GPT-4V in its framework, but the associated cost in USD is not included in the analysis.

[1] Cheng K, Sun Q, Chu Y, et al. Seeclick: Harnessing gui grounding for advanced visual gui agents[J]. arXiv preprint arXiv:2401.10935, 2024.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 3
questions: Small issue -- Typo:
line 837 -- QWen -> Qwen

- Instead of applying both BLIP-2 or YOLO, how about train an open-world detector that let GPT4 to determine which element we should ground and output a text query, then use this query for object grounding? 

- would the model, training and benchmark data will be open-source?
soundness: 3
strengths: 1. This work focuses on a critical problem in UI vision understanding: recognizing elements solely from visual data, i.e., screenshots, rather than relying on DOM data.

2. The proposed strategy is flexible and can be applied across multiple planning frameworks.

3. The grounding data takes into account the functionality of the icon and has a high-level understanding.

4. The experiment has an online setting on Window Agent Arena, which is helpful.
summary: The paper introduces Ominiparser, a modular method designed to enhance large vision-language models e.g., GPT-4V, specifically for parsing user interface (UI) screenshots and grounding abilities to screen elements effectively. 
Ominiparser addresses current limitations in screen parsing, especially the inability of existing models to detect interactable regions and associate actions with specific areas in a UI. 
To achieve this, the authors:
1. curate a dataset for interactable icon detection and icon description from DOM tree; 
2. subsequently using them to fine-tune models including a captioner BLIP-2 and a detector YOLO, served as a combination solution.
3. Ominiparser is tested on benchmarks like ScreenSpot, Mind2Web, and AITW, which connects with an advanced planner, show substantial improvements over baseline models (e.g., GPT-4V), enabling process screenshots alone without relying on additional HTML or view hierarchy data.
weaknesses: 1. While this work provides great practical contributions, the main focus is on its novelty and innovation; it mainly involves fine-tuning existing models using new data sources and applying these models through ensemble methods. The existing innovations do not meet the criteria for ICLR submissions, but are suitable for submission in technical reports or industry workshops.

2. There is limited analysis and discussion regarding UI visual understanding across platforms or data sources. For instance, while the community has substantial OCR data related to UI, icon-specific data remains scarce. The paper does not include distribution details on the proposed data, such as the number or sources of icons and software applications represented.

3. The authors do not provide analysis on the impact of prompt length when combining data from different sources (e.g., GPT-4, GPT-4V, with or without additional tool-generated information). Additionally, there is no breakdown of the financial cost or computation time associated with using GPT-4V alongside Omniparser to complete individual tasks.

4. The paper does not propose an innovative strategy for handling UI visual understanding with models like BLIP-2 or YOLO, particularly given that UI screenshots are often large (e.g., 2K resolution). Efficiency strategies could enhance processing in such high-resolution contexts.

Meta-review: This paper is borderline with two reviewers slightly positive, and two reviewers on the negative side. Overall, the reviewers generally agree that the paper is well-written and the proposed method shows promising results on various benchmarks. However, they raise concerns about the novelty and complexity of the approach. More specifically, the reviewers liked strong empirical results that Omniparser shows on multiple benchmarks. Omniparser seems also be able to generalize to work with different LVLMs and across various platforms. The reviewers also liked the dataset contribution. On the other hand, the reviewers are concerned with the limited novelty of the work and feel the work doesn't meet the bar of ICLR. The pipeline is also complex, which seems an integration of existing models. 

Based on the reviews and discussion, the paper appears to fall short of the acceptance threshold. The lack of fundamental technical novelty and the complexity of the pipeline outweigh the empirical results and dataset contribution. Addressing these concerns can significantly improve the contribution of the work.

Decision: Reject