VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
Authors: Junpeng Liu
Yifan Song
Bill Yuchen Lin
Wai Lam
Graham Neubig
Yuanzhi Li
Xiang Yue
Venue: COLM (2024)
Keywords: multimodal large language model, evaluation, web understanding, grounding
TL;DR: VisualWebBench: How Far Have LMMs Evolved in Web Page Understanding and Grounding?

Abstract: Multimodal Large Language models (MLLMs) have shown promise in web-related tasks, but evaluating their performance in the web domain remains a challenge due to the lack of comprehensive benchmarks. Existing benchmarks are either designed for general multimodal tasks, failing to capture the unique characteristics of web pages, or focus on end-to-end web agent tasks, unable to measure fine-grained abilities such as OCR, understanding, and grounding. In this paper, we introduce VisualWebBench, a multimodal benchmark designed to assess the capabilities of MLLMs across a variety of web tasks. VisualWebBench consists of seven tasks, and comprises 1.5K human-curated instances from 139 real websites,  covering 87 sub-domains. We evaluate 16 open-source MLLMs, Gemini Pro, Claude-3 series, and GPT-4V(ision) on VisualWebBench, revealing significant challenges and performance gaps. Further analysis highlights the limitations of current MLLMs, including inadequate grounding in text-rich environments and subpar performance with low-resolution image inputs. We believe VisualWebBench will serve as a valuable resource for the research community and contribute to the creation of more powerful and versatile MLLMs for web-related applications.

Published as a conference paper at COLM 2024
VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
Junpeng Liu Yifan Song Bill Yuchen Lin
Wai Lam Graham Neubig
Yuanzhi Li Xiang Yue
Carnegie Mellon University
The Chinese University of Hong Kong
School of Computer Science, Peking University
MBZUAI Allen Institute for Al
https://visualwebbenndd github..ia io/
Abstract
Multimodal Large Language models (MLLMs) have shown promise in web-related tasks, but evaluating their performance in the web domain remains challenge due the lack of comprehensive benchmarks. Existing benchmarks are either designed for general multimodal tasks, failing to capture the unique characteristics of web pages, or focus on end-to-end web agent tasks, unable to measure fine-grained abilities such as OCR, un- derstanding, and grounding. this paper, we introduce Visual Hisualeeeeenhh a multimodal benchmark designed to assess the capabilities of MILMs across a variety of web tasks. VisualWebBench consists of seven tasks, and com- prises 1.5K human-curated instances from 139 real websites, covering 87 sub-domains. We evaluate 16 open-source MLLMs, Gemini Pro, Claude-3 series, and GPT-4V(ision) on VisualWebBench, revealing significant chal- lenges and performance gaps. Further analysis highlights the limitations current MLLMs, including inadequate grounding in text-rich environments and subpar performance with low-resolution image inputs. We believe Vi Wibhenh will serve valuable resource for the research commu- nity and contribute to the creation of more powerful and versatile MLLMs for web-related applications.
Introduction
The web is an indispensable platform for information exchange and interaction, presenting unique challenges and opportunities for multimodal learning. While web content has been primary source of training data for multimodal large language models (MLLMs) (OpenAI, 2023; Google et al., 2023; Liu et al., 2023a), largely overlooked aspect is understanding of websites themselves. Every website is designed be visually rendered for consumption by human users, with structured layouts, rich textual information, and diverse interactive elements. Enabling MILMs to accurately comprehend websites would unlock numerous applications in the web domain.
However, evaluating the performance of multimodal models in the web domain chal- lenging task. Unlike object- or scene-centric images in most existing benchmarks (Young et al., 2014; Goyal et al., 2017; Lin et al., 2014; Singh et al., 2019; Li et al., 2023b; Liu et al., 2023b; Yu et al., 2023; Yue et al., 2023), web pages present a complex interplay of visual and textual information, along with interactive elements, requiring models to possess rigorous understanding abilities over hierarchical structures and contextual relationships. More- over, web elements are often small, numerous, and scattered across the page, demanding
'Equal contribution.
*Corresponding jpliuëling inlik cuhk. edu.hk xyue2Bandrew. cmu. edu.

---

Published as a conference paper at COLM 2024
Heading
Description Ceneration
WebQA
Visual Web Bench
Quaotion: Tell me the baswding bod of this scremnhorr of wehppagh
Question What the metn description this wibsitu?
plotferm mentioned for following the websitt's content? Answer YoueTube Cherrel
Questin: What additional
Webcht-wriee Task
Answer Disppver, Apprecions & Undersard the Animal World!
Answer. The world's langest most trusted
blement-witee Taok
callection of animal facts puctures and chard
Astion-sriar
Element OCR
Rew Abouk
Tears
Grounding
Question Tell me the Sent content in the red.
Discover, Appreciate, & Understand the Animal World!
Question: Thave labeled bright IDe for some HTMI elkmarts in this welsity scrorddurt Tell me which one shouldd complete instnactions the day
bocrdding hox
Answwer We beliece that people know about the wookd's creatures they will peyt you
Len
Seuch cord balon
AEAAANAAA A-3
Action Prediction
Grounding
Question Thove labeled
to
momen will bocker
Qustioo dicking
bright IDe for wome HTMI.
Laborde foe thaeed Sie sereeee gouinn ba
clemants in this willitu scrocrsbur, Tell me which one dlmment
Discover Your Favorite Animal Today!
the web page?
desrrpptiu
(A) Arumal
comespandingg to descriptions
Soarch
(C) The 7 bust
(B) All unimmll A-Z
Antmals
(D) Search any arimale
Answer C
Figure 1: VisualWebBench contains seven QA-style tasks, covering website, element, action- level understanding, reasoning, and grounding capabilities.
fine-grained recognition and accurate spatial reasoning and grounding. The vast diversity of website designs, layouts, and content across different domains further complicates the creation of representative and robust evaluation benchmarks, necessitating the inclusion of a wide range of website categories ensure the generalizability of the evaluated models.
Despite the growing importance of the web domain in multimodal learning, existing bench- marks fall short of comprchensively evaluating the fundamental capabilities of models in this context. General MLLM benchmarks (Young et al., 2014; Liu et al., 2023b; Yue et al., 2023), do not adequately capture the unique characteristics of the web domain. On the other hand, web-agent benchmarks, like WebShop (Yao et al., 2022), Mind2Web (Deng et al., 2024), and (Visual)WebArrena (Zhou et al., 2023; Koh et al., 2024), focus on end-to-end abilities without offering fine-grained assessment of essential skills such OCR, semantic understanding, and grounding. Measuring these fine-grained abilities is crucial, as they serve building blocks for complex web-related tasks, enable targeted improvements, and provide clearer picture of a model's performance. The lack of granularity in existing benchmarks hinders the development of more capable multimodal models for the web domain, emphasizing the need for a comprehensive evaluation benchmark.
To address these limitations, we introduce Vi VisualWebBench a comprehensive multimodal benchmark designed to assess the capabilities of MLLMs in the web domain. Inspired by the human interaction process with web browsers, Webbeehh consists of seven tasks that map to core abilities required for web tasks: meta description generation, webpage QA, heading OCR, element OCR, element grounding, action prediction, and action grounding, as detailed in Figure 1. The benchmark comprises 1.5K instances, all uniformly formulated in the QA style, making it easy to evaluate and compare the performance of different MLLMs.
We evaluate 16 open-source MLLMs, Gemini Pro (Google et al., 2023), Claude Sonnet, Claude Opus (Anthropic, 2024), and GPT-4V(ision) (OpenAI, 2023) on VisualWebBench; our key findings are follows:
VisualWebBench presents significant challenges for current MLLMs, with GPT-4V and Claude Sonnet achieving average scores of 64.6 and 65.8, respectively, indicating substan- tial room for improvement.
2

---

Published as a conference paper at COLM 2024
Understanding
Reasoning
Visual
VQA, GOA
MMMU
Reasoning
Q What the
Q:
Webpage Scoeenshtts
Action Prediction the best deseription the new page af click this button?
doing A: walking
this imago
A 3.5
Understanding
Meta Description Generation: meta desctition? WebQA What the muximum amoonnd I Lomn save?
OCR
Grounding
OCR
Heading OCR: Tell me heoding text
ReCoCC,, GRIT Givee the coordinate of the child. (:66602246, 10,2400
CUTERO, Total-Text Receggnize the test this image. AVILON
Grounding
Element Gounding: elemant best matichas the deseription?
Celebration Honel
Action Gruudding Which element should click to complete the instruction?
Figure 2: Comparison between WisualWebBench (right) and other multimodal benchmarks (left).
A notable performance gap exists between open-source MLLMs and proprietary counter- parts such as GPT-4V and Claude series, with the leading open-source model, LLaVA-1.6- 34B, achieving an average score of 50.5.
MLLMs' abilities in general domains, such as general reasoning on MMMU (Yue et al., 2023), and web agent tasks, such as Mind2Web (Deng et al., 2024), do not correlate much with their performance on WebBenc, highlighting the importance of web-specific benchmarks like WebBeehh
The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent.
Grounding ability, a crucial skill for developing MLLM-based web applications like autonomous web agents, is weakness for most MLLMs.
In summary, VisualWebBench offers standardized benchmark for evaluating MLLMs in web understanding, enabling the development of more capable and efficient models, autonomous web agents, and web-related applications.
2
Related Work
Before detailing VisualWebBench, we briefly outline its differences with existing MLLM benchmarks, also outlined in Figure
2.1 MLLM Benchmarks
In concert with improvements these MLLMs, benchmarks have also evolved. These range from traditional single task benchmarks like VQA (Antol et al., 2015; Goyal et al., 2017), RefCOCO (Mao et al., 2016), and Flickr30K (Young et al., 2014), to more holistic evaluation benchmarks like LAMM (Yin et al., 2024), MMBench (Liu et al., 2023b), and MMMU (Yue et al., 2023), recently. this work, we focus on images in web-based scenarios characterized by structured layouts, copious textual data, and diverse interactive elements, which pose new challenges for current MLLMs. The most closely related scenario to this work is GUI- based tasks, exemplified by Screen2Words (Wang et al., 2021), Widget Captioning (Li al., 2020), and WebSRC (Chen et al., 2021) which is a web-based VQA dataset. Different from offers comprehensive evaluation for MLLMs, spanning
previous works, perception, comprehension, grounding, and reasoning capabilities.
2.2 Web Agent Benchmarks
As a vital aspect of daily life, methods that perform various tasks in web scenarios have garnered widespread attention from researchers. Earlier efforts introduce simplified sim-
3

---

Published as a conference paper at COLM 2024
MTSC
Task
Level
Capability
Metric
#Num.
Scienn
Anteal ())
Meta Deseription Generation
Understanding
ROUGE-L
134
Tirkoos 02
WebQA
Webpage
Understanding
F1
314
Heading OCR
OCR
ROUGE-L
46
Govermmnnt
Bloch 29
(P)
Home
(1)
Element Grounding
Element OCR
Element
Grounding
OCR
Accuracy
ROUGE-L
245
Thael
Action Prediction
Reasoning
Accuracy
281
Inghheeiing
Liberyle
Action Grounding
Action
Grounding
Accuracy
Total
1534
Figure 3: Overview of Vi WebbBehc.. Left: Domain distribution. The numbers represent the count of sub-domains within each domain. Right: Tasks in VisualWebBench.
ulated environments for web navigation tasks, such as MiniWob++ (Liu et al., 2018) and WebShop (Yao et al., 2022). Recently, Mind2Web (Deng et al., 2024), WebArena (Zhou et al., 2023), VisualWeb (Koh et al., 2024) construct realistic and reproducible web environments to facilitate the development of web agents. There are also various studies to improve the web understanding or grounding capabilities of MLLMs (Gao et al., 2024; Kil et al., 2024) or develop agents for autonomous web navigation (Hong al., 2023; Zheng et al., 2024; Cheng et al., 2024; He et al., 2024). Despite their success, the community still lacks a comprehensive evaluation of MLLMs' basic performance in web scenarios, including perception, understanding, grounding, and reasoning.
The VisualWebBench Benchmark
3.1 Overview of VisualWebBench
We present VisualWebBench: a multimodal benchmark designed thoroughly evaluate the understanding and grounding capabilities of MLLMs in web scenarios. The proposed Wibbenhe possesses the following features: 1) Comprehensiveness: VisualWebBench spans 139 websites with 1.5K samples, encompassing 12 different domains (e.g., travel, sports, hobby, lifestyle, animals, science, etc.) and 87 sub-domains. 2) Multi-granularity VisualWebBence assesses MLLMs at three levels: website-level, element-level, and action- level. 3) Multi-tasks: VisualWebBench encompasses seven tasks designed to evaluate the understanding, OCR, grounding, and reasoning capabilities of MLL.Ss. 4) High quality: Quality ensured through careful human verification and curation efforts. The domain distribution and statistics of benchmark are presented in Figure 3.
3.2
Website Selection
To ensure comprehensive coverage across diverse domains and top-ranking websites, our website selection process conducted based on SimilarWeb We start from 12 top-level domains in SimilarWeb such as Science, Engineering, Sports, Lifestyle, and more, which are subsequently broken down into 87 sub-domains. Then we manually select representative websites from the top-5 most ranking websites in each sub-domain. Our selection criteria prioritize websites with rich interactive elements, including images and buttons, while excluding those that have been used in prior web agent benchmarks like Mind2Web and WebArena. We use Playwright? to render and save the websites automatically.
hhtps:/wwww.sinilaavv..com
4

---

Published as a conference paper at COLM 2024
3.3 Task Construction
This section details the proposed seven tasks of VisualWebBench and the process of con- structing data for each task; examples are shown in Figure 1.
Meta Description Generation. This task aims to evaluate the MLLMs' ability to com- prehend and summarize the content of webpage screenshot. The meta description, i.e., <meta name="desct name="descrrptiion tags in the head section of HTML, is a brief snippet of text that helps humans or search engines understand the content of websites. However, the quality of extracted meta descriptions cannot be ensured and their styles are pretty different on diverse websites. For example, some meta descriptions only consist of list of keywords or short title of the website, instead of natural language description. Hence, we instruct GPT-4V to generate a better meta description, given both the screenshot and the extracted meta description. The final descriptions are verified and curated by the authors.
WebQA. To assess the understanding capabilities of MLLMs in the web scenario, DisualWebBence involves a webpage QA task, where the MLLM will answer open questions that demand a thorough comprehension of the visual layout. Human annotators are in- structed to examine each screenshot and craft up to five challenging questions which satisfy: 1) A degree of reasoning ability is required to answer the question, 2) The answers should be precise and objective.
Heading OCR. This task requires MLLMs to locate and recognize the text of the heading of a website. Different from the traditional OCR task where a target element is given, as shown in Figure 1, the input of heading OCR is simply a raw screenshot, and the expected output is the heading content. The ground-trute target is automatically extracted from the first <h1> element in the HTML
Element OCR. This task evaluates the capability of MLLMs to conduct OCR on lengthy texts. Firstly, we traverse the HTML DOM tree and extract the bounding boxes and textual description of each element Then, we select elements whose text descriptions comprise over 20 words. The task input consists of screenshot with a bounding box indicating the position of the element to be recognized.
Element Grounding. Grounding, or Referring Expression Comprehension (REC), is crucial image-text alignment capability, particularly for MILMs interacting with web environments, Given description of an HTML element, MLLMM needs to locate the corresponding region in the screenshot. However, our preliminary studies reveal that current MI.LMs struggle to directly give the coordinate of the target's bounding box (see 4.6). Inspired by Yang et al. (2023), we adopt simplified setting where eight candidate bounding boxes are presented. Differently, the candidate elements here are extracted automatically using Playwright, with each assigned an alphanumeric ID. MLLMs are then prompted to select the box that best matches the given element description. The element description, golden bounding box, and negative bounding boxes of randomly chosen elements are automatically extracted from the webpage.
Action Prediction. This task asks MLLMs to predict the title of the redirected website after clicking an element, in a multi-choice QA way. In terms of the construction process, firstly, we employ Playwright to click all clickable elements within the web page and save the <title> or <meta ta name="title" tag as titles of new redirected web pages. Subsequently, we randomly sample seven additional elements distinct from the target element and take the titles of their respective redirect destinations as negative choices. Cases where a click does not lead to title change are omitted from consideration. The task presents input in the form of screenshots highlighting the clickable target with a red bounding box. Accompanying each screenshot is eight choices, each labeled with letter. The ground truth output is the letter corresponding to the correct answer.
Action Grounding. In addition to directly grounding elements from their descriptions, we further introduce the action grounding task. In this task, the MLLMs are given a human instruction, such as "search for the hotels in NYC", and are prompted to determine the correct element to click to fulfill the instruction. Similar with Element Grounding, MLLMs take in screenshot containing bounding boxes of eight candidate elements and select the
5

---

Published as a conference paper at COLM 2024
Website
Element
Action
Model
Average
MetaGen
WebQA
HeadOCR
OCR
Ground
Prediction
Ground
General MLLMs
Otter
5.3
0.7
3.5
0.5
0.7
14.6
0.0
3.6
InstructBLIP-13B
11.6
5.2
7.6
6.0
114
11.4
17.5
10.1
BLIP-2
11.0
5.2
20.6
2.6
15.5
14.9
8.7
11.2
Fuyu-8B
3.5
5.2
5.8
12.4
19.4
13.2
10.7
Yi-VI-66
8.0
14.3
43.8
3.5
16.2
13.9
13.6
16.2
LLaVA-1.5-BB
15.3
13.2
41.0
5.7
12.1
17.8
13.6
17.0
mPLUG-OWl2
12.7
19.9
51.6
7.2
11.9
23.1
3.9
18.6
LLaVA-1,5-13B
20.0
16.2
41.1
11.8
15.0
22.8
8.7
19.4
SPHINX
13.7
11.6
48.1
7.7
18.4
14.2
22.3
19.4
Qwen-VL
21.8
32.2
48.4
13.4
14.0
26.7
10.7
23.9
CogVLM
16.6
30.6
65.9
10.0
17.7
11.7
25.1
VILA-13B
12.7
28.8
67.9
12.6
36.3
27.3
DeepSeek-VL-7B
18.1
30.0
63.4
18.1
16.2
35.2
15.5
28.1
LLaVA-1667B
27.0
39.8
57.3
54.8
31.7
30.6
10.7
36.0
LLaVA-1.6-13B
26.5
44.5
52.8
56.1
31.7
48.4
15.5
39.4
HaVA--16-34B
24.3
48.2
67.1
71.9
43.1
74.0
25.2
50.5
Gemini 1.0 Pro
25.0
55.5
75.1
65.4
44.3
26.7
43.7
48.0
Gemini 1.5 Pro
31.6
69.0
54.5
76.6
70.0
74.4
77.7
64.8
Claude Sonnet
28.9
81.8
70.3
89.2
68.8
63.4
58.3
65.8
Claude Opus
26.7
75.4
63.7
87.1
57.7
60.4
58.5
GPT-4V(ision)
34.5
75.0
68.8
62.8
67.5
67.6
64.6
GUT Agent MLLMs
SeeClick CogAgent-Chat 16.3
0.0
19.6
34.8
0.0
9.9
1.8
1.9
9.7
53.3
20.2
32.4
41.6
13.5
23.3
28.7
Table 1: Overall results of different models VisualWebBench benchmark. The best- performing model in-bold, and the second best underlined. The maximum the metrics is 100.
most appropriate one. The task data is completed by seven experienced annotators, and an annotation tool is developed to streamline the annotation workflow. Further details about the annotation tool and the annotation process can be found in Appendix C.
All tasks above adopt a VQA-style formulation similar to customary multimodal bench- marks. All screenshots in WibuelWebBench are unified in standard 1280-pixel width. All samples of our benchmark undergo careful verification and curation through collaborative effort and a division tasks by two authors. See Appendix for more details.
3.4 Evaluation Metrics
We adopt different evaluation metrics for different tasks in VisualWebBench For open-ended generation tasks, ROUGE-L (Lin, 2004) is used to measure the quality of the generated responses. For the WebQA task, SQuAD style Fl (Rajpurkar et al., 2016) is employed the evaluation metric. For multiple-choice tasks, we measure accuracy.
4 Experiments
4.1 Evaluated MLLMs
We evaluate open-source general MI.LMs on VisualWebBench (See Appendix D for model details). By default, for each model family, use the largest available checkpoint. We consider three scales LLaVA, 7B, 13B, and 34B, for model scaling analysis. Several strong close-source MLLMs, Gemini Pro (Google et al., 2023), Claude series (Anthropic, 2024), and GPT-4V(ision) (OpenAI, 2023), are also included for evaluation.
Recent studies have introduced several MLLMs tailored to create agents for GUI tasks, such as web and smartphones (Cheng et al., 2024; Hong et al., 2023; Gao et al., 2024). Therefore, consider two open-source GUI-speciaiized MILMMs for evaluation: SeeClick (Cheng

---

Published as a conference paper at COLM 2024
Open Source MLLMS Proprietary ML
Open Source MILMs
Chande GPTT-V
Propristary MLLMs
CUI Agent MLLMs
Cemini 1.5 Pro
Mind2Web Score
MMMU Scure
20-
SeeClick
CogAgent
Centri 10Pro
VL. U Deepleck VL
W--L-68
LLVA 138
16-0
LaVA-1678
Quen-VV
30
mFLUG-OWis
CugVEM
aLL 150
LE
0
10
20
30
40
50
10
40
50
60
VisualWebBench Score
VisualWebBench Soore
Figure 4: Scores of MLLMs on
Figure 5:
Scores of MLLMs on WusuleeeBench and Mind2Web.
VisualWebBench and MMMU.
et al., 2024) is developed by GUI grounding pretraining based on Qwen-VL (Bai et al., 2023). CogAgent (Hong et al., 2023) is built upon CogVLM (Wang et al., 2023), focusing on GUT interpretation and navigation, with support for high-resolution image inputs.
4.2 Main Results
In this section, we present a comprehensive comparison of different MLLMs on VisualWebBench in Table 1. From the results, we highlight the following findings.
Challenging Nature of Web Tasks: Even the powerful GPT-4V achieves an average score of only 64.6 on VisualWebBench, leaving ample room for improvement For tasks requiring strong reasoning and grounding abilities (Action Prediction and Action Grounding), many MLLMs struggle
e to surpass random chance (12.5). This underscores that current models cannot effectively handle many tasks within the web scenario.
Disparity between Open-source and Proprietary MLLMs: GPT-4V, Gemini 1.5 Pro and Claude outperform open-source MLLMs including GUI agent MLLMs by a large margin, highlighting discernible gap in the capabilities of current open-source MLLMs compared to proprietary ones like GPT-4V. Meanwhile, LLaVA-1.6-34B achieves a commendable result (50.5) and beats all other open-source MLLMs, even outperforming the performance of Gemini 1.0 Pro (48.0). Notably, we find Claude Sonnet surpasses Opus on all tasks in WisualWebBench, suggesting that Sonnet may possess more powerful capabilities in web scenarios.
Scaling Leads Better Performance: Compared with the 7B and 13B versions of LLa.6,, the 34B model achieves a performance boost across almost all tasks, reaching an average score of 50.5. Although there are factors other than scale, such as different backbone LLMs, this indicates that increasing model size is a promising avenue for enhancing the capabilities of open-source MLLMs in web-related tasks.
General MLLMs vs. GUI Agent MLLMs: SeeClick and CogAgent are two MLLMs pre- trained on GUT grounding tasks. However, we observe that these GUI agent MILMs do not exhibit significant performance improvement. For example, SeeClick fails to outperform Qwen-VL, its base MLLM, across all tasks. Notably, we find these models suffer catastrophic forgetting (Wang et al., 2024) general instruction following capability after training GUI grounding data. These results underscore the necessity for more general GUI-specific training techniques to enhance the MLLMs' performance in the web scenario. To further investigate the effectiveness of GUI grounding training, we perform a comprehensive comparison of various grounding settings in Section 4.6.
7

---

Published as a conference paper at COLM 2024
4.3 Correlations with General Scenario and Agent Benchmarks
We delve into the relationship between the performance of MLLMs in the web scenario and that in general and agent scenarios. Specifically, we use MMMU (Yue et al., 2023) as the proxy of MILMs' capability in general scenario and Mind2Web (Deng et al., 2024) is used for the evaluation of the agent scenario.
While Figure 4 somewhat suggests some correlation between VisualWebBench score and MMMU score, the relationship does not appear to be significant. In other words, performing well in the general domain does not necessarily guarantee the same trend in the web scenario. For example, while Yi-VL-6B and BLIP2 outperform CogVLM on MMMU, they fall short in achieving a good score on VisualWebBench. It is also noteworthy that LLaVA-1.6-000 performs well on both tasks, nearly matching the performance level of GPT-4V.
As illustrated in Figure 5, generally, VisualWebBench scores are higher than those of Mind2Web+, which demonstrates that there still exists large room for improvement of agent ability empowered by webpage understanding, grounding abilities, as well other abilities like planning. GUI agent MLLMs tend to exhibit overfitting in terms of agent capability, resulting underperformance in understanding web pages.
Moreover, in Figure 6, we conduct an in-depth corre- lation analysis between MMMU and VisualWebBench seven subtasks, as well as similar analysis be- tween Mind2Web and VisualWebBench tasks. From the results in Table 1, we have seven sets of metrics (columns), and Pearson correlation coefficients are calculated between every two columns. For MMMU, the correlations are generally strong. Specifically, the two subtasks requiring heavy reasoning, WebQA and Action Prediction, strongly correlate with MMMU. For Mind2Web, the correlation between scores on Visual and those on Mind2Web is low, even exhibiting two negative correla- tions in Meta Description Generation and Action Grounding. These findings suggest that VisualWebBench offers a different evaluation perspective for MLLMs in the web scenario.
0.81
088 0.75 0.78
MirdZWWbb
-003
0.58
0.15
0.4 0.27 Eleoc
0.2 04
-0.06
MattCGen
HeetOck
EleCround
AntPred
AnGround Average
Figure 6: Correlations between VisualWebBench and Mind2Web, respectively.
MMMU,
4.4 Correlation Between VisualWebBench Tasks
Figure 7 illustrates the correlations between tasks in VisualWebBench. This analysis reveals strong corre- lation among specific tasks, namely Meta Description Generation, WebQA, and Element OCR, all demand- ing a comprehensive understanding of textual con- tent within webpages. contrast, Action Prediction and Action Grounding tasks exhibit minimal corre- lation, implying distinct skill sets necessary for pre- dicting action outcomes versus pinpointing elements for actions. Moreover, Action Grounding seems to be less correlated with all other non-grounding tasks, highlighting its distinctive and specialized skill re- quirements.
MetaGen-
100
0:78
WebQA
088
0.63
HeadCCR
073
0:74
1:00
0.70
0:62
0:1
EleOCR
0.89
0:90
0:70
1.00
0.89
0:78
0.76
EleGround
075
1.00
ActPred
0.77
0.62
ActGround
0.73 MetaCan
066
063
0.76
0.64
643
1,00
QA
Head ElOOC
EleCround
AcPred
AcGGruund
Figure 7: Correlations between subtasks in DisualWebBendh
4.5 Analysis of Image Resolution
Most current MILMs can only process low-resolution images, typically x 448. However, the screenshots in WisuahhebBench are captured high resolution (1280 pixels in width), presenting challenges in identifying intricate details at lower reso-
The overall score on the validation set of MMMU is used for comparison.
4 Detailed experimental results are included in Appendix F.
8

---

Published as a conference paper at COLM 2024
aisual Score
Proprietary MLLMs
Spen-Source MILMs
GPI-V
1.5 Pro
WebuelWeBBench Score
50
Cladde Opus
OLLAVA-1A:4B
Gemini 1.0
40
@LLVA
30
CogAgent
VL.
@oogLLM
Dee
20
C
20
LLaVA-16-12
10
66UJP
LLAVA-I6-BB
224
448
672
1024
2048
3072
224
336
448
672
Input Image Resolution
Input Image Resolution
(a)
(b)
Figure 8: The effect of image resolution on VisualWebBench score.
lutions. In this section, we explore the effects of input resolution on model performance. We plot the relation of max input image resolution and VisualWebBende scores for different MLLMs in Figure 8(a). Notably, MLLMs with higher input resolution generally achieve higher scores. For instance, DeepSeek-VI. with 1024x 1024 resolution achieves a higher score than Qwen-VL with x48x 448 resolution.
Based on LLaVA-1.6 series models, we further conduct a formal ablation study on input image resolution. As depicted in Figure 8(b), a significant performance improvement is observed as input image resolutions increase for all three model sizes. Additionally, the models exhibit greater benefits when increasing resolution from 336 to 448, compared with from 448 to 672. This finding suggests that, for LLaVA-1.6, a resolution of 448x 448 stands as the minimal requirement to achieve adequate performance in web-related tasks.
4.6 Analysis of Grounding Capability
In our experiments in Section 4.2, for Element and Action Grounding tasks, we provide eight candidate elements and use multiple-choice setting to evaluate different MLLMs. However, in many applications, the screenshots of webpages cannot annotated with candidate bounding boxes. Hence, evaluate the grounding capability in unannotated images by framing the grounding tasks as a Referring Expression Comprehension (REC) problem, where the MLLMs must generate the position (bounding box [x [x V1, X2, V2 or central point coordinate y)) of the selected HTML element. For the setting of the bounding box, we follow the standard REC task and use Al's0 (Lin et al., 2014) as the metric. For the setting of point prediction, a predicted point is regarded as correct if it falls into the true bounding box.
As Table 2 shows, GUI agent MLLMs significantly outperform general MLLMs (e.g., LLaVA- 1.6 and GPT-4V) in generating the positions (Bbox or point) of target elements, confirming the efficacy of grounding pre-training through the point or bounding box prediction. For other MLLMs that have been trained on general grounding data like RefCOCO, they still fail to accurately give the coordinates of the correct elements.
4.7 Case Studies
We show few case studies for LLaVA-16-34B, CogAgent, and GPT-4V on action prediction and action grounding tasks. For the action prediction task (Figure 9), CogAgent generates wrong choice without any explanation, while LLaVA locates wrong element ("Go" button). Notably, GPT-4V shows reasonable thinking process and the correct answer. For action grounding (Figure 10), despite LLaVA and GPT-4V generating reasonable thought processes, all three models fail to answer correctly. See Appendix F for more case studies.

---

Published as a conference paper at COLM 2024
Element Ground
Action Ground
Model
Multi-choice
Bbox
Point
Multi-choice
Bbox
Point
Fuyu-8B
19.4
0.0
0
15.5
0.0
0.0
VILA-13B
16.5
1.0
7.8
16.5
0.0
5.9
L.AVA-1.6-7B
31.7
0.2
4.6
10.7
0.0
5.9
LLaVA-1.6-13B
31.7
0.0
0.7
15.5
1.0
5.9
LLaVA-1.6-44B
43.1
1.7
10.7
25.2
3.0
10.9
Qwen-VL
14.0
1.5
3.9
10.7
0.0
3.0
(is-4V(sson)
67.5
0.2
1.5
75.7
0.0
1.0
SeeClick
9.9
0.0
70.0
1.9
0.0
42.6
CogAgent-Chat
41.6
29.3
46.3
23.3
36.6
58.4
Table 2: Three evaluation settings for grounding tasks. "Multi-Choice" is the default setting in VisualWebBench "Bbox" and "Point" denote the setting of predicting the coordinate of the target bounding box and central point, respectively.
Action Prediction
Grounding
Internet Archive Center Archive: highlight elemann?? Archare NEWS Ardiive Search Engine Ardiive:
Which
Instruction: View the new artidess in Astrophysiics
Cullectur's Legay
Astropapsios would look fox the link
buthon that reloted lateet or new anticise
LLaVA-340
within the Astroppyysiiia section Answac
Thought: The red rectangle
button the Advrnssd Soarch"
bounding box locased the "GO"
Juiton condinates scree and perfocrs dizkk operation...
"houught More the cunurr to the 'Heeeet"
CogAgant
Seatume Answeet
GPT-4V
Thought
vaave new
Thoughht bounding appears buttonn which likely Adviasdd given posttton page.
Achophyyins
alod nost "Jattppossio (adto- ph news nevent, awarchy" which riggeets leads to ocre utticies in the Zutoophhyiia calogies. Annwve:
Figure 9: Case study of Action Prediction.
Figure 10: Case study of Action Grounding.
Conclusion
In this work, we introduce VisualWebBench: a comprehensive benchmark to evaluate the web page understanding and grounding capabilities of MLLMs. VisualWebBench encom- passes seven tasks spanning three different levels covering web page, element, and user action. Unlike existing benchmarks, our benchmark aims comprehensively evaluate MLLMs in web contexts, including understanding, OCR, grounding, and reasoning. Our evaluation 14 open-source MILMss, Gemini Pro, Claude Sonnet, Claude Opus, and GPT-4Vissoon shows the substantial challenges posed by realistic web tasks. Further anal- ysis highlights several limitations of current MLLMs, including inadequate grounding in text-rich environments and subpar performance with low-resolution image inputs. We believe VisualWebBende will serve as a catalyst for further exploration in the development of MLLMs towards artificial general intelligence.
Acknowledgement
The authors would thank Boyuan Zheng, Shuyan Zhou, Yizhong Wang, and Jie Huang for their insightful discussions and comments. The authors would also thank seven annotators for their help in annotating the action grounding task samples. Xiang Yue is supported by Carnegie Bosch Fellowship. Junpeng Liu and Wai Lam are supported by grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14200719).
10

---

Published as a conference paper at COLM 2024
References
Anthropic. The claude model family: Opus, sonnet, haiku, 2024. URL https://api.
semanticscholar, www.ar.rgrrooooe hrggCorpusID::220 2688332499.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. 2015 IEEE International Conference
Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 2425- 2433. IEEE Computer Society, 2015. doi: 10.110//CCVV20152277, URL https:/ //os: org/ 1errg/10.110 1111//1CIIV.... 201CV.2215..2 2015.279.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. ArXio preprint, 2tts//230.12200 2023. URL https: /
abs//308. 12966.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşılla. Introducing our multimodal models, 2023. URL https: //ww.adept ai hal/blo//// fowggg//uuuubbbbuo
Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. WebSRC: A dataset for web-based structural reading comprehension. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 4173-4185, Online and Punta Cana, Dominican Republic, 2021. Associ- ation for Computational Linguistics. doi: 10.18653/v1. http//lll/222.//// ht1//22..emnlp-mai..23. https://ac.anttooloy..orr/ org/2221 wtt.:rr/.......mmmmaa......... 343.
URL
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. ArXiv preprint, abs/ / 222121111111131 2024. URL https: //arxiv. org/ arrg/abs/2201... 10935.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024. URL htgppptps// meeereeeeeedennnneneneeeeeeeeeee ineurips.... cccapps.ccpppprrrr Samper_filee www.fflll/ppaaeee /aper/ pdf.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024. URL http:: pttoces//pooeeeeinnggs neurips. cc/paper files/ /ttes//ppe/////// 20w-08882222111-PPaPrrree
and Benchmarks pdf.
Yuan Gao, Kunyu Shi, Pengkai Zhu, Edouard Belval, Oren Nuriel, Srikar Appalaraju, Shabnam Ghadar, Vijay Mahadevan, Zhuowen Tu, and Stefano Soatto. Enhancing vision- language pre-training with rich supervisions. ArXiv preprint, abs/ abs//4403,3334 2024. URL
httbs://arriv.orr/ab////200 03346.
Gemini Google, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemíni: family of highly capable multimodal models. ArXiv preprint, abs/2312.11805 2023. URL https://arxi: http://axxxvv.ogg/bbbb.s/e wtgp 11805.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 6325-6334. IEEE Computer Society, 2017. doi: 10.110//CVPR.2012
URL https:tts///mmmmmm //to:i org/10..////. /0/0//..111 1109/CVPR. 201.............. 2017.67 6700.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. ArXie preprint, abs 220.1/211..199,,, 2024. URL https: org/ htngg/bb//240 13919. 9.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. ArXiv preprint,
2023. URL https: :/ //arxiv. wwtpr/i..orgg/s..... abgg/bbs/331....m 08914.
11

---

Published as a conference paper at COLM 2024
Jihyung Kil, Chan Hee Song, Boyuan Zheng, Xiang Deng, Yu Su, and Wei-Lun Chao. Dual- view visual contextualizationn for web navigation. ArXiv preprint, abs/2402 2440.00476, 2024. 04476.
URL https://arxiv.org/abbs/
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwe- barena: Evaluating multimodal agents on realistic visual web tasks. ArXiv preprint, 13649.
abs/ Abs/2401.13649, 2024. URL
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. Arviv preprint, abs/ ////2220600555555 2023a. URL https://arxiv.org org/ abs/2306. 05425.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal Ilms with generative comprehension. ArXiv preprint, 2023b. URL https: /aaps:///axivvv orw. abrgg//b/300.... .
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. B2ip-2: Bootstrapping language-
image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023c. URL httpt: /ttpc//eooceeen... mlr.prees//162/111 http///v/2/1222nnniiiia pdf.
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5495-5510, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1. 2020.emnlp-main... 443. URL
org/2020. emnlp-main. 443.
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summa- rization Branches Out, pp. 74-81, Barcelona, Spain, 2004. Association for Computational Linguistics. URL https://aclaanthollgg. htt.aannnnooooooooo/ee----0000
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. 07533.
ArXiv preprint, abs/ 2312.00333,,, 2023a. URL https:://arkiiii htttps:/arxxi..org//b////3
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland,, September 6-12, 2014, Proceedings, Part 13, pp. 740-755. Springer, 2014. URL https://linkk. springer. com/ com/chapter/19 1007 9978-978333 3-9-9- 3-319-10602 10602-1 _8##preview.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. ArXio preprint, abs/2311.07557, 2023b. URL httpss///rxivv
htt//bbs/2311..
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Re- inforcement learning on web interfaces using workflow-guidee exploration. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openrevieer net//orrmiid-ry1p3f---
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 34892-34916. Curran As- sociates, Inc., 2023a. URL httpps://proceeeiiggg.. mewrrediiiiinnnnniisseeeeee.eiss ooterarreeeeeeeeeeep-------reeee ConerrCooeerrnecee.. pdf.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. URL https:
12

---

Published as a conference paper at COLM 2024
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? ArXiv preprint, abs/2307.066 06281, 2023b. URL https://arxiv www.woorgrabboo org/abs/ 06281.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: Towards real-world vision-language understanding. ArXio preprint, abs/2403.05525, 2024. URL https: //arxiv.org//abs/4483.
Junhua Mao, Jonathaan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In 2016 IEEÉ Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, Pp. 11-20. IEEE Computer Society, 2016. doi: 10.1109/CVPR201199
URL
1109/CVPR. 2016.9.
OpenAl. Gpt-4v(ision) system card, 2023. URL http:///api.ssmaniicccoollrrrrr CorpusID:263218031.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, 2016. Association for Computational Linguistics. doi: (0.18653/v1/D16 //aclanthology.org org. 0010//////11////// 1264.
URL https:
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Paríkh, and Marcus Rohrbach. Towards VQA models that can read. In IEFF Con- ference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 8317-8326. Computer Vision Foundation IEEE, 2019. doi: 10.1109
00055. URL http: http://openaccess. thecvf. http:/iinnh.owwareee VQA Models That Models,TTattCaa Read_CVPR 2019 wwp.pp....t... html.
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile summarization with multimodal learning. In The 34th Annual ACM Symposium User Interface Software and Technology, pp. 498-510, 2021. URL https://dl acm. org/doi/pdf /10. 1145/3472749 3474765.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. URL https: //libcon/..bco/////o www.iiicooo.buu.....m bupt. edu. cn/http/ abtt:///taaaa///mmmaap
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, al. Cogylm: Visual expert for pretrained language models. ArXiv preprint, 0bb//2311.0079 2023. URL http::///axxv.orr/bbbbbb..
030779.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of- mark prompting unleashes extraordinary visual grounding in gpt-4v. ArXiv preprint, abs//310.11441, 2023. URL httpp:://axxivoorr/aa////11
11441.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Web- shop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 05:20744-20733 2022.
URL https://proocediings.. neurips wwtpscccpppppeeeeeeeeeeee hwtpppppr...llleppprr//22221121. Bec01
pdf.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizzing multi-modal large language model with modality collaboration. Xrv preprint, abs/2311.04257, 2023. URL abs/2311 142277.
13

---

Published as a conference paper at COLM 2024
Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, et al. Lamm: Language-assisted multi-modal instruction-tuningg dataset, framework, and benchmark. Advances in Neural Information Processing Systems, 36, 2024. URL httph/tt//// http://rroreddnnn............... nwwrips.eeem cc/pper files/ Files/paper/ h1t.://a00000.aeaoe//////ee00000
www.eetsssanneeecaaaraaaa.e pdf..
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. Arx preprint, abs/248 2203.04652 2024. URL httpp://arxii org/ abs/2403, 04652.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014. doi: 10.1162/
tacl 00166. URL https: //al:::///aaanne httph//ccalnntooo...ooooooooooo. org//14-1111 1-006
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. ArXiv preprint, 0////23080009999 2023. URL http::///axxivorr//aaaaaa 02490.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. ArXiv preprint, abs/2311.1650x 2023. URL htt.h:tt///aatee wtr.iv.orggabb///111..1 16502.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. ArXiv preprint, abs abs/2440.01114, 2024. URL httpt::/arxiv..co org/ 0ass/2401.01611
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, al. Webarena: A realistic web environment for building autonomous agents. ArXiv preprint, abs/2307.13854 2023. URL https: warriv.orrgabb/22550 13854.
14

---

Published as a conference paper at COLM 2024
A Data Construction and Annotation Details
For the WebQA task, two co-authors carefully examined each screenshot and formulated questions and corresponding answers. For the Action Grounding task, the annotation process was carried out by seven experienced researchers, including four co-authors. The construction processes for the other subtasks were automated through program execution and were relatively straightforward.
B Data Verification and Curation
All samples of our benchmark undergo careful verification and curation through a collabo-
rative effort and division of tasks by two authors. The process encompasses:
Ensuring the main content within screenshots remains unobscured by advertisements or intrusive banners.
Verifying that the meta descriptions describe the most important information generally of the websites.
Headings of websites are correctly extracted for Heading OCR.
The annotated bounding boxes properly encapsulate the target web element description for Element OCR and Element Grounding.
The annotated bounding boxes well aligned with the title of redirected websites for Action Prediction.
The instructions are appropriately matched with their annotated bounding boxes for Action Grounding.
C Annotation Tool of Action Grounding
We developed annotation tool to facilitate the annotation of the action grounding task. The annotation procedure is as follows:
1. Learn about what the shown website is for, based on the presented website descrip- tions, and you may still need to search for the website name in Google to have better understanding.
Refer to action description examples generated by GPT-4V, and then write your instruc- tion. Then, click "Confirm instruction". Please make your instructions diverse, and do not write too many instructions like "search for an item".
3. Move the Mouse to hover over the corresponding element that will be interacted with to accomplish the action description, then press key "s" (instead of CLICK) to select it. After that, green rectangle will be shown to indicate the selected element. Note that the element should be interactive (e.g., clickable or inputtable, etc.
Confirm that the selected element (indicated by blinking green rectangle) correctly corresponds to the action description and click the "submit" button, then click "allow" to allow screen capture.
D Details of Evaluated MLLMs
We consider various general large multimodal models. By default, for each model family, we use the latest, largest, and best-performing available checkpoint date. (i) BLIP-2 (Li et al., 2023c) series bridges the vision-language modality gap with lightweight Q-Former. (ii) InstructBLIP (Dai al., 2024) further performs vision-language instruction tuning based BLIP-2 models. (iii) mPLUG-Ow12 (Ye et al., 2023) adapts modularized network to facilitate modality collaboration while preserving specific features. (iv) Otter (Li et al., 2023a) has improved instruction following and in-context learning capabilities.
(v) VILA (Lin et al., (vi) Fuyu (Bavishi et al., 2023)
2023a) is pretrained with interleaved image-text data at scale. is decoder-only transformer and treats image tokens like text tokens. (vii) SPHINX (Lin
15

---

Published as a conference paper at COLM 2024
Annotation Tool for Action Grounding Task of VisualWebBench
frished
Continus annotaiion
Figure 11: Illustration of the annotation tool (1).
Unieenity
iiv
eilr i. lise cieibbatie renics and seetuacu aecivs for reafy malloer schitat unddes
sorving al
Physics
tonpllack Aumolhiics Haerary splesic Enary Aumpheicc Presoeenc Methcc Arong
Coldectd Joad
Harooaa
vebatta decription atoagg wcches
Prisks
deserption excmple
adustedd aed seach optioes
there otfices
Theay Pain weee)
Confir Intra0od
Figure 12: Illustration of the annotation tool (2).
et al., 2023b) mixes different tuning tasks, and visual embeddings to build a versatile MLLM. (wiii) LLaVA-1.5 (Liu et al., 2023a) combines vision encoder and Vicuna for general- purpose visual and language understanding, and LLaVA-1.6 (Liu et al., 2024) family is the enhanced version with improved image resolution, reasoning, OCR, and world knowledge. We consider three scales: Vicuna-7B, Vicuna-13B, and Hermes-Yi-Ba Yi-34B for model scaling analysis. (ix) Qwen-VL (Bai et al., 2023) introduces trainable query embeddings and single- layer cross-attention module bridge the modalities.
(x) DeepSeek-VL. (Lu al., 2024) incorporates hybrid vision encoder to processe high-resolution images. (xi) Yi-VL (Young et al., 2024) connects the vision encoder with MLLM with a simple MLP projection module and undergoes three-stage training process. (xii) CogVLM (Wang et al., 2023) bridges the modality gap by a trainable visual expert module in the attention and FFN layers of the transformer. We also include Gemini Pro (Google et al., 2023), Claude Sonnet, Claude Opus (Anthropic, 2024), and GPT-4V(ision) (OpenAT, 2023) for comparison.
For all MLLMs, set the temperature to 0.0 for deterministic generations. All experiments are conducted on NVIDIA A100 80G GPUs
E VisualWebBench vs. Mind2Web
Table details the scores MLLMs on VisualWebBench and Mind2Web.
F Additional Case Studies
In Figure 13, CogAgent cannot understand the instruction correctly and output unrelated content. LLa captures the instruction, yet generates wrong answer. In contrast,
16

---

Published as a conference paper at COLM 2024
Vebenlhe
Mind2Web
SeeClick
9.7
20,9*
Qwen-VL.
23.9
10.2*
CogAgent
28.7
15.5t
LLaVA-1.5-7B
17.0
4.0
LLaVA-1.5-13B
19.4
9.6
LLaVA-1.6-7B
36.0
3.6
LLaVA-1.6-13B
39.4
6.3
LLaVA-1166344
50.5
13.6
Gemini 1.0 Pro
48.0
17.7f
GPT-4V(ison)
64.6
36.5t
Table 3: The comparison between scores of VisualWebBench and Mind2Web. indicates the results taken from Cheng et al. (2024), while t denotes those taken from Zheng et al. (2024).
WebQA
Question: How many threads are there in the Electric Power Systems' forum?
LLaVA-34B
Thought: will count the
number of threads 'Electric
CA
Power Systems' forum.
Answer: 10
BED MAINS
CogAgent
Thought: Locate the 'Electric
Power Systems' on the page read the number of...
GPT-4V
Thought: The number of threads in 'Electric Power
robar
Systems clearly listed
under the forum's name.
Answer: 64,418 threads
Figure 13: Case study of WebQA.
both the thinking process and the answer of GPT-4V are correct. This demonstrates the strong understanding and reasoning ability of GPT-4V.
Case studies for Meta Description Generation, Heading OCR, Element Grounding, and Element OCR are shown in Figure 14, Figure 15, Figure 16, and Figure 17.
17

---

Published as a conference paper at COLM 2024
Meta Description Generation
Instruction:
Generate meta description for the webpage.
Answer: Shop al Aircraft Sprace for selection certified and homebeill aircraft parts, avionics, pilot supplies, and everything aviation needs one place.
LL-VA-34B (0.275): The avialion supersiore for
all your pilot Shop brand, part Fun.
beyword. Celebrating years
your
CogAgent (0.202): website
online store of Aircraft Sprace Specialty Co, Inc. leading distributor of aviation site offers variety products including aircrali parts, avionics, and wood produutis, site features navigation bar with optins such as 'Shop By Brand', and search bar. There also promotional barmer celebrating the arniversary, and section sabseribing their newsletter. Additionally: there are featured products like the 'erobrew Coffee Company and Gill Aviation Watteries.
and supplies.
50
is
CPT-4V (0.355): Explore wide range aviation superstone, Find avionies, engine parts, hardware, and more for airevaft Stay updated promotions, and discounts.
parts and pilot supplies the
weekly newsletter new product announcements,
Figure 14: Case study of Meta Description Generation.
Heading OCR
Instruction: Identify the position heading and generate its content.
USAA insurance, banking and retirement solutions.
Answer:
retre mand rile
USAA has insurance, banking and retirement solutions.
LLaVA-34B:
Cog Nognt: <EMPTY>>
GPT-AV: USAA has insurance, banking and retirement solutions.
thruugh
sentos aland
Figure 15: Case study of Heading OCR.
Element Grounding
Instruction: Select the element with the description "Help Center Safety"
C toums Emeame S misuuee
Local, trusted pet care
Bink ete but comgleee wor
Answer:
LLaVA-34B:
CogAgent:
Why Wag!?
GPT-IV:
Safett oe
Figure 16: Case study of Element Grounding.
18

---

Published as a conference paper at COLM 2024
Element OCR
Instruction: Conduct OCR the element with red rectangle.
This question is the most important for investors answer. After all, experts suggest limits on how much any types of investments should go into...
Answer:
Shop Pop Collectibles
LLaVA-34B: question one the most important for investors answer. After all, experts suggest limits on how much of any types of investments should go into...
Duikk Bop leventing
CogAgent: <INVALID CONTENT>
GPT-4V: 2023 Niue or Silver $2 Mandalorian...
Figure 17: Case study of Element OCR.
19


Reviews:
[{"review_id": "74HWm2mwaz", "reviewer": "Reviewer_f5BP", "summary": "This work presents “WebBench,” a new benchmark for evaluating multimodal language models (e.g., image, text → text), specifically focusing on applications involving reasoning over webpages. The benchmark consists of ~1.5K example prompts (webpage/input/response tuple) spanning 139 individual websites, and **7 task families** each with their own evaluation metric:\n\n1. *Captioning* (ROUGE Score): Given a webpage screenshot, generate a natural language summary of the website (where ground-truth is a GPT-4V rewrite of the HTML “meta” tag)\n2. *WebQA* (F1 Score): Given a question about the webpage (written by a human annotator) generate the answer (open-set generation). \n3. *Heading OCR* (ROUGE Score): Generate the heading text for the given webpage (extracted from the first <h1> tag).\n4. *Element OCR* (ROUGE Score): Given a webpage with a highlighted DOM element > 20 words (drawn bounding box), transcribe the given element in natural language.\n5. *Element Grounding* (Multiple Choice Accuracy): Given a human-provided description of an HTML DOM element, and 8 potential candidates (drawn bounding boxes), select the candidate element that best matches the description.\n6. *Action Prediction* (Multiple Choice Accuracy): Given a highlighted DOM element (with an href tag), select the title of the “target” webpage given a set of candidates (with negatives scraped from other elements of the same webpage).\n7. *Action Grounding* (Multiple Choice Accuracy): Given a human-provided high-level instruction (e.g., “search for flights to Montreal”) and eight candidate DOM elements (drawn bounding boxes), identify the candidate element that is most relevant to the instruction (e.g., search bar for “Flights”).\n\nBeyond curating this extensive benchmark, the work also evaluates and performs a thorough analysis of 14 state-of-the-art multimodal language models (14 open-source, 2 closed-source) across all tasks, including a stratified breakdown of how models behave as a function of scale (7B vs. 13B vs. 34B), input resolution, and manual error analysis. The appendix provides additional detail as to the data annotation and verification process. The full evaluation benchmark and evaluation harnesses for open-source models will be released.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of our benchmark as high-quality, clearly defined, and well-motivated!\n\n**Metric for OCR Tasks:** We reported the F1 value of ROUGE-L because the average ground-truth length in our dataset is longer than that of common scene OCR datasets (e.g., Element OCR averages 39.5 words). Consequently, the longest common subsequence may be more appropriate than exact match for evaluation. For reference, we also provide exact match (EM), word-level F1, and ROUGE-L results below. **The EM metric is overly strict and does not accurately reflect OCR capabilities, while word-level F1 is similar to ROUGE-L.** For simplicity, we will continue to use ROUGE-L as the evaluation metric for the two OCR tasks, and we will include the following table in the appendix.\n\n|  | HeadOCR(ROUGE-L) | HeadOCR(EM) | HeadOCR(F1) | EleOCR(ROUGE-L) | EleOCR(EM) | EleOCR(F1) |\n|-|-|-|-|-|-|-|\n| LLaVA-1.6-7B | 57.3 | 39.1 | 57.3 | 54.8 | 4.1 | 55.4 |\n| LLaVA-1.6-13B | 52.8 | 43.5 | 52.8 | 56.1 | 8.6 | 56.6 |\n| LLaVA-1.6-34B | 67.1 | 47.8 | 67.1 | 71.9 | 13.9 | 72.2 |\n| Gemini 1.0 Pro|75.1|58.7|75.1|65.4|18|60.3|\n| Gemini 1.5 Pro|54.5|26.1|54.8|76.6|23.7|73.2|\n| Claude Opus|63.7|45.7|63.7|87.1|45.7|87.1|\n| GPT-4V(ision)|68.8|50|68.8|62.8|27.8|63|\n\n**Annotator Details:** For the WebQA task, **two co-authors** carefully examined each screenshot and formulated questions along with corresponding answers. For the Action Grounding task, the annotation process was carried out by **seven experienced researchers, including four co-authors**. We have included the annotator IDs for each sample in the anonymous dataset (see the link below). Additionally, **two co-authors** conducted the final verification for all subtasks. \n\n**Annotation Pipeline for Other Subtasks:** Thanks for the catch! All subtasks involved human annotation as described in Appendix B. Since the annotation processes for the other subtasks were relatively straightforward, we did not provide extensive details. We will expand Section B to clarify the annotation processes for all tasks.\n\n**Open-Source Raw Data:** **Yes! We plan to make all resources open-source.** This includes URLs, access dates, raw crawled data, final data samples for different tasks, the code for rendering websites, and the inference code for evaluation. These resources have been uploaded to an anonymous repository for preview (https://anonymous.4open.science/r/Anonymous_WebBench-E7D4) and will be publicly released.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a strong paper introducing a new evaluation that 1) fulfills a demonstrated need given the recent proliferation of multimodal language models, and 2) clearly distinguishes itself from prior web-based evaluation benchmarks such as Web Shop and Visual WebArena by focusing on clearly defined and fine-grained “subtasks” necessary for reasoning over websites. \n\nWhile the overall sample size per each of the 7 subtasks is on the smaller side, I do believe the existing examples are clean and high-signal. The effort the work makes to leverage human annotators to both provide input prompts and verify outputs is clear, and further strengthens the benchmark. Finally, the analysis of existing models demonstrates that this benchmark is not only valid, but that there are subtasks with a ton of “headroom” — showing that this evaluation will be useful to the field and those working on developing stronger MLLMs in the future.", "reasons_to_reject": "There are a few minor issues that I believe the authors can easily fix for the final submission. First, it’s a bit strange to me that the evaluation metric for all of the OCR-based subtasks (e.g., Element OCR, Heading OCR) is ROUGE score rather than exact match (or F1 score); if we want high-fidelity OCR, shouldn’t we be penalizing our MLLMs for any generations that deviate from the ground-truth?\n\nFurthermore, some of the details around the annotation procedure are missing (how annotators are selected, which annotators are used for generating which data per subtask, which data the authors of the work actually verified vs. external annotators). It would be really great to be specific and clear about these details in the final version of the paper.", "questions_to_authors": "Were external annotators only used for the action grounding task? If not, could you provide a description of the annotation pipeline used for the other subtasks (similar to Appendix A)?\n\nBeyond the webpage screenshots and input prompts provided in the benchmark, can you clarify what other metadata about the webpages you plan to release? Minimally, it would be great to get original URLs, access dates, as well as code for rendering websites (e.g., the Playwright pipeline described in the paper).", "extra_scores": {}}, {"review_id": "xY3nwupET9", "reviewer": "Reviewer_RE1U", "summary": "This work presents WebBench, a new webpage understanding benchmark of just over 1.5k samples. The benchmark is designed to target seven different web page related tasks: captioning, webQA, heading OCR, element OCR, element grounding, action prediction and action grounding. Using the new dataset, 18 different model variants (of 16 unique model architectures) are evaluated. Along with the quantitative task evals, there is substantial analysis, including correlation analysis among the tasks themselves, as well as the correlation between other vision-language scores with WebBench aggregate scores and the relationship between WebBench scores and image resolution size. The work also includes a couple of qualitative examples of failure cases and ablation studies for grounding.\n\nThis paper presents high quality work with an appropriate amount of analysis and discussion. Its thorough benchmarking across many model variants is also appreciated. While the benchmark is not very large, this topic is a growing research area and having additional evaluation resources would positively impact the community.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer finds our benchmark well-designed, the experiments comprehensive, and the analyses insightful.\n\n**Differences from Existing Datasets:** 1) As described in the third paragraph of the introduction, web-agent benchmarks like Webshop [1] **focus on end-to-end capabilities without a fine-grained assessment of specific skills** such as OCR, semantic understanding, and grounding, which was also mentioned by **Reviewer f5BP (thanks!)**. 2) The WikiWeb2M dataset [2] includes only three captioning tasks on a webpage, and the WIT dataset [3] focuses on image-text retrieval. Additionally, **both datasets are limited to Wikipedia web pages and do not reflect the diverse scenarios web agents encounter**. In contrast, WebBench spans 139 most popular websites, covering 87 sub-domains and encompassing seven different sub-tasks, including captioning, OCR, web QA, and element or action grounding.\n\n**Noise in Task Annotation:** As detailed in Appendix B, **the human annotators undertook meticulous efforts to curate and verify all subtasks carefully**. For the element grounding subtask, we identified and removed the problematic cases, including those noted by the reviewer.\n\n**Not that many samples per task**: Our dataset contains multiple examples of different subtasks within a single screenshot. **We include more screenshots rather than annotating numerous examples within the same screenshot to ensure comprehensive coverage of diverse real-world scenarios.** Consequently, our dataset spans 139 of the most popular websites across 87 sub-domains. Notably, several widely adopted multimodal datasets prioritize diversity over quantity, such as MMVet with 218 samples and MME with 2374 samples \\[4\\]\\[5\\].\n\n[4] Yu, Weihao, et al. \"Mm-vet: Evaluating large multimodal models for integrated capabilities.\" International Conference on Machine Learning, 2024, PMLR.\n\n[5] Fu, Chaoyou, et al. \"MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models.\" 2023, arXiv, eprint 2306.13394.\n\n**Calculation Detail of Correlations Between Subtasks in Figure 5:** From the results in Table 1, we have seven sets of metrics (columns), and the Pearson correlation coefficient is calculated between every two columns.\n\n**Typos**: We will correct the typos in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a well designed new benchmark for multimodal webpage understanding, consisting of a range of 7 different tasks. The tasks capture different levels of webpage understanding: the full webpage, element, and action level.\n2. Along with the data there are substantial evals run across many models, providing comprehensive baselines, most of which are open source and easily reproduced.\n3. There are numerous analyses and ablations provided, which verifying desirable qualities in the dataset and its utility in future work.", "reasons_to_reject": "1. While more resources are certainly useful in this topic area, there are captioning and grounding datasets that would capture several of the tasks proposed in WebBench; either from prior human annotation efforts or automatically synthesized datasets. From the related work discussion and examples, it is not clear to me what distinguishes the dataset from prior work.\n\n[1] Yao et al. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. NeurIPS 2022.\n\n[2] Burns et al. A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding. EMNLP 2023.\n\n[3] Srinivasan et al. WIT: Wikipedia-Based Image Text Dataset for Multimodal Multilingual Machine Learning. ACM SIGIR 2021.\n\n2. Noise in task annotations. In the construction of the element grounding task, it's highly probable that there is noise in the annotations. For example, are there elements selected as negatives that could be congruent with the target element in action prediction task?\n\n3. Another drawback is that it seems the tasks are annotated on separate samples, and ultimately there aren’t that many samples per task. This makes the benchmark a bit less well motivated because there are other datasets that cover the same task, and it would have been nice to have annotations for many tasks on a single screen, instead of them being non-overlapping / disjoint.", "questions_to_authors": "Questions\n1. How is the correlation between tasks in Figure 5 calculated exactly?\n\nBelow are comments/suggestions for the authors\n1. As mentioned in the reasons to reject, there are a couple of highly relevant related work that I would suggest adding to the prior work/citations:\n\n[1] Yao et al. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. NeurIPS 2022.\n\n[2] Burns et al. A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding. EMNLP 2023.\n\n[3] Srinivasan et al. WIT: Wikipedia-Based Image Text Dataset for Multimodal Multilingual Machine Learning. ACM SIGIR 2021.\n\n2. The last sentence in Figure 3 seems unnecessary, it was probably from an earlier version of the draft (I'd suggest removing).\n3. Figure 2 is never referenced in text.", "extra_scores": {}}, {"review_id": "2LWXpY3Giv", "reviewer": "Reviewer_3CW7", "summary": "The paper proposes WebBench, a fine-grained benchmark for webpage understanding. Specifically, the paper argues that existing web benchmarks for multimodal LLMs are focused on end-to-end or agent capabilities, conflating multiple challenges in one task and thus hard to pin down what is truly challenging for a tested model.\n\nWebBench has 1.5k samples from 39 websites.  It breaks down web understanding into 3 levels: (1) website (2) element and (3) action. Under each level, there are elementary skills needed such as HeadOCR and Captioning for (1) and grounding for (2).\n\nThe authors evaluated both closed-source and open-source multimodal LLMs on WebBench and found even the SOTA model (GPT-4V) achieves only ~65%, indicating room for improvement.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s recognition of our efforts to break down the challenges in web understanding across a wide range of models. Below, we add the results of Gemini 1.5 Pro and Claude 3 Opus. **Overall, Gemini 1.5 Pro performs the best on the majority of the subtasks and achieves the highest average score.** Claude Opus shows exceptional performance on Element OCR. We will include the results and the analysis in our revision!\n\n\n| Model          | Caption  | WebQA    | HeadOCR  | EleOCR   | EleGround | ActPred  | ActGround | Average  |\n| -------------- | -------- | -------- | -------- | -------- | --------- | -------- | --------- | -------- |\n| Gemini 1.0 Pro | 25       | 55.5     | **75.1** | 65.4     | 44.3      | 26.7     | 43.7      | 48       |\n| Gemini 1.5 Pro | 31.6     | 69       | 54.5     | 76.6     | **70**    | **74.4** | **77.7**  | **64.8** |\n| Claude Opus    | 26.7     | **75.4** | 63.7     | **87.1** | 57.7      | 60.4     | 38.8      | 58.5     |\n| GPT-4V(ision)  | **34.5** | 75       | 68.8     | 62.8     | 67.5      | 67.6     | 75.7      | 64.6     |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ Overall I find WebBench as a worthwhile effort to break down challenges in web understanding. Web understanding is known to be complex and often requires understanding of different levels and sometimes agentic workflow. It makes a lot of sense to test these capabilities individually instead of having one end-to-end task which can end up being too challenging and uninterpretable.\n\n+ The authors tested a wide range of models, especially open-sourced ones. One nit-picking suggestion is to include more state-of-the-art closed source models such as Gemini 1.5 Pro and Claude 3 Opus.", "reasons_to_reject": "I do not have a major reason to reject.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zzm6vX1dyF", "reviewer": "Reviewer_VZGD", "summary": "The paper proposes a new evaluation dataset called WebBench that tries to cover the issues of existing benchmarks being: failing to capture the unique features of web pages and to measure fine grained capabilities such as OCR, understanding and grounding.\n\nWebBench consists of 1.5k instances covering seven tasks: captioning, webpage QA, heading OCR, element OCR, element grounding, action prediction and action grounding.\n\nThe authors evaluate 14 open-source MLLMs covering closed and open source models.\n\nThe benchmark is constructed using different processes depending on the task. Some use LLMs and human verification, others are completely carried out by human annotators.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer recognizes our benchmark as a valuable resource for evaluating multimodal language models in webpage understanding!\n\n**Bias Introduced by GPT-4V**: We considered the potential bias introduced by GPT-4V during our data annotation process. **Consequently, we have undertaken further data verification by human annotators as detailed in Appendix B.** For the action grounding task, as described in Appendix A, annotators were encouraged to write their own instructions. Ultimately, **80% of annotated action descriptions are different** from the ones initially generated by GPT-4V.\n\n**Unique Screenshot Numbers:** The total number of unique screenshots is 332, instead of 139. While captioning, heading OCR, and action grounding tasks generate one sample per website, other subtasks construct multiple samples for each website.\n\n**Related Work:** We discussed the related work that relies on the DOM/source code to carry out tasks on web pages **in Introduction and Section 2.2** (e.g., web agent benchmarks like MiniWob++, WebShop, Mind2Web, and WebArena). **The mentioned work DocLLM, focuses on documents and tables understanding rather than web pages, which clearly has different goals from our work.** We will include the comparison between DocLLM and our WebBench in the revised version. \n\n**Automatically Extracted h1 Tag**: As indicated in Figure 3, **only 46 out of 139 samples** are properly displayed in screenshots and can be considered good headings for web pages. **This underscores the necessity of our human verification process.**\n\n**Results of Gemini Pro in Figures 4 and 6**: We have uploaded the updated figures to the anonymous link. (https://postimg.cc/gallery/F0yRZ0f). \n\n**Model-Based Metrics for Open-Ended Generation Tasks**: Given that model-based metrics may introduce **uninterpretable bias and additional computational cost**, we prefer lightweight and rule-based evaluation metrics. For instance, for the WebQA task, annotators are instructed to propose objective questions with short answers (less than five words in most cases), making SQUAD-F1 the appropriate metric.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The benchmark covers a sufficiently wide range of tasks on website screenshots and it is definitely a useful resource for evaluating multimodal language models.\n\nThe evaluation includes the main closed and open source LLMs.", "reasons_to_reject": "There is one issue in the way the evaluation data is produced for one task, namely captioning. In this task, metadata that is mostly a sequence of keywords is converted into a more natural description with GPT-4V. In addition, the latter is seeing both screenshot and metadata. At evaluation time, each model is asked to generate the description. This whole process gives an unfair advantage to GPT-4V, which is exacerbated by using ROUGE-L as the evaluation metric for open-ended tasks. It is fair to say that gold and predicted descriptions would exhibit similar structure and wording. This raises a question about how much GPT-4V relies on the keywords to output the description, and how much information is actually coming from the screenshot. Maybe the model relies more on the image?\n\nOn a similar note, the appendix shows that action descriptions are also suggested to annotators by GPT-4V. What if the model is generating and biasing the annotators towards actions that it can successfully perform? Would this possibility conflate GPT-4V performance on this task too?", "questions_to_authors": "Is it correct saying that the number of unique screenshots is 139?\n\nWhy is a discussion / related work on methods that relies on the DOM/source code to carry out tasks on web pages not in the paper?\nE.g. DocLLM: A layout-aware generative language model for multimodal document understanding\n\nFor the heading OCR task, how often the H1 header is the right one?\n\nWhy is Gemini Pro missing from Figure 4 and 6?\n\nWhat about using model-based metrics or critics for evaluating the open ended generation tasks?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new benchmark for evaluating multimodal language models. All reviewers found the paper to be useful and of high quality. The authors are highly encouraged to implement the valuable feedback given by the reviewers since this will likely contribute to further improve the paper.
Decision: Accept