VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?

Authors: junpeng  liu
yifan  song
bill yuchen lin
wai  lam
graham  neubig
yuanzhi  li
♢  xiang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-19 07:24:31

Abstract: Multimodal Large Language models (MLLMs) have shown promise in web-related tasks, but evaluating their performance in the web domain remains a challenge due to the lack of comprehensive benchmarks. Existing benchmarks are either designed for general multimodal tasks, failing to capture the unique characteristics of web pages, or focus on end-to-end web agent tasks, unable to measure fine-grained abilities such as OCR, understanding, and grounding. In this paper, we introduce VisualWebBench, a multimodal benchmark designed to assess the capabilities of MLLMs across a variety of web tasks. VisualWebBench consists of seven tasks, and comprises 1.5K human-curated instances from 139 real websites,  covering 87 sub-domains. We evaluate 16 open-source MLLMs, Gemini Pro, Claude-3 series, and GPT-4V(ision) on VisualWebBench, revealing significant challenges and performance gaps. Further analysis highlights the limitations of current MLLMs, including inadequate grounding in text-rich environments and subpar performance with low-resolution image inputs. We believe VisualWebBench will serve as a valuable resource for the research community and contribute to the creation of more powerful and versatile MLLMs for web-related applications.

Main paper: 
 introduction: 
 The web is an indispensable platform for information exchange and interaction, presenting unique challenges and opportunities for multimodal learning. While web content has been a primary source of training data for multimodal large language models (MLLMs) (OpenAI, 2023; Google et al., 2023; Liu et al., 2023a) , a largely overlooked aspect is understanding of websites themselves. Every website is designed to be visually rendered for consumption by human users, with structured layouts, rich textual information, and diverse interactive elements. Enabling MLLMs to accurately comprehend websites would unlock numerous applications in the web domain. fine-grained recognition and accurate spatial reasoning and grounding. The vast diversity of website designs, layouts, and content across different domains further complicates the creation of representative and robust evaluation benchmarks, necessitating the inclusion of a wide range of website categories to ensure the generalizability of the evaluated models.Despite the growing importance of the web domain in multimodal learning, existing benchmarks fall short of comprehensively evaluating the fundamental capabilities of models in this context. General MLLM benchmarks (Young et al., 2014; Liu et al., 2023b; Yue et al., 2023) , do not adequately capture the unique characteristics of the web domain. On the other hand, web-agent benchmarks, like WebShop (Yao et al., 2022) , Mind2Web (Deng et al., 2024) , and (Visual)WebArena (Zhou et al., 2023; Koh et al., 2024) , focus on end-to-end abilities without offering a fine-grained assessment of essential skills such as OCR, semantic understanding, and grounding. Measuring these fine-grained abilities is crucial, as they serve as building blocks for complex web-related tasks, enable targeted improvements, and provide a clearer picture of a model's performance. The lack of granularity in existing benchmarks hinders the development of more capable multimodal models for the web domain, emphasizing the need for a comprehensive evaluation benchmark.To address these limitations, we introduce VisualWebBench, a comprehensive multimodal benchmark designed to assess the capabilities of MLLMs in the web domain. Inspired by the human interaction process with web browsers, VisualWebBench consists of seven tasks that map to core abilities required for web tasks: meta description generation, webpage QA, heading OCR, element OCR, element grounding, action prediction, and action grounding, as detailed in Figure 1 . The benchmark comprises 1.5K instances, all uniformly formulated in the QA style, making it easy to evaluate and compare the performance of different MLLMs. In summary, VisualWebBench offers a standardized benchmark for evaluating MLLMs in web understanding, enabling the development of more capable and efficient models, autonomous web agents, and web-related applications. 
related work: 
 Before detailing VisualWebBench, we briefly outline its differences with existing MLLM benchmarks, also outlined in Figure 2 .In concert with improvements in these MLLMs, benchmarks have also evolved. These range from traditional single task benchmarks like VQA (Antol et al., 2015; Goyal et al., 2017) , RefCOCO (Mao et al., 2016) , and Flickr30K (Young et al., 2014) , to more holistic evaluation benchmarks like LAMM (Yin et al., 2024) , MMBench (Liu et al., 2023b) , and MMMU (Yue et al., 2023) , recently. In this work, we focus on images in web-based scenarios characterized by structured layouts, copious textual data, and diverse interactive elements, which pose new challenges for current MLLMs. The most closely related scenario to this work is GUIbased tasks, exemplified by Screen2Words (Wang et al., 2021) , Widget Captioning (Li et al., 2020) , and WebSRC (Chen et al., 2021) which is a web-based VQA dataset. Different from previous works, VisualWebBench offers a comprehensive evaluation for MLLMs, spanning perception, comprehension, grounding, and reasoning capabilities.As a vital aspect of daily life, methods that perform various tasks in web scenarios have garnered widespread attention from researchers. Earlier efforts introduce simplified sim- ulated environments for web navigation tasks, such as MiniWob++ (Liu et al., 2018) and WebShop (Yao et al., 2022) . Recently, Mind2Web (Deng et al., 2024) , WebArena (Zhou et al., 2023 ), VisualWebArena (Koh et al., 2024) construct realistic and reproducible web environments to facilitate the development of web agents. There are also various studies to improve the web understanding or grounding capabilities of MLLMs (Gao et al., 2024; Kil et al., 2024) or develop agents for autonomous web navigation (Hong et al., 2023; Zheng et al., 2024; Cheng et al., 2024; He et al., 2024) . Despite their success, the community still lacks a comprehensive evaluation of MLLMs' basic performance in web scenarios, including perception, understanding, grounding, and reasoning.3 The VisualWebBench Benchmark 
overview of visualwebbench: 
 We present VisualWebBench: a multimodal benchmark designed to thoroughly evaluate the understanding and grounding capabilities of MLLMs in web scenarios. The proposed VisualWebBench possesses the following features: 1) Comprehensiveness: VisualWebBench spans 139 websites with 1.5K samples, encompassing 12 different domains (e.g., travel, sports, hobby, lifestyle, animals, science, etc.) and 87 sub-domains. 2) Multi-granularity:VisualWebBench assesses MLLMs at three levels: website-level, element-level, and actionlevel. 3) Multi-tasks: VisualWebBench encompasses seven tasks designed to evaluate the understanding, OCR, grounding, and reasoning capabilities of MLLMs. 4) High quality: Quality is ensured through careful human verification and curation efforts. The domain distribution and statistics of our benchmark are presented in Figure 3 .To ensure comprehensive coverage across diverse domains and top-ranking websites, our website selection process is conducted based on SimilarWeb1 . We start from 12 top-level domains in SimilarWeb such as Science, Engineering, Sports, Lifestyle, and more, which are subsequently broken down into 87 sub-domains. Then we manually select representative websites from the top-5 most ranking websites in each sub-domain. Our selection criteria prioritize websites with rich interactive elements, including images and buttons, while excluding those that have been used in prior web agent benchmarks like Mind2Web and WebArena. We use Playwright2 to render and save the websites automatically.This section details the proposed seven tasks of VisualWebBench and the process of constructing data for each task; examples are shown in Figure 1 .Meta Description Generation. This task aims to evaluate the MLLMs' ability to comprehend and summarize the content of a webpage screenshot. The meta description, i.e., <meta name="description"> tags in the head section of HTML, is a brief snippet of text that helps humans or search engines understand the content of websites. However, the quality of extracted meta descriptions cannot be ensured and their styles are pretty different on diverse websites. For example, some meta descriptions only consist of a list of keywords or a short title of the website, instead of a natural language description. Hence, we instruct GPT-4V to generate a better meta description, given both the screenshot and the extracted meta description. The final descriptions are verified and curated by the authors.WebQA. To assess the understanding capabilities of MLLMs in the web scenario, VisualWebBench involves a webpage QA task, where the MLLM will answer open questions that demand a thorough comprehension of the visual layout. Human annotators are instructed to examine each screenshot and craft up to five challenging questions which satisfy: 1) A degree of reasoning ability is required to answer the question, 2) The answers should be precise and objective.Heading OCR. This task requires MLLMs to locate and recognize the text of the heading of a website. Different from the traditional OCR task where a target element is given, as shown in Figure 1 , the input of heading OCR is simply a raw screenshot, and the expected output is the heading content. The ground-truth target is automatically extracted from the first <h1> element in the HTML.Element OCR. This task evaluates the capability of MLLMs to conduct OCR on lengthy texts. Firstly, we traverse the HTML DOM tree and extract the bounding boxes and textual description of each element Then, we select elements whose text descriptions comprise over 20 words. The task input consists of a screenshot with a bounding box indicating the position of the element to be recognized.Element Grounding. Grounding, or Referring Expression Comprehension (REC), is a crucial image-text alignment capability, particularly for MLLMs interacting with web environments. Given a description of an HTML element, MLLM needs to locate the corresponding region in the screenshot. However, our preliminary studies reveal that current MLLMs struggle to directly give the coordinate of the target's bounding box (see 4.6). Inspired by Yang et al. (2023) , we adopt a simplified setting where eight candidate bounding boxes are presented. Differently, the candidate elements here are extracted automatically using Playwright, with each assigned an alphanumeric ID. MLLMs are then prompted to select the box that best matches the given element description. The element description, golden bounding box, and negative bounding boxes of randomly chosen elements are automatically extracted from the webpage.Action Prediction. This task asks MLLMs to predict the title of the redirected website after clicking an element, in a multi-choice QA way. In terms of the construction process, firstly, we employ Playwright to click all clickable elements within the web page and save the <title> or <meta name="title"> tag as titles of new redirected web pages. Subsequently, we randomly sample seven additional elements distinct from the target element and take the titles of their respective redirect destinations as negative choices. Cases where a click does not lead to a title change are omitted from consideration. The task presents input in the form of screenshots highlighting the clickable target with a red bounding box. Accompanying each screenshot is eight choices, each labeled with a letter. The ground truth output is the letter corresponding to the correct answer.Action Grounding. In addition to directly grounding elements from their descriptions, we further introduce the action grounding task. In this task, the MLLMs are given a human instruction, such as "search for the hotels in NYC", and are prompted to determine the correct element to click to fulfill the instruction. Similar with Element Grounding, MLLMs take in a screenshot containing bounding boxes of eight candidate elements and select the most appropriate one. The task data is completed by seven experienced annotators, and an annotation tool is developed to streamline the annotation workflow. Further details about the annotation tool and the annotation process can be found in Appendix C.All tasks above adopt a VQA-style formulation similar to customary multimodal benchmarks. All screenshots in VisualWebBench are unified in a standard 1280-pixel width. All samples of our benchmark undergo careful verification and curation through a collaborative effort and a division of tasks by two authors. See Appendix B for more details.We adopt different evaluation metrics for different tasks in VisualWebBench. For open-ended generation tasks, ROUGE-L (Lin, 2004 ) is used to measure the quality of the generated responses. For the WebQA task, SQuAD style F1 (Rajpurkar et al., 2016) is employed as the evaluation metric. For multiple-choice tasks, we measure accuracy. 
evaluated mllms: 
 We evaluate 16 open-source general MLLMs on VisualWebBench (See Appendix D for model details). By default, for each model family, we use the largest available checkpoint. We consider three scales of LLaVA, 7B, 13B, and 34B, for model scaling analysis. Several strong close-source MLLMs, Gemini Pro (Google et al., 2023) , Claude series (Anthropic, 2024) , and GPT-4V(ision) (OpenAI, 2023) , are also included for evaluation.Recent studies have introduced several MLLMs tailored to create agents for GUI tasks, such as web and smartphones (Cheng et al., 2024; Hong et al., 2023; Gao et al., 2024) . Therefore, we consider two open-source GUI-specialized MLLMs for evaluation: SeeClick (Cheng et al., 2024) is developed by GUI grounding pretraining based on Qwen-VL (Bai et al., 2023) . CogAgent (Hong et al., 2023) is built upon CogVLM (Wang et al., 2023) , focusing on GUI interpretation and navigation, with support for high-resolution image inputs.In this section, we present a comprehensive comparison of different MLLMs on VisualWebBench in Table 1 . From the results, we highlight the following findings.Challenging Nature of Web Tasks: Even the powerful GPT-4V achieves an average score of only 64.6 on VisualWebBench, leaving ample room for improvement. For tasks requiring strong reasoning and grounding abilities (Action Prediction and Action Grounding), many MLLMs struggle to surpass random chance (12.5). This underscores that current models cannot effectively handle many tasks within the web scenario.Claude outperform open-source MLLMs including GUI agent MLLMs by a large margin, highlighting a discernible gap in the capabilities of current open-source MLLMs compared to proprietary ones like GPT-4V. Meanwhile, LLaVA-1.6-34B achieves a commendable result (50.5) and beats all other open-source MLLMs, even outperforming the performance of Gemini 1.0 Pro (48.0). Notably, we find Claude Sonnet surpasses Opus on all tasks in VisualWebBench, suggesting that Sonnet may possess more powerful capabilities in web scenarios.Scaling Leads to Better Performance: Compared with the 7B and 13B versions of LLaVA-1.6, the 34B model achieves a performance boost across almost all tasks, reaching an average score of 50.5. Although there are factors other than scale, such as different backbone LLMs, this indicates that increasing model size is a promising avenue for enhancing the capabilities of open-source MLLMs in web-related tasks.SeeClick and CogAgent are two MLLMs pretrained on GUI grounding tasks. However, we observe that these GUI agent MLLMs do not exhibit significant performance improvement. For example, SeeClick fails to outperform Qwen-VL, its base MLLM, across all tasks. Notably, we find these models suffer catastrophic forgetting (Wang et al., 2024) on general instruction following capability after training on GUI grounding data. These results underscore the necessity for more general GUI-specific training techniques to enhance the MLLMs' performance in the web scenario. To further investigate the effectiveness of GUI grounding training, we perform a comprehensive comparison of various grounding settings in Section 4.6.We delve into the relationship between the performance of MLLMs in the web scenario and that in general and agent scenarios. Specifically, we use MMMU (Yue et al., 2023) as the proxy of MLLMs' capability in general scenario3 , and Mind2Web (Deng et al., 2024) is used for the evaluation of the agent scenario.While Figure 4 somewhat suggests some correlation between VisualWebBench score and MMMU score, the relationship does not appear to be significant. In other words, performing well in the general domain does not necessarily guarantee the same trend in the web scenario. For example, while Yi-VL-6B and BLIP2 outperform CogVLM on MMMU, they fall short in achieving a good score on VisualWebBench. It is also noteworthy that LLaVA-1.6-34B performs well on both tasks, nearly matching the performance level of GPT-4V.As illustrated in Figure 5 , generally, VisualWebBench scores are higher than those of Mind2Web4 , which demonstrates that there still exists large room for improvement of agent ability empowered by webpage understanding, grounding abilities, as well as other abilities like planning. GUI agent MLLMs tend to exhibit overfitting in terms of agent capability, resulting in underperformance in understanding web pages. Moreover, in Figure 6 , we conduct an in-depth correlation analysis between MMMU and VisualWebBench seven subtasks, as well as a similar analysis between Mind2Web and VisualWebBench tasks. From the results in Table 1 , we have seven sets of metrics (columns), and Pearson correlation coefficients are calculated between every two columns. For MMMU, the correlations are generally strong. Specifically, the two subtasks requiring heavy reasoning, WebQA and Action Prediction, strongly correlate with MMMU. For Mind2Web, the correlation between scores on VisualWebBench and those on Mind2Web is low, even exhibiting two negative correlations in Meta Description Generation and Action Grounding. These findings suggest that VisualWebBench offers a different evaluation perspective for MLLMs in the web scenario. VisualWebBench. This analysis reveals a strong correlation among specific tasks, namely Meta Description Generation, WebQA, and Element OCR, all demanding a comprehensive understanding of textual content within webpages. In contrast, Action Prediction and Action Grounding tasks exhibit a minimal correlation, implying distinct skill sets necessary for predicting action outcomes versus pinpointing elements for actions. Moreover, Action Grounding seems to be less correlated with all other non-grounding tasks, highlighting its distinctive and specialized skill requirements.Most current MLLMs can only process low-resolution images, typically 448×448. However, the screenshots in VisualWebBench are captured in high resolution (1280 pixels in width), presenting challenges in identifying intricate details at lower reso- lutions. In this section, we explore the effects of input resolution on model performance.We plot the relation of max input image resolution and VisualWebBench scores for different MLLMs in Figure 8 (a). Notably, MLLMs with higher input resolution generally achieve higher scores. For instance, DeepSeek-VL with 1024×1024 resolution achieves a higher score than Qwen-VL with 448×448 resolution.Based on LLaVA-1.6 series models, we further conduct a formal ablation study on input image resolution. As depicted in Figure 8 (b), a significant performance improvement is observed as input image resolutions increase for all three model sizes. Additionally, the models exhibit greater benefits when increasing resolution from 336 to 448, compared with from 448 to 672. This finding suggests that, for LLaVA-1.6, a resolution of 448×448 stands as the minimal requirement to achieve adequate performance in web-related tasks.In our experiments in Section 4.2, for Element and Action Grounding tasks, we provide eight candidate elements and use a multiple-choice setting to evaluate different MLLMs. However, in many applications, the screenshots of webpages cannot be annotated with candidate bounding boxes. Hence, we evaluate the grounding capability in unannotated images by framing the grounding tasks as a Referring Expression Comprehension (REC) problem, where the MLLMs must generate the position (bounding box [x 1 , y 1 , x 2 , y 2 ] or central point coordinate [x, y]) of the selected HTML element. For the setting of the bounding box, we follow the standard REC task and use AP 50 (Lin et al., 2014) as the metric. For the setting of point prediction, a predicted point is regarded as correct if it falls into the true bounding box.As Table 2 shows, GUI agent MLLMs significantly outperform general MLLMs (e.g., LLaVA-1.6 and GPT-4V) in generating the positions (Bbox or point) of target elements, confirming the efficacy of grounding pre-training through the point or bounding box prediction. For other MLLMs that have been trained on general grounding data like RefCOCO, they still fail to accurately give the coordinates of the correct elements.We show a few case studies for LLaVA-1.6-34B, CogAgent, and GPT-4V on action prediction and action grounding tasks. For the action prediction task (Figure 9 ), CogAgent generates a wrong choice without any explanation, while LLaVA locates a wrong element ("Go" button). Notably, GPT-4V shows a reasonable thinking process and the correct answer. For action grounding (Figure 10 ), despite LLaVA and GPT-4V generating reasonable thought processes, all three models fail to answer correctly. See Appendix F for more case studies. Table 2 : Three evaluation settings for grounding tasks. "Multi-Choice" is the default setting in VisualWebBench. "Bbox" and "Point" denote the setting of predicting the coordinate of the target bounding box and central point, respectively.Thought: The red rectangle bounding box is located on the "GO" button of the "Advanced Search" feature. Answer: EAThought: The bounding box appears to be around a button or link, which is likely to be the "Advanced Search" button given its position on the page.  
conclusion: 
 In this work, we introduce VisualWebBench: a comprehensive benchmark to evaluate the web page understanding and grounding capabilities of MLLMs. VisualWebBench encompasses seven tasks spanning three different levels covering web page, element, and user action. Unlike existing benchmarks, our benchmark aims to comprehensively evaluate MLLMs in web contexts, including understanding, OCR, grounding, and reasoning. Our evaluation of 14 open-source MLLMs, Gemini Pro, Claude Sonnet, Claude Opus, and GPT-4V(ision) shows the substantial challenges posed by realistic web tasks. Further analysis highlights several limitations of current MLLMs, including inadequate grounding in text-rich environments and subpar performance with low-resolution image inputs. We believe VisualWebBench will serve as a catalyst for further exploration in the development of MLLMs towards artificial general intelligence. 
 
 Appendix: For the WebQA task, two co-authors carefully examined each screenshot and formulated questions and corresponding answers. For the Action Grounding task, the annotation process was carried out by seven experienced researchers, including four co-authors. The construction processes for the other subtasks were automated through program execution and were relatively straightforward.All samples of our benchmark undergo careful verification and curation through a collaborative effort and a division of tasks by two authors. The process encompasses:• Ensuring the main content within screenshots remains unobscured by advertisements or intrusive banners. • Verifying that the meta descriptions describe the most important information generally of the websites. • Headings of websites are correctly extracted for Heading OCR.• The annotated bounding boxes properly encapsulate the target web element description for Element OCR and Element Grounding. • The annotated bounding boxes are well aligned with the title of redirected websites for Action Prediction. • The instructions are appropriately matched with their annotated bounding boxes for Action Grounding.We developed an annotation tool to facilitate the annotation of the action grounding task. The annotation procedure is as follows:1. Learn about what the shown website is for, based on the presented website descriptions, and you may still need to search for the website name in Google to have a better understanding. 2. Refer to action description examples generated by GPT-4V, and then write your instruction. Then, click "Confirm instruction". Please make your instructions diverse, and do not write too many instructions like "search for an item". 3. Move the Mouse to hover over the corresponding element that will be interacted with to accomplish the action description, then press key "s" (instead of CLICK) to select it. After that, a green rectangle will be shown to indicate the selected element. Note that the element should be interactive (e.g., clickable or inputtable, etc. ). 4. Confirm that the selected element (indicated by a blinking green rectangle) correctly corresponds to the action description and click the "submit" button, then click "allow" to allow screen capture.We consider various general large multimodal models. By default, for each model family, we use the latest, largest, and best-performing available checkpoint to date. (i) BLIP-2 (Li et al., 2023c) series bridges the vision-language modality gap with a lightweight Q-Former.(ii) InstructBLIP (Dai et al., 2024) further performs vision-language instruction tuning based BLIP-2 models. (iii) mPLUG-Owl2 (Ye et al., 2023) For all MLLMs, we set the temperature to 0.0 for deterministic generations. All experiments are conducted on NVIDIA A100 80G GPUs.Table 3 details the scores of MLLMs on VisualWebBench and Mind2Web.In Figure 13 , CogAgent cannot understand the instruction correctly and output unrelated content. LLaVA-34b captures the instruction, yet generates a wrong answer. In contrast, Instruction: Generate a meta description for the webpage.Answer: Shop at Aircraft Spruce for a vast selection of certified and homebuilt aircraft parts, avionics, pilot supplies, and more. Find everything you need for your aviation needs in one place.GPT-4V (0.355): Explore a wide range of aircraft parts and pilot supplies at the aviation superstore. Find avionics, engine parts, hardware, and more for your aircraft needs. Stay updated with our weekly newsletter for new product announcements, promotions, and discounts. https://www.similarweb.comhttps://github.com/microsoft/playwrightThe overall score on the validation set of MMMU is used for comparison.Detailed experimental results are included in Appendix E. 


Decision: This paper introduces a new benchmark for evaluating multimodal language models. All reviewers found the paper to be useful and of high quality. The authors are highly encouraged to implement the valuable feedback given by the reviewers since this will likely contribute to further improve the paper.

Reviews: review: paper_summary: paper_summary: This work presents “WebBench,” a new benchmark for evaluating multimodal language models (e.g., image, text → text), specifically focusing on applications involving reasoning over webpages. The benchmark consists of ~1.5K example prompts (webpage/input/response tuple) spanning 139 individual websites, and **7 task families** each with their own evaluation metric:

1. *Captioning* (ROUGE Score): Given a webpage screenshot, generate a natural language summary of the website (where ground-truth is a GPT-4V rewrite of the HTML “meta” tag)
2. *WebQA* (F1 Score): Given a question about the webpage (written by a human annotator) generate the answer (open-set generation). 
3. *Heading OCR* (ROUGE Score): Generate the heading text for the given webpage (extracted from the first <h1> tag).
4. *Element OCR* (ROUGE Score): Given a webpage with a highlighted DOM element > 20 words (drawn bounding box), transcribe the given element in natural language.
5. *Element Grounding* (Multiple Choice Accuracy): Given a human-provided description of an HTML DOM element, and 8 potential candidates (drawn bounding boxes), select the candidate element that best matches the description.
6. *Action Prediction* (Multiple Choice Accuracy): Given a highlighted DOM element (with an href tag), select the title of the “target” webpage given a set of candidates (with negatives scraped from other elements of the same webpage).
7. *Action Grounding* (Multiple Choice Accuracy): Given a human-provided high-level instruction (e.g., “search for flights to Montreal”) and eight candidate DOM elements (drawn bounding boxes), identify the candidate element that is most relevant to the instruction (e.g., search bar for “Flights”).

Beyond curating this extensive benchmark, the work also evaluates and performs a thorough analysis of 14 state-of-the-art multimodal language models (14 open-source, 2 closed-source) across all tasks, including a stratified breakdown of how models behave as a function of scale (7B vs. 13B vs. 34B), input resolution, and manual error analysis. The appendix provides additional detail as to the data annotation and verification process. The full evaluation benchmark and evaluation harnesses for open-source models will be released.
strength_weakness: strength_weakness: There are a few minor issues that I believe the authors can easily fix for the final submission. First, it’s a bit strange to me that the evaluation metric for all of the OCR-based subtasks (e.g., Element OCR, Heading OCR) is ROUGE score rather than exact match (or F1 score); if we want high-fidelity OCR, shouldn’t we be penalizing our MLLMs for any generations that deviate from the ground-truth?

Furthermore, some of the details around the annotation procedure are missing (how annotators are selected, which annotators are used for generating which data per subtask, which data the authors of the work actually verified vs. external annotators). It would be really great to be specific and clear about these details in the final version of the paper.
questions: questions: Were external annotators only used for the action grounding task? If not, could you provide a description of the annotation pipeline used for the other subtasks (similar to Appendix A)?

Beyond the webpage screenshots and input prompts provided in the benchmark, can you clarify what other metadata about the webpages you plan to release? Minimally, it would be great to get original URLs, access dates, as well as code for rendering websites (e.g., the Playwright pipeline described in the paper).
paper_summary: paper_summary: This work presents WebBench, a new webpage understanding benchmark of just over 1.5k samples. The benchmark is designed to target seven different web page related tasks: captioning, webQA, heading OCR, element OCR, element grounding, action prediction and action grounding. Using the new dataset, 18 different model variants (of 16 unique model architectures) are evaluated. Along with the quantitative task evals, there is substantial analysis, including correlation analysis among the tasks themselves, as well as the correlation between other vision-language scores with WebBench aggregate scores and the relationship between WebBench scores and image resolution size. The work also includes a couple of qualitative examples of failure cases and ablation studies for grounding.

This paper presents high quality work with an appropriate amount of analysis and discussion. Its thorough benchmarking across many model variants is also appreciated. While the benchmark is not very large, this topic is a growing research area and having additional evaluation resources would positively impact the community.
strength_weakness: strength_weakness: 1. While more resources are certainly useful in this topic area, there are captioning and grounding datasets that would capture several of the tasks proposed in WebBench; either from prior human annotation efforts or automatically synthesized datasets. From the related work discussion and examples, it is not clear to me what distinguishes the dataset from prior work.

[1] Yao et al. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. NeurIPS 2022.

[2] Burns et al. A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding. EMNLP 2023.

[3] Srinivasan et al. WIT: Wikipedia-Based Image Text Dataset for Multimodal Multilingual Machine Learning. ACM SIGIR 2021.

2. Noise in task annotations. In the construction of the element grounding task, it's highly probable that there is noise in the annotations. For example, are there elements selected as negatives that could be congruent with the target element in action prediction task?

3. Another drawback is that it seems the tasks are annotated on separate samples, and ultimately there aren’t that many samples per task. This makes the benchmark a bit less well motivated because there are other datasets that cover the same task, and it would have been nice to have annotations for many tasks on a single screen, instead of them being non-overlapping / disjoint.
questions: questions: Questions
1. How is the correlation between tasks in Figure 5 calculated exactly?

Below are comments/suggestions for the authors
1. As mentioned in the reasons to reject, there are a couple of highly relevant related work that I would suggest adding to the prior work/citations:

[1] Yao et al. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. NeurIPS 2022.

[2] Burns et al. A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding. EMNLP 2023.

[3] Srinivasan et al. WIT: Wikipedia-Based Image Text Dataset for Multimodal Multilingual Machine Learning. ACM SIGIR 2021.

2. The last sentence in Figure 3 seems unnecessary, it was probably from an earlier version of the draft (I'd suggest removing).
3. Figure 2 is never referenced in text.
paper_summary: paper_summary: The paper proposes WebBench, a fine-grained benchmark for webpage understanding. Specifically, the paper argues that existing web benchmarks for multimodal LLMs are focused on end-to-end or agent capabilities, conflating multiple challenges in one task and thus hard to pin down what is truly challenging for a tested model.

WebBench has 1.5k samples from 39 websites.  It breaks down web understanding into 3 levels: (1) website (2) element and (3) action. Under each level, there are elementary skills needed such as HeadOCR and Captioning for (1) and grounding for (2).

The authors evaluated both closed-source and open-source multimodal LLMs on WebBench and found even the SOTA model (GPT-4V) achieves only ~65%, indicating room for improvement.
strength_weakness: strength_weakness: I do not have a major reason to reject.
paper_summary: paper_summary: The paper proposes a new evaluation dataset called WebBench that tries to cover the issues of existing benchmarks being: failing to capture the unique features of web pages and to measure fine grained capabilities such as OCR, understanding and grounding.

WebBench consists of 1.5k instances covering seven tasks: captioning, webpage QA, heading OCR, element OCR, element grounding, action prediction and action grounding.

The authors evaluate 14 open-source MLLMs covering closed and open source models.

The benchmark is constructed using different processes depending on the task. Some use LLMs and human verification, others are completely carried out by human annotators.
strength_weakness: strength_weakness: There is one issue in the way the evaluation data is produced for one task, namely captioning. In this task, metadata that is mostly a sequence of keywords is converted into a more natural description with GPT-4V. In addition, the latter is seeing both screenshot and metadata. At evaluation time, each model is asked to generate the description. This whole process gives an unfair advantage to GPT-4V, which is exacerbated by using ROUGE-L as the evaluation metric for open-ended tasks. It is fair to say that gold and predicted descriptions would exhibit similar structure and wording. This raises a question about how much GPT-4V relies on the keywords to output the description, and how much information is actually coming from the screenshot. Maybe the model relies more on the image?

On a similar note, the appendix shows that action descriptions are also suggested to annotators by GPT-4V. What if the model is generating and biasing the annotators towards actions that it can successfully perform? Would this possibility conflate GPT-4V performance on this task too?
questions: questions: Is it correct saying that the number of unique screenshots is 139?

Why is a discussion / related work on methods that relies on the DOM/source code to carry out tasks on web pages not in the paper?
E.g. DocLLM: A layout-aware generative language model for multimodal document understanding

For the heading OCR task, how often the H1 header is the right one?

Why is Gemini Pro missing from Figure 4 and 6?

What about using model-based metrics or critics for evaluating the open ended generation tasks?

Comments: title: Thank you!
Thank you!
Post-Rebuttal Response
comment: Thank you so much for your positive feedback! We sincerely express our gratitude for your constructive comments and suggestions!

Best,
Authors
Thank you for your replies and for updating the figures. I think the current score captures my excitement about the paper.
Thank you for your positive feedback and enthusiasm about the paper! Your strong support for accepting the paper is much appreciated. Thank you again for taking the time to review the paper thoroughly and provide constructive comments!

Best,
Authors
Thank you for the explanation of the ROUGE-L vs. F1 metrics; I think this is an important discussion, and I'm glad to see these results. I'm also very excited that data and rendering code will be open-sourced!

This was a very strong paper to begin with, and my opinion hasn't changed; I definitely will be pushing for acceptance at this time!