Title: VisualWebArena: Evaluating Multimodal Agents on Realistic Visually
Grounded Web Tasks
Benchmark: VisualWebArena
Summary: VisualWebArena is a benchmark designed to assess the performance of multimodal agents on diverse and complex web-based tasks (requiring reasoning and visual understanding) that evaluate various capabilities of autonomous multimodal agents in three different environments (Classifieds, Shopping, and Reddit). Paper also introduces new agent design using Set-of-Marks (SoM) prompting to simplify the visual action space and improve performance.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multimodal reasoning, web-based action execution
Phenomenon defined by authors: Yes
Phenomenon definition: "To perform well, agents need to accurately process image-text inputs, interpret natural language instructions, and execute actions on websites to accomplish user-defined objectives." (page 1)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Task definition: Tasks are divided into two types: Information seeking tasks, which expect a string output from the model, and navigation and actions tasks, which require navigating through multiple webpages, and executing actions to change the underlying state s of the environment.
Overall the task requires an autonomous agent to complete visually grounded objectives on web interfaces, such as ordering items, making posts, or navigating forums, using both visual and textual information.
Task item definition: Webpage / Input Image(s), Example Intent, Reward Function Implementation
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: Tasks were created by 6 graduate student co-authors using a templating method and realistic visual web environments; input images came from MS-COCO and royalty-free sources.
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 910 newly created tasks
Dataset metadata available: Yes
Dataset metadata detail: task complexity, human performance, evaluation metrics (exact_match, must_include, fuzzy_match, must_exclude, eval_vqa, • eval_fuzzy_image_match), task type (e.g., OCR, exact image match), task domain (site), presence of input images.


Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Extended interaction (e.g. conversation, calling an API and processing the response), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Custom reward functions (e.g. must_include, eval_vqa, eval_fuzzy_image_match)
Metric aggregation: Simple Mean
Metric statistics: Simple mean success rates across tasks and subsets. There is no formal hypothesis testing or statistical significance tests.
Metric subscores: Yes
Metric subscores detail: OCR Tasks: OCR required vs No OCR required
Exact Image Match: Exact image match vs No exact image match
Image Input Tasks: Image inputs vs No image inputs
Model type: Text-only, Caption-augmented, Multimodal, Multimodal (SoM)
Source type: Reddit, Classifieds, Shopping, Overall
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: It is publicly available, but the paper itself does not provide the link.
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: The authors argue the benchmark is valid by designing tasks that resemble real web interactions and require visual understanding, supported by human baselines and task diversity. However, the use of self-hosted websites may reduce ecological validity, and the composite nature of the tasks makes it hard to isolate specific capabilities. No formal construct validity analysis is provided beyond success rates and subset performance.
Taxonomy: Agents / Web
New taxonomy: General Application / Web Agents