Title: WebArena: A Realistic Web Environment for Building Autonomous Agents
Benchmark: WebArena
Summary: WebArena introduces an open‑domain benchmark and environment where language‑guided agents must navigate fully functional websites (e.g. e‑commerce stores, forums, GitLab, maps) to satisfy natural‑language task intents in a similar to real setting.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: performing tasks on the web, natural language understanding
Phenomenon defined by authors: No
Phenomenon definition: Authors describe the environment and evaluation setup in detail, but never distil the phenomenon being measured into a single conceptual definition. They mention their goal is to create a realistic and reproducible web environment and they are interested in measuring: autonomous agents understanding and executing diverse, realistic web-based tasks given in natural language, and the benchmark's role in evaluating their success in functionally completing these tasks.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Purpose notes: While tasks fall into categories (Information Seeking, Site Navigation, Content & Config), they are evaluated with unified success metrics. However, there are separate success rate results depending on different prompting strategies.
Task definition: An agent receives a high-level natural language intent and must complete the corresponding web-based task by interacting with realistic, functional websites using browser-like actions.
Task item definition: Each task item includes a natural language order/intent (possibly from a template) and a programmatic validator to assess if the interaction successfully meets the task's goal.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The environment uses fully functional websites that mirror popular domains with real-world data. The tasks are designed to emulate routine human web activities, to be representative of real-world scenarios, yet still performed within a controlled benchmark setting.
Dataset size: 812
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Extended interaction (e.g. conversation, calling an API and processing the response)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), must_include', 'fuzzy_match', and programmatic checks which don't fit standard categories
Metric definition detail: Fuzzy_match uses GPT-4 to semantically compare the predicted answer with the correct answer as in template. For procedural tasks, success is defined through programmatic inspection of the final state (e.g. checking a repository README, verifying a post was created).
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Success Rate is broken down for achievable and unachievable tasks.
Human performance is reported separately for information-seeking tasks and all the rest.
Different models and prompting strategies (e.g. CoT vs. no-CoT) are also reported separately.
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: No
Author validity detail: The authors do not explicitly argue for construct validity. While they provide human-agent comparisons, ablation studies, and error analysis, these are primarily used to characterise performance rather than justify that the benchmark measures the intended capability.
Taxonomy: Agents / Web
New taxonomy: General Application / Web Agents
Other notes: There is no comparison of results compared to other benchmarks, but there is comparison of measured traits (e.g. Dynamic Interaction, Realistic Environment) between WebArena and other benchmarks.