Title: ASSISTANTBENCH: Can Web Agents Solve Realistic and Time-Consuming Tasks?
Benchmark: AssistantBench
Summary: This paper introduces AssistantBench - benchmark designed to evaluate whether web agents can solve realistic and time-consuming web-based tasks, such as finding gym schedules or real-estate prices. It also presents SPA (See-Plan-Act), a new web agent for more effective open-web navigation.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications), Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: web-based information-seeking
Phenomenon defined by authors: Yes
Phenomenon definition: "We examine whether such agents can perform realistic and time-consuming tasks on the web, e.g., monitoring real-estate markets or locating relevant nearby businesses.”
“Tasks in ASSISTANTBENCH are based on real information needs encountered by humans. To solve these tasks, an agent must autonomously browse the web to identify relevant web pages and dynamically interact with them to produce an output."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Purpose notes: The distinction between specific application and general capability is tricky because it evaluates general capabilities like planning, retrieval, and reasoning - but within the specific context of realistic web-based tasks. While the underlying skills are broadly applicable, the benchmark is grounded in application setting (open-web agenting), making it hard to tell whether it’s measuring general abilities or just how well those abilities transfer to this use case (which is also quite broad).
Task definition: The task is to evaluate whether language agents can autonomously complete realistic, time-consuming web-based information-seeking tasks by browsing, interacting with, and synthesising content from multiple websites.
Task item definition: user-like query (e.g., "Which gym near X has classes before 7am?"), gold answer, relevant URLs where the answer can be found, and a step-by-step explanation of how to solve it.
Task definition detail: They evaluate two types of tasks: more broad and less experienced (like real estate, travel, fitness, education etc.), and created by experts in fields as biology and law.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 214
Dataset size notes: validation: 33
Dataset metadata available: Yes
Dataset metadata detail: difficulty level (easy, medium, hard), time-dependency class (static, stable, time-sensitive)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Validation
Response format: Free response (e.g. summary paragraph, executable code), Extended interaction (e.g. conversation, calling an API and processing the response), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: Simple mean and standard error of the mean (for plots like accuracy vs. steps)
Metric subscores: Yes
Metric subscores detail: Task difficulty (easy, medium, hard), task source (seed, crowd, expert), and time-dependency class (static, stable, likely-to-change).
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors manually filter tasks to remove fast-changing or ambiguous cases, structure answers for auto-evaluation, and analyse error types (navigation, hallucination, grounding). They also categorise tasks by difficulty (based on model success) and by time-dependency (static, stable, volatile) to support the validity of benchmark in measuring its intended phenomenon.
Taxonomy: Agents / Web
New taxonomy: General Application / Web Agents