Title: WorkArena: How Capable are Web Agents at Solving Common Knowledge Work Tasks?
Benchmark: WorkArena
Summary: WorkArena introduces a benchmark of 33 enterprise software tasks using the ServiceNow platform to evaluate LLM-powered web agents. Alongside it, the authors present BrowserGym - a multimodal environment for training and assessing agents on web-based workflows.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: web navigation,  Instruction Following, reasoning, natural language understanding of long context, 
Phenomenon defined by authors: Yes
Phenomenon definition: "the agents’ ability to perform tasks that span the typical daily work of knowledge workers utilizing enterprise software systems" (abstract)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Each task requires the agent to complete a real-world workflow action within the ServiceNow UI, such as filtering a list, submitting a form, or answering a knowledge base query, based on an explicit natural language goal.
Task item definition: "In WorkArena, each task is coupled with a natural language goal that provides instructions to the agent. Each goal is automatically generated from a human-designed template filled with pre-defined values (menu name, field value, item specifications etc.), and explicitely provides all the information required to solve the task" (page 3)
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: Though set up in developer instances, the tasks mirror real usage patterns on enterprise platforms like ServiceNow used daily by knowledge workers.
Dataset size: 19,912
Dataset size detail: Each of the 33 tasks has 10–1,000 instances; some are capped for feasibility.
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Extended interaction (e.g. conversation, calling an API and processing the response)
Response format detail: Responses are Python-based actions and/or structured commands (e.g., click, fill), often embedded in a dialogue-based interface.
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Mean and standard error, with bootstrap confidence intervals
Metric subscores: Yes
Metric subscores detail: Subscores provided by task category (e.g., forms, lists, dashboards).
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors argue that benchmark reflects real-world enterprise workflows and UI challenges, making it suitable for assessing LLM-based web agents. 
Taxonomy: Agents / Web
New taxonomy: General Application / Web Agents
Other notes: The definition of a task is a bit misleading in WorkBench. The abstract states that "WorkArena consists of a suite of 33 tasks and 19,912 unique instances", but as we read later in the paper, actually the unique instances are closer to our definition of a task and 33 is the number of type/categories.