Title: WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks
Benchmark: WorkArena++
Summary: WorkArena++ is a benchmark building on top of WorkArena with 682 tasks that simulate realistic workflows for knowledge workers using the ServiceNow platform. It aims to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding of web agents, highlighting current challenges for AI in workplace automation compared to human performance.  The benchmark also provides a method for generating large datasets of observation/action traces for fine-tuning models.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: "planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual
understanding abilities of web agents"
Phenomenon defined by authors: Yes
Phenomenon definition: "evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents." (abstract)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: An agent is placed in a live ServiceNow instance (via BrowserGym) and must complete an assigned workflow, where success is judged by an automatic validator that inspects the UI state and underlying database. ​
Task item definition: Each task in WorkArena++ consists of a logical combination of simpler tasks chained together to form a realistic workflow.  Tasks are presented at two difficulty levels, L2 (explicit instructions) and L3 (instructions via ticket and knowledge base), and success validator.
Task definition detail: L2 and L3 are main elements that were not present in original WorkArena.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 682
Dataset size detail: There is even number task in L2 and L3 (342 workflows)
Dataset size notes: Paper mentions a human curriculum of 98 task instances and an agent curriculum with 235 tasks for each L2 and L3 difficulty.
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Extended interaction (e.g. conversation, calling an API and processing the response)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean and standard error
Metric subscores: Yes
Metric subscores detail: Subset scores are provided separately for the difficulty levels (L2 and L3) and across the five skill categories.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Human baseline confirms tasks are solvable and error analysis details where agents fail.
Taxonomy: Agents / Web
New taxonomy: General Application / Web Agents
Procedural notes: Both WorkArena (from ICML 2024) and WorkArena++ (NeurIps 2024) share the same github link.