# Why a New Benchmark? — Gap Analysis

> V5.15.h.3 artefact — JOT submission readiness.
>
> One-page argumentative summary of why the public benchmarks reviewed in
> `related_benchmarks_comparison.md` cannot be substituted for
> mssense-eval-benchmark v1.1.

## The simultaneous-properties argument

The problem of **closed-vocabulary action trace generation for
conversational RPA authoring** is operationally defined by the **joint**
satisfaction of four properties:

1. The output vocabulary is a **closed catalogue of typed actions tied to
   an executable runtime**, organised by channel (WEB_DOM, EXCEL,
   DESKTOP_UIA, FILE_IO, CONTROL, MAIL, API_REST, DATABASE, …). A
   method's output must resolve in this catalogue or fail at runtime.
2. The input is **conversational**, with possibly implicit references,
   multi-turn ambiguity, and the requirement to decide between
   generating, asking, validating, or closing the conversation
   (`expected_decision ∈ {INVALID, VALID, UNDERSTOOD, QUESTION, THINKING,
   DONE}`).
3. The expected output is a **durable multi-step action trace**, not a
   single tool call. Correctness propagates across step dependencies
   (variables produced and consumed, control-flow blocks, scoped
   contexts).
4. The input modality covers **text, voice transcript, screenshot, demo
   recording, and mixed combinations**.

A public benchmark substitutes only if it satisfies all four. The audit
in `related_benchmarks_comparison.md` shows that **none does**.

## The disjunction of partial substitutes

Each related corpus covers a strict subset:

- **WONDERBREAD** covers (4) and parts of (2) but lacks (1) (no closed
  RPA catalogue — its action trace is a low-level GUI event log) and
  lacks (3) in the typed-trace sense (its expected output is a free-form
  SOP).
- **WorkArena / WorkArena++** cover (3) on web but neither (2)
  (single-task spec, no conversation) nor (4) (web-only) nor (1) (web
  HTML actions, not RPA platform actions).
- **Mind2Web / Multimodal-Mind2Web** cover (3) on web, (4) partially
  with screenshots, but neither (1) nor (2).
- **WebArena / OSWorld** cover live agent execution, not authoring of a
  persistent artefact — orthogonal to (3) as defined here.
- **JSONSchemaBench / XGrammar** cover only the schema layer; the
  manuscript Section 1 explicitly argues that schema validity is
  necessary but **insufficient**.
- **Toolformer / ReAct / APIBench / AgentBench** target single-step or
  short-sequence tool calling, not multi-step authoring of an
  executable trace.
- **FlowMind / SmartFlow** are the closest in spirit but **release no
  public evaluation corpus** — both papers report on proprietary
  internal datasets.

The disjunction of any subset of these still leaves at least one of
(1)–(4) uncovered.

## The reuse-and-extend argument

mssense does **not** ignore the existing corpora. The
`mssense_public_creation_wonderbread_v1_full` sub-corpus reuses
WONDERBREAD's intent + action trace + SOP fields after a documented
transformation pipeline (internal, not distributed). The transformation
maps WONDERBREAD's open-vocabulary GUI events to the closed RPA action
catalogue, attaching the canonical fields (`task_family`,
`channel_family`, `expected_decision`, `oracle`) that the manuscript's
evaluation requires.

This reuse demonstrates that we are **not rebuilding from scratch what
already exists** — we are **augmenting** what exists with the missing
labels and structure, and we are **adding** the internally-authored
audit and validation cases that no public corpus contains.

## What a reviewer should walk away with

The manuscript is not making the strong claim "no relevant public
benchmark exists." It is making the precise claim **"no public benchmark
simultaneously satisfies the four properties that the proposed problem
formally requires."** The strong form is empirically falsifiable, the
precise form is the actual contribution.

A reviewer convinced that one of WONDERBREAD / WorkArena / Mind2Web is
sufficient is invited to indicate which of the four properties they
contest. The table in `related_benchmarks_comparison.md` provides a
testable per-property verdict for each candidate.

## Reference for the in-text paragraph

A drop-in paragraph for Section 2.6 of the manuscript is in
`manuscript_section_2_6.md`.