# Related Public Benchmarks — Comparison

> V5.15.h.3 artefact — JOT submission readiness.
>
> This document compares mssense-eval-benchmark v1.1-eval against the most
> closely related public benchmarks and justifies, dataset-by-dataset, why
> no existing public corpus can substitute for it. It is the source material
> for the related-work paragraph and Section 8.1.x of the JOT manuscript
> (cf. `manuscript_section_2_6.md`).

## Comparison axes (the four required properties)

The mssense problem of **closed-vocabulary action trace generation for
conversational RPA authoring** simultaneously requires four properties on
the evaluation corpus:

1. **Closed-vocabulary actions tied to an executable RPA runtime.** Each
   action must belong to a fixed channel-specific catalogue with typed
   parameters (e.g. `WEB_DOM.LeftClick(xpath, instance, tab_id)`,
   `EXCEL.ReadWorksheetRange(workbook, sheet, range)`, `CONTROL.ForLoop`).
   A schema-only constraint is insufficient — the names and parameters must
   resolve in a real RPA runtime.
2. **Conversational input.** The user supplies natural language, possibly
   voice or screen recording, possibly multi-turn with implicit references
   ("yes, that one", "do the same for the other workflow"), possibly
   ambiguous. The benchmark must include samples that *require* a
   clarification or a verdict, not just a generation.
3. **Action trace as a durable artefact.** The expected output is a
   multi-step typed action sequence that will be persisted and later
   interpreted by a workflow runtime. This is distinct from single-step
   tool calling (one tool per turn) and from generic structured output
   (one JSON object per call): correctness extends to multi-step
   dependencies that may span dozens of actions.
4. **Multi-modal input coverage.** Real conversational RPA authoring
   blends text, voice, screenshot, and demonstration recording. A
   benchmark restricted to one modality cannot evaluate cross-modal
   robustness or modality-conditioned error rates (RQ3 of the manuscript).

A candidate public benchmark must satisfy all four to be a drop-in
substitute. The table below shows that **none does**.

## Comparison table

Legend: ✅ covered ; 🟠 partial ; ❌ absent.

| Benchmark | Year | (1) Closed-vocab RPA | (2) Conversational | (3) Multi-step trace | (4) Multi-modal | Verdict for mssense |
|---|---|---|---|---|---|---|
| **mssense-eval-benchmark v1.1** | 2026 | ✅ 9 channels, ≥50 actions | ✅ multi-turn, ambiguity, clarification labels | ✅ LAT with control flow | ✅ text/audio/capture/demo/mixed | — |
| WONDERBREAD [Wornow et al. 2024] | 2024 | ❌ SOP texts, no RPA catalogue | 🟠 single-turn intent | 🟠 SOP is procedural text, not typed trace | ✅ demo + keyframes | Source for our `creation` block only; cannot substitute |
| WorkArena [Drouin et al. 2024] | 2024 | 🟠 ServiceNow web actions | ❌ single-turn agentic task | 🟠 web agent actions, not RPA traces | ❌ web-only | Web-only, agent-oriented |
| WorkArena++ [Boisvert et al. 2024] | 2024 | 🟠 ServiceNow composed tasks | ❌ single task spec | 🟠 multi-step on web | ❌ web-only | Long-horizon web, not RPA |
| Mind2Web [Deng et al. 2023] | 2023 | ❌ generic web HTML actions | ❌ task description | 🟠 multi-step web | ❌ web-only | Generic web agents |
| Multimodal-Mind2Web [Zheng et al. 2024] | 2024 | ❌ generic web | ❌ task description | 🟠 multi-step web | 🟠 web + screenshot | Web grounding, not RPA |
| WebArena [Zhou et al. 2024] | 2024 | ❌ web agent actions | ❌ single goal | ✅ multi-step interactive | ❌ web-only | Interactive agent execution, not authoring |
| OSWorld [Xie et al. 2024] | 2024 | ❌ OS-level interactions | ❌ task instruction | ✅ long horizons | 🟠 screenshot | OS agent execution, not authoring |
| MiniWoB++ [Shi et al. 2017, Liu et al. 2018] | 2018 | ❌ tiny synthetic web actions | ❌ short task | 🟠 short trajectories | ❌ web-only | Smoke-test scale |
| RICO / RICO-Semantics [Deka et al. 2017] | 2017 | ❌ no action vocabulary | ❌ no intent | ❌ no traces | 🟠 UI screenshots | UI dataset, not workflow |
| FlowMind [Zeng et al. 2024] | 2024 | 🟠 internal workflow lang | ❌ single instruction | ✅ workflow output | ❌ text-only | Closed-source corpus; not an open eval set |
| SmartFlow [Jain et al. 2024] | 2024 | 🟠 RPA workflows | ❌ single task | ✅ workflow output | 🟠 text + demo | Proprietary; no public eval suite released |
| Toolformer [Schick et al. 2023] | 2023 | ❌ generic tool calling | ❌ single turn | ❌ single tool call | ❌ text-only | Single-step tool use |
| ReAct (HotpotQA / ALFWorld variants) [Yao et al. 2023] | 2023 | ❌ generic reasoning + acting | 🟠 multi-turn | 🟠 episode level | ❌ text-only | Generic ReAct, not RPA |
| JSONSchemaBench [Geng et al. 2025] | 2025 | ❌ JSON schema fidelity only | ❌ no conversation | ❌ structured output, not trace | ❌ schema-only | Constrained decoding eval |
| XGrammar [Dong et al. 2025] | 2025 | ❌ grammar-constrained gen | ❌ no conversation | ❌ structured output | ❌ schema-only | Decoding engine, not benchmark |
| APIBench / API-Bank [Patil et al. 2023, Li et al. 2023] | 2023 | ❌ generic APIs | 🟠 short dialog | ❌ short tool sequence | ❌ text-only | API selection, not RPA |
| AgentBench [Liu et al. 2024] | 2024 | ❌ multi-domain agent | 🟠 multi-turn | 🟠 multi-step | 🟠 some modalities | Generic agent, not RPA authoring |
| SWE-bench [Jimenez et al. 2024] | 2024 | ❌ Python repo bugs | ❌ issue description | ❌ patch generation | ❌ text-only | Software engineering, unrelated |
| XCorpus [Dietrich et al. 2017] | 2017 | ❌ Java code corpus | ❌ no conversation | ❌ programs, not traces | ❌ code only | Cited for JOT methodology precedent only |

## Why each candidate cannot substitute (per-dataset paragraph)

### WONDERBREAD (Stanford, 2024)

**Strengths.** Largest public corpus of GUI demonstrations with intent +
action trace + key frames + SOP, sourced from real human captures.

**Why it does not substitute for mssense.** WONDERBREAD's "action trace" is
a low-level GUI event sequence (clicks, keystrokes, scroll positions),
*not* an executable closed-vocabulary RPA trace. There is no fixed catalogue
of typed channel-specific actions to validate against. The SOPs are
free-form natural language, which is the *input* to the closed-vocabulary
generation problem, not its *output*. WONDERBREAD is therefore an
*upstream source* for the `workflow_creation` task family of mssense (we
extract `metadata + intent + action_trace + SOP` fields via an internal
extraction pipeline), but it cannot be used as
the evaluation target: a method that generates a WONDERBREAD-shaped output
is not testing what the manuscript claims to test.

**Citation expected.** Wornow, Riedel, Narayan, Mahbub, Ling, Yan, &
Liang, "WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation
Models on Business Process Management Tasks," 2024.

### WorkArena and WorkArena++ (ServiceNow, 2024)

**Strengths.** Real enterprise web tasks on ServiceNow, well-curated, with
end-to-end goal satisfaction signals.

**Why it does not substitute.** Both are **web-only** (single channel
family), **single-turn** task descriptions (no conversational
clarification or implicit reference resolution), and **agent-oriented**
(the agent acts on a live ServiceNow instance) rather than
**authoring-oriented** (the system produces a durable workflow artefact).
Closed-vocabulary RPA action traces are not the evaluation target.

### Mind2Web and Multimodal-Mind2Web (2023–2024)

**Strengths.** Large-scale web task corpus across many sites, with
trajectory annotations.

**Why it does not substitute.** Generic web HTML action vocabulary
(`click`, `type`, `select`) without RPA-platform binding, no closed
multi-channel catalogue (EXCEL / DESKTOP_UIA / CONTROL absent), no
conversational dialogue layer.

### WebArena and OSWorld (2024)

**Strengths.** Realistic interactive environments for agent evaluation,
WebArena on web, OSWorld on OS-level.

**Why it does not substitute.** Both evaluate agents that *execute*
workflows live, not systems that *author* persistent workflow artefacts.
A system can pass WebArena by completing a task without producing any
durable trace; mssense requires the trace to be inspected post hoc.

### MiniWoB++ (2017–2018)

**Strengths.** Reproducible micro-tasks.

**Why it does not substitute.** Toy synthetic web pages, single-turn
tasks, no closed RPA vocabulary. Suitable as a smoke test, not as a
publication-grade evaluation set.

### FlowMind (JPMorgan, 2024) and SmartFlow (TCS, 2024)

**Strengths.** Both target RPA authoring with LLMs; FlowMind generates
workflows, SmartFlow combines task descriptions and demonstrations.

**Why neither substitutes.** **The evaluation corpora are not publicly
released as open benchmarks.** Both papers report results on proprietary
internal datasets. A reviewer cannot reproduce mssense results by pointing
to either, and no open distribution accompanies the original publications.

### Constrained decoding (JSONSchemaBench, XGrammar)

**Strengths.** Important methodological foundations for the structured
output stage of the proposed pipeline.

**Why neither substitutes.** Both target **schema validity only**. The
manuscript Section 1 explicitly argues that schema validity is necessary
but insufficient — vocabulary validity and workflow executability are
distinct, higher-level correctness criteria. A method that passes
JSONSchemaBench can still emit `WEB_DOM.LeftClick` parameters that
reference a non-existent variable or call a non-existent action.

### Tool-augmented LLM agents (Toolformer, ReAct, APIBench, AgentBench)

**Strengths.** Define the *tool calling* paradigm that informs the action
selection step of the pipeline.

**Why none substitute.** Tool calling concerns the **runtime choice** of
one tool per turn. Action trace generation concerns the **off-line
authoring** of a multi-step durable artefact whose actions remain valid
after the conversation has ended. The benchmark must therefore expose
multi-step compositional structure, which AgentBench partially exposes
but without closed RPA vocabulary.

### Java program corpora (XCorpus, DaCapo, Qualitas)

**Strengths.** Cited as **methodology precedents** for the JOT-publishable
dataset-paper format (cf. XCorpus, JOT 2017, 76 programs).

**Why none substitute.** Unrelated domain (executable Java programs ≠ RPA
action traces). Listed for the orienting comparison only.

## Summary

The **four required properties simultaneously** isolate a gap that no
single public benchmark fills. WONDERBREAD covers (4) demonstrations but
not (1) closed RPA vocab; WorkArena covers (1)–(3) on web but not multi-
modal or conversational; agent benchmarks cover execution but not
authoring; constrained decoding benchmarks cover schema validity but not
vocabulary or executability. mssense-eval-benchmark v1.1 is the first
open evaluation corpus that simultaneously satisfies (1)–(4) and is
therefore the only candidate consistent with the manuscript's claim
("closed-vocabulary action trace generation for conversational RPA
authoring").

A pre-emptive paragraph for the manuscript is provided in
`manuscript_section_2_6.md` (Section 2.6 *Why a New Benchmark?*).