# Datasheet for mssense Evaluation Benchmark

Following the *Datasheet for Datasets* template (Gebru et al., 2018, arXiv:1803.09010).
Companion to the lighter [dataset_card.md](dataset_card.md) and the
[evaluation_protocol.md](evaluation_protocol.md). Platform-specific terms
(`mssense`, `LAT`, `LeBrain`, `IRIS`, `Intentia`) are defined in the
**Terminology** table of the [README](../README.md).

## Motivation

**For what purpose was the dataset created?**
To provide an evaluation suite for `mssense` — the conversational
intent-understanding and workflow-validation component developed within the
Intentia research programme (Novelis R&D, 2026). Tasks: clarification policy,
LAT audit, semantic judgment, workflow creation, business-rule extraction,
visual grounding / governance and modification-intent handling. The benchmark
covers the heterogeneity of inputs (text instructions, audio transcripts, screen
captures, demo recordings) and outputs (`UNDERSTOOD`, `VALID`, `INVALID`,
`QUESTION`, `THINKING`, `DONE`).

The systems it evaluates are two method variants of `mssense`: **HCA-WA**
(Hybrid Checkpointed Agent for Workflow Authoring) and **SGAR-WA**
(Skill-Grounded Agentic Reasoning for Workflow Authoring, the current default).

**Who created the dataset and on behalf of which entity?**
Authored within the Intentia research programme (Novelis R&D). The `mssense_public_creation_wonderbread_v1_full`
sub-corpus is derived from the public WONDERBREAD dataset (see Provenance
in [LICENSE-DATA](../LICENSE-DATA)).

**Who funded the creation of the dataset?**
Internal R&D budget.

## Composition

**What do the instances comprising the dataset represent?**
Each instance represents one mssense-flavoured task: validate a LAT,
audit a candidate LAT for ambiguity, generate a workflow from a description,
extract a business rule from a free-form prompt, or judge a modification
intent. The unit of an instance is one JSONL record carrying:
- `benchmark_version`
- `input_payload` (text, audio_url, capture artifact reference, demo reference)
- `lat_candidate` (the LAT under analysis, when relevant)
- `business_rules` (BR list, when relevant)
- `channel_family` (`web` | `desktop` | `other`)
- `input_modality` (`text` | `audio` | `capture` | `demo`)
- `expected_decision` (`VALID` | `INVALID` | `UNDERSTOOD` | `QUESTION` |
  `THINKING` | `DONE`)
- `expected_audit_question` (when `expected_decision == INVALID`)
- `expected_issue_types` (list — `AMBIGUOUS_SELECTOR`, `MISSING_PRECONDITION`,
  `UNRESOLVED_VARIABLE`, `INCONSISTENT_FLOW`, `MISSING_VALUE`)
- `expected_choices`
- `oracle` (machine-readable ground truth)
- `provenance` (block, origin_sample_id, source)
- `difficulty`, `content_hash`

**How many instances are there in total?**
- v1.1-eval: **1865 evaluable samples** in
  `data/mssense_eval_benchmark_v1_1.jsonl` (this release).
- v1.0-eval: 1772 samples (superseded; see
  [`../reports/CHANGELOG_v1.0_to_v1.1.md`](../reports/CHANGELOG_v1.0_to_v1.1.md)).

No train/dev/test split.

**Does the dataset contain all possible instances or is it a sample?**
A curated sample union of three sub-corpora:
- `mssense_public_audit_internal_v1*` — internal-origin audit cases
- `mssense_public_validation_internal_v1*` — internal-origin validation cases
- `mssense_public_creation_wonderbread_v1_full*` — WONDERBREAD-derived
  creation/demo cases

**What data does each instance consist of?**
One JSONL record in `data/mssense_eval_benchmark_v1_1.jsonl`, schema validated
by `schema/evaluation_sample.v1_1.schema.json`. Cross-references to upstream
WONDERBREAD ids are preserved in `provenance.origin_sample_id` when applicable.
No PII; all names are synthetic.

**Are there recommended data splits?**
No. Pure evaluation suite. Reports are stratified by `task_family`,
`channel_family`, `input_modality`, `expected_decision` and `difficulty`.

**Are there any errors, sources of noise, or redundancies in the dataset?**
- Multiple `expected_decision` codes coexist; the most frequent are `INVALID`,
  `VALID`, `UNDERSTOOD`, `QUESTION`. Methods that emit only a subset of these
  labels must document how they normalize.
- WONDERBREAD-derived samples carry a more permissive ambiguity profile than
  internal-origin samples; stratified reporting captures this.

**Is the dataset self-contained, or does it link to or otherwise rely on external resources?**
Self-contained for the text / metadata path. Audio / capture / demo modalities
reference relative artifact paths under `data/`; the original WONDERBREAD
videos and key frames are not republished here (license-restricted) and must
be obtained from the upstream dataset if needed for grounding studies.

## Collection process

**How was the data acquired?**
- **Internal audit / validation sub-corpora**: authored in-house by domain
  experts using a structured authoring rubric.
- **WONDERBREAD-derived creation sub-corpus**: extracted programmatically by an
  internal pipeline that reads upstream `metadata.json + intent + action_trace +
  SOP` and emits records conforming to the canonical schema
  (`schema/evaluation_sample.v1_1.schema.json`).

**What mechanisms or procedures were used to collect the data?**
- Internal authoring tools producing JSONL.
- A Python extractor for the WONDERBREAD-derived sub-corpus (internal, not
  distributed).
- Manual review of hard cases against a publication-readiness checklist.

**Over what timeframe was the data collected?**
Authoring + extraction: April–June 2026.

**Were any ethical review processes conducted?**
Not formally. Internal review focused on (a) absence of PII in
audit/validation cases, (b) compliance with WONDERBREAD's upstream license
for the creation sub-corpus, (c) no real customer data exposed.

## Preprocessing / cleaning / labeling

**Was any preprocessing/cleaning/labeling done?**
- Label canonicalisation: upstream WONDERBREAD `intent` is mapped to the
  mssense expected-decision taxonomy.
- LAT extraction: `action_trace` is normalised into the LAT shape
  (`{id, channel, action, params, …}`).
- Content hashing per record (`content_hash`).
- Hard-case adjudication: a subset of high-difficulty WONDERBREAD samples
  was manually reviewed for label correctness.

**Was the "raw" data saved in addition to the preprocessed/cleaned/labeled data?**
The extractor inputs and intermediate manifests are retained internally for
reproducibility; they are not vendored here. The original WONDERBREAD media is
not republished either (see LICENSE-DATA).

## Uses

**Has the dataset been used for any tasks already?**
- Internal `mssense` regression evaluation across successive method versions.
- End-to-end approach sweeps over the `mssense` HTTP surface.

**Is there a repository that links to any or all papers or systems that use the dataset?**
This dataset is the evaluation suite associated with the manuscript
*Closed-Vocabulary Action Trace Generation for Conversational RPA Authoring*
(Yahaya Alassan, Ettifouri, Dahhane; Novelis), submitted to the
*Journal of Object Technology*.

**What (other) tasks could the dataset be used for?**
Any evaluation of clarification-policy agents, structured-question
generation, LAT auditing methods, or workflow-creation systems on a
multi-modal corpus.

**Are there tasks for which the dataset should not be used?**
- Training a model from scratch (evaluation-only by design).
- Republishing WONDERBREAD images / videos via this benchmark (the original
  upstream license governs those artifacts).

## Why a New Benchmark? (V5.15.h.3)

A reviewer of this dataset will reasonably ask whether an existing public
corpus could substitute for it. The detailed answer is in
[`related_benchmarks_comparison.md`](related_benchmarks_comparison.md)
(twelve candidate public datasets analysed property-by-property) and
[`why_new_benchmark.md`](why_new_benchmark.md) (one-page gap analysis).
In short: the problem requires four properties — (i) closed-vocabulary
RPA actions, (ii) conversational input, (iii) multi-step durable trace,
(iv) multi-modal coverage — and no public corpus satisfies all four
simultaneously. WONDERBREAD is reused as the upstream source of the
`workflow_creation` sub-corpus but cannot substitute for the whole.

## Statistical power and inter-annotator agreement (V5.15.h.3)

- A priori statistical-power analysis per research question is in
  [`statistical_power_analysis.md`](statistical_power_analysis.md). The
  full corpus supports detection of 5–10 % absolute effect-size
  differences at α = 0.05, power = 0.80, on the four RQ-relevant
  comparisons.
- Inter-annotator agreement is **not applicable** to 1593 of the 1865
  samples (the controlled-balanced and augmented blocks) by
  construction; the remaining 272 internally-authored seeds followed a
  documented rubric with hard-case team review. See
  [`inter_annotator_agreement.md`](inter_annotator_agreement.md). A
  formal κ measurement on a stratified subset is planned for the v1.2
  release.

## Distribution

**Will the dataset be distributed to third parties outside of the entity on behalf of which it was created?**
Yes for the internal-origin sub-corpora under CC-BY-4.0. Distribution of the
WONDERBREAD-derived sub-corpus is conditioned on attribution to WONDERBREAD
upstream.

**How will the dataset be distributed?**
JSONL + schemas + harness + reports + manifest, with the LICENSE-DATA
provenance clause attached. Hosting channel TBD.

**Will the dataset be distributed under a copyright or other intellectual property (IP) license?**
- Code, scripts, schema: Apache-2.0 (`LICENSE`).
- Internal-origin data: CC-BY-4.0 (`LICENSE-DATA`).
- WONDERBREAD-derived sub-corpus: CC-BY-4.0 + cumulative attribution to
  WONDERBREAD upstream.

**Have any third parties imposed IP-based or other restrictions on the data associated with the instances?**
The WONDERBREAD upstream license imposes attribution; this benchmark complies
by (a) explicit citation in LICENSE-DATA, (b) not republishing upstream media.

## Maintenance

**Who will be supporting/hosting/maintaining the dataset?**
The Intentia research programme (Novelis R&D). The dataset is hosted at its
DOI landing page (`https://doi.org/10.5281/zenodo.21105006`).

**Will the dataset be updated?**
Yes. v1.0-eval and v1.1-eval are the first published milestones; changes are
tracked in [`../reports/CHANGELOG_v1.0_to_v1.1.md`](../reports/CHANGELOG_v1.0_to_v1.1.md).

**Will older versions of the dataset continue to be supported/hosted/maintained?**
Yes — each versioned release is preserved under its own DOI on the hosting
platform.

**If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?**
Contributions are welcome via the dataset's DOI landing page. Extensions must
respect the canonical schema (`schema/evaluation_sample.v1_1.schema.json`) and
document any new label codes.