# Statistical Power Analysis — mssense-eval-benchmark v1.1

> V5.15.h.3 artefact — JOT submission readiness.
>
> A priori power analysis for the four research questions declared in
> the manuscript (Section 7). Demonstrates that mssense-eval-benchmark
> v1.1 (N = 1865) carries enough samples to detect the effect sizes that
> the empirical follow-up will need to claim, at conventional α = 0.05
> and power = 0.80.

## Method

Power was computed analytically with classical formulas:

- **Two-proportion comparison** (RQ1, RQ2, RQ4):

  n per arm ≈ ((z<sub>α/2</sub> + z<sub>β</sub>)² · (p₁(1−p₁) + p₂(1−p₂))) / (p₁ − p₂)²

  with z<sub>α/2</sub> = 1.96 (two-sided α = 0.05) and z<sub>β</sub> =
  0.84 (power = 0.80).
- **Per-stratum (RQ3)** comparisons follow the same formula, applied
  inside each (modality × channel) cell.

Equivalent computations can be reproduced with
`statsmodels.stats.power` or G*Power.

## Sample sizes available in mssense v1.1

| Stratum | N v1.1 |
|---|---|
| Whole corpus | 1865 |
| Seeds only (filter `augmentation_type == None`) | 1772 |
| Web | 569 |
| Desktop | 285 |
| Other (aggregated) | 1011 |
| Text modality | 1237 |
| Demo modality | 225 |
| Capture modality | 188 |
| Audio modality | 125 |
| Easy | 595 |
| Medium | 817 |
| Hard | 453 |
| Per `expected_decision` label | INVALID 857, VALID 450, UNDERSTOOD 230, QUESTION 218, THINKING 60, DONE 50 |

## RQ1 — Dynamic skill retrieval reduces closed-vocabulary violations

**Comparison.** Two-proportion test of `vocabulary_violation_rate`
between dynamic-retrieval and prompt-only baselines on the same
held-out set.

**Pilot estimate.** Prompt-only RPA generation literature (FlowMind,
SmartFlow) reports vocabulary-violation rates between 25 % and 45 %.
Conservative baseline: **p₁ = 0.30**.

| Δ to detect (absolute) | Required n per arm | mssense v1.1 has | Verdict |
|---|---|---|---|
| 15 percentage points (0.30 → 0.15) | 138 | 1865 | ✅ ≫ |
| 10 percentage points (0.30 → 0.20) | 291 | 1865 | ✅ ≫ |
| 5 percentage points (0.30 → 0.25) | 1075 | 1865 | ✅ |
| 3 percentage points (0.30 → 0.27) | 2800 | 1865 | ❌ |

**Conclusion.** Detection of ≥ 5 % absolute reduction is supported on
the full corpus. A 3-point reduction would require a v2.0 expansion.

## RQ2 — Full pipeline > each component alone

**Comparison.** Two-proportion test of `executable_trace_rate` for the
full pipeline vs each of four ablations.

**Pilot estimate.** **p_full ≈ 0.70**, **p_baseline ≈ 0.50**.

| Δ to detect | Required n per arm | mssense v1.1 has | Verdict |
|---|---|---|---|
| 20 percentage points (0.50 → 0.70) | 97 | 1865 | ✅ ≫ |
| 10 percentage points (0.50 → 0.60) | 388 | 1865 | ✅ ≫ |
| 5 percentage points (0.50 → 0.55) | 1564 | 1865 | ✅ marginal |
| 3 percentage points (0.50 → 0.53) | 4350 | 1865 | ❌ |

**Conclusion.** Pairwise comparison supports Δ ≥ 10 % comfortably and
Δ ≥ 5 % with a tight margin.

## RQ3 — Which modalities × channels produce highest residual errors?

**Comparison.** Per-cell estimation in the modality × channel grid;
pairwise comparisons across cells.

**Per-cell N (approximate).**

| Modality \ Channel | web | desktop | other (aggregated) | Total |
|---|---|---|---|---|
| text | high | high | high | 1237 |
| demo | low | medium | medium | 225 |
| capture | medium | low | low | 188 |
| audio | low | low | medium | 125 |
| mixed | low | low | low | 90 |

Detailed sub-cell counts in `reports/evaluation_balance_report_v1_1.md`.

**Effect-size targets.** ±10 % half-width 95 % CI requires n ≈ 96 at
p̂ ≈ 0.50; ±15 % requires n ≈ 43.

**Conclusion.** Cells with n ≥ 96 (text × web, text × desktop,
text × other, demo × any, capture × any, audio × `other`) support
±10 % half-width CIs. Cells with n < 96 (audio × web, audio × desktop,
mixed × any) support ±15 % half-width CIs only and should be reported
with the larger CI explicitly.

## RQ4 — Targeted clarification reduces silent workflow errors

**Comparison.** Two-proportion test of `silent_workflow_error_rate`
on the subset where `expected_decision ∈ {QUESTION, INVALID}`
(n = 1075).

| Δ to detect | Required n per arm | Available n | Verdict |
|---|---|---|---|
| 15 percentage points | 138 | 1075 | ✅ ≫ |
| 10 percentage points | 291 | 1075 | ✅ |
| 5 percentage points | 1075 | 1075 | marginal |
| 3 percentage points | 2800 | 1075 | ❌ |

**Conclusion.** Comfortably powered for Δ ≥ 10 %.

## Per-label confidence intervals after V5.15.h augmentation

Wilson 95 % CI half-widths for proportion estimates per
`expected_decision`:

| Label | n | Half-width at p̂ = 0.50 | Half-width at p̂ = 0.80 |
|---|---|---|---|
| INVALID | 857 | ±3.3 % | ±2.7 % |
| VALID | 450 | ±4.6 % | ±3.7 % |
| UNDERSTOOD | 230 | ±6.4 % | ±5.2 % |
| QUESTION | 218 | ±6.6 % | ±5.4 % |
| THINKING | 60 | ±12.4 % | ±10.5 % |
| DONE | 50 | ±13.5 % | ±11.6 % |

THINKING and DONE remain the weakest cells; v1.2 should target
n ≥ 200 per label to reduce CI half-width below ±7 %.

## Threats and limits

- Methods may produce correlated outputs (in-context cache, session
  memory). The empirical version should report **bootstrap CIs** in
  addition to the analytical figures above.
- Pilot estimates (p₁ ≈ 0.30, p_full ≈ 0.70) are working hypotheses;
  empirical follow-up should recompute power post-hoc.
- Augmented THINKING/DONE samples share their template with their
  seeds. A benchmark consumer restricting to seed-only
  (`augmentation_type == None`) reverts to n = 12 (THINKING) and
  n = 5 (DONE) for those two labels and should report only
  descriptive statistics on them.

## Bottom line

mssense-eval-benchmark v1.1 supports detection of:

- 5–10 % absolute differences on the **full corpus** for RQ1 / RQ2 /
  RQ4
- **per-cell estimates** with ±10 % to ±15 % 95 % CI for RQ3
- **non-degenerate** per-label CIs for all six `expected_decision`
  labels after the V5.15.h augmentation

These figures exceed the manuscript's Section 8.1 target of 1000
samples and are consistent with the JOT-published precedent of dataset
papers at smaller scale (XCorpus 76 programs, JOT 2017).