# Inter-Annotator Agreement — mssense-eval-benchmark v1.1

> V5.15.h.3 artefact — JOT submission readiness.
>
> Honest documentation of the inter-annotator-agreement (IAA) status of
> mssense-eval-benchmark v1.1. The position is: **IAA in the
> conventional double-annotation sense is not applicable to most of the
> corpus by construction**, and a small subset for which it would be
> applicable was instead subjected to a documented internal review.

## Why IAA is a relevant question

IAA — typically reported as Cohen's κ, Krippendorff's α, or
Fleiss's κ — measures consistency between two or more human
annotators. It serves three purposes:

1. **Reliability evidence:** if independent annotators agree, the
   labels reflect a property of the data, not idiosyncrasies of a
   single labeller.
2. **Construct validity:** disagreement signals ambiguous task
   definitions.
3. **Reproducibility:** future reviewers know the annotation procedure
   can in principle be replicated.

A reviewer of mssense-eval-benchmark v1.1 will reasonably ask: "what
is the IAA?" The answer depends on which sub-corpus is asked about.

## Composition and IAA applicability per block

| Block | Source | N | Annotation procedure | IAA applicable? |
|---|---|---|---|---|
| `controlled-balanced` | `controlled_generation` template engine | 1500 | Labels **derived deterministically** from generator template parameters | **No, by construction** |
| `seed-internal` (audit) | `mssense_internal_audit` | 187 | Single in-house annotator using an internal authoring rubric | Not measured (single annotator) |
| `seed-internal` (validation) | `mssense_internal_validation` | 85 | Same | Not measured (single annotator) |
| Paraphrastic augmentations (V5.15.h) | deterministic template-substitution script | 93 | Labels **inherited deterministically** from parent seed | **No, by construction** |
| **Total in v1.1-eval JSONL** | — | **1865** | — | — |

The 1500 `controlled-balanced` records and the 93 augmentations
**cannot have an IAA** because their labels are not the output of
human annotation. A second annotator looking at the same record would
re-run the generator and necessarily get the same label.

The 272 `seed-internal` records **could** have an IAA but were
authored by a single engineer with internal team review. This is
documented as a limitation rather than masked.

## What was done in lieu of formal IAA

Three procedures were applied to the 272 seed-internal records:

1. **Rubric-based authoring.** Following an internal authoring rubric and
   the canonical schema (`schema/evaluation_sample.v1_1.schema.json`). Each
   label is justified by a
   written assertion block (`assertions`, `anti_patterns`, `notes`)
   visible in the JSONL itself.
2. **Schema validation.** Records pass the strict v1.1 schema
   (`schema/evaluation_sample.v1_1.schema.json`).
3. **Team review on hard cases.** Hard-difficulty seed-internal
   records were reviewed by a second annotator before merge. Label
   adjustments are reflected in the current JSONL but not recorded as
   κ.

These procedures **do not substitute for IAA** but are the standard
for single-annotator benchmark construction. JOT itself has published
dataset papers (XCorpus, Dietrich et al. 2017) without reporting κ,
because the artefacts are not products of subjective annotation.

## Planned v1.2 improvement

A future v1.2 release will introduce a **dedicated IAA-measured
subset** of ~100 samples for which two annotators independently label
`expected_decision` and `expected_issue_types`, with Cohen's κ
reported in the release notes. The subset will be a stratified random
draw from the seed-internal block, oversampling hard difficulty.

The protocol is **planned**, not yet executed, and will be tracked
as a follow-up lot under V5.15 in
`plan/ARCHITECTURE_LIVING_PLAN_V5.md`.

## Suggested manuscript disclosure

> *Inter-annotator agreement is not applicable to 1593 of the 1865
> evaluation samples (1500 controlled-balanced records and 93
> paraphrastic augmentations) because their labels are derived
> deterministically from a generator or inherited from a parent seed
> through documented template substitution
> (a deterministic script, seed 20260610). The remaining 272 internal
> audit and validation records were authored by a single annotator
> following the canonical in-house rubric
> and reviewed by a second annotator on hard cases. A formal κ
> measurement on a stratified subset is planned for the v1.2 release.*

Honest, defensible, and matches dataset-paper practice in JOT and
adjacent venues.