# Agentic corpus calibration log

Target: Qwen3.5-4B resolves 15-20%. Measured after each corpus revision.

| revision | tasks | resolved | peak ctx | cumulative |
|---|---|---|---|---|
| v1 `ledger` (417 lines) | 6 | 83% | 3-6k | 9-26k |
| v2 `flow` (989 lines) | 4 | 50% | 6-17k | 27-60k |
| v3 `flow` symptom-only + multi-file | 4 | 75% | 6-13k | 26-62k |

## What actually controls difficulty

**The fix being structural, not a token flip.** This dominates everything else.
Of the 17 tasks authored so far, ~14 are repaired by changing one operator or
one identifier, and the 4B resolves those at 50-83% regardless of how vaguely
the instruction is worded. The single task it failed -- `expr-precedence` --
requires restructuring a recursive-descent parser so `and` binds tighter than
`or`. There is no token to flip.

**Breaking several files does not force a multi-file fix.** `retry-transition-illegal`
broke `state.py` and `scheduler.py` deliberately. The model repaired it with
ONE edit, by permitting RETRYING -> RUNNING directly in the state table. That
is a legitimate fix that passes every test. Models find the minimal path; the
number of files broken is not a difficulty lever.

**Instruction vagueness helps, but only a little.** Rewriting instructions from
diagnosis ("capacity is going missing") to symptom ("the runner crashes") moved
the needle far less than expected.

## Therefore

Author for these, in order of effect:

1. Fixes requiring an algorithm to be understood and restructured: precedence
   climbing, topological ordering, LRU recency, tie-break rules, cumulative
   accounting.
2. Bugs whose obvious local fix passes the failing test but trips a regression
   guard -- the `since-inclusive` trap shape, which does work.
3. Repo size, for peak context rather than for difficulty: localisation is not
   where the 4B struggles.

Do NOT rely on: breaking multiple files, vague wording, or exotic domains.

## Measurement round 2

| batch | tasks | resolved |
|---|---|---|
| `flow` batch 3 (symptom-only, "multi-file") | 4 | 75% |
| `router` batch (structural fixes) | 4 | 50% |

Structural-fix framing helped (75% -> 50%) but not nearly enough. Solved:
`specificity-by-sum`, `negotiate-tiebreak` -- both amount to rewriting one
comparison. Failed: `middleware-reentrancy` (a dropped guard must be
reintroduced) and `lru-recency-on-read` (the model made no edit at all).

## The arithmetic problem

A 60-task corpus at 15-20% means only 9-12 tasks may be solvable. Tasks written
one-defect-at-a-time land around 50% solvable, so roughly 80% of the corpus has
to be out of reach. That cannot be reached by making individual bugs subtler --
it needs a difficulty dial that compounds.

## The dial: independent compound defects

One task, two or three genuinely separate faults, each with its own failing
test, where no single edit repairs more than one. Independent probabilities
multiply: two 50% sub-defects give ~25%, three give ~12%.

Properties that make this the right dial:

* **Fair.** Every sub-defect is individually findable by reading the code. No
  guessing, no trick.
* **Realistic.** Real reports frequently have several contributing causes.
* **Tunable.** Difficulty is set by the number of sub-defects, measured rather
  than guessed.
* **It actually forces breadth**, unlike breaking several files, which
  `retry-transition-illegal` showed a model can defeat with one legal edit.

Each sub-defect must be covered by its own fail_to_pass test, so partial credit
still reports honestly how far a model got.

## Measurement round 3 -- stratified sample (in progress)

Compound calibration, 3 defects each, both unresolved with honest partial credit:

| task | resolved | f2p | edits | regressions |
|---|---|---|---|---|
| scheduler-incident-triage | no | 5/9 | 3 | 0 |
| router-api-review-findings | no | 6/9 | 3 | 2 |

Early stratified-sample results:

| tier | resolved |
|---|---|
| single | 2/2 |
| 2-defect | 1/2 |

## Sub-defect success is correlated, not independent

The dial was designed on the assumption that independent probabilities
multiply: two 50% sub-defects giving ~25%. Measurement does not support that.
Two-defect compounds are resolving near 50%, not 25% -- a model that locates
one defect in a repository it has just read tends to locate the next one too,
because the expensive part (orienting in the code) is paid once and then shared
across every defect in the same task.

So the dial is real but far weaker than modelled: each added defect costs much
less than a factor of two. Practical consequences:

* Three- and four-defect compounds are the only tiers that move the number.
* Corpus composition must lean overwhelmingly on those tiers.
* Projected rates from multiplying per-defect probabilities are not usable.
  Every tier rate must be measured directly, which is what the stratified
  sample is for.

## Host repo matters as much as defect count

`ledger-audit3-02` -- three defects -- resolved completely, 15/15. Its
ingredients are fifo-lifo + import-syntax + since-inclusive, all on the 417-line
`ledger` repo, and one of them is the trivial syntax error that exists to hold
the floor off zero.

So compound difficulty is not a property of the defect count alone. A compound
inherits the difficulty of the repository it sits in, and `ledger` is small
enough to read end to end, which removes the orientation cost that makes
compounds hard elsewhere.

Consequence for composition: `ledger` is the easy tier at every defect count and
should not be used for the hard tiers. The hard tiers must come from `flow`
(989 lines) and `router` (578 lines, 12 modules). Compound tasks built on
`ledger` are retained only as part of the deliberate easy floor.

## Repo size is the strong dial; defect count is the weak one

Stratified sample, split by host repo rather than by tier:

| tier | ledger (417 lines) | flow / router |
|---|---|---|
| single | 2/2 | 0/1 |
| 2-defect | 2/2 | 0/1 |
| 3-defect | 1/2 | pending |

A single-defect task on `flow` (`acquire-not-atomic`) went unresolved at 0/4,
while a three-defect compound on `ledger` resolved fully at 15/15. That settles
it: what makes these tasks hard is having to orient in a repository too large to
hold at once, not how many faults are hidden in it.

This inverts the assumption the previous three rounds were built on. Defect
count still helps -- it is why the hard-repo compounds are the hardest tier --
but it is a second-order effect on top of repo size.

### Composition that follows

With easy ~80% and hard ~15%, a 60-task corpus hitting 17.5% is about 4 easy
plus 56 hard. Four solvable tasks is a deliberate, non-zero floor: a corpus
that scores a leading small model at zero discriminates no better than one that
scores it at 100%.

Available: 15 tasks on `ledger`, 83 on `flow` + `router`, so the hard tier is
not supply-constrained.

### If a harder corpus is ever wanted

Add a repo larger than `flow`, not more defects per task. Peak context on
hard-repo episodes is already 7.3-8.9k against 4.7-5.6k on easy-repo ones,
which is the same effect showing up in the token numbers.

## Stratified sample, 13/15 complete

| group | resolved |
|---|---|
| `ledger` (417 lines) | 7/8 = 88% |
| `flow` + `router` | 1/5 = 20% |

Hard-repo breakdown: single 1/2, two-defect 0/1, three-defect 0/2.

**Hard-repo tasks alone sit at 20%, the top of the wanted band.** So the corpus
does not need an easy tier to hold the floor -- 20% of 60 is about 12 solved,
which is already a non-zero floor -- and including `ledger` tasks would push the
aggregate above the band rather than protect it.

Composition that follows: ship ~60 tasks drawn from `flow` and `router` only,
weighted toward compounds over singles to sit mid-band rather than at the
ceiling. Roughly 8 hard singles (~50%) plus ~52 hard compounds (~10%) projects
to about 15%.

`ledger` is retained in the corpus source as the easy tier for anyone
calibrating a weaker model, but is excluded from the default 60.

## Does the benchmark measure coding, or tool-driving?

Across all 31 measured episodes:

| signal | count |
|---|---|
| tool errors | 1 total, in 1 episode |
| malformed JSON arguments | 1 |
| path-escape attempts | 0 |
| terminated via `finish` | 23 |
| terminated at the nudge limit | 7 |
| zero-edit episodes | 5 |

Qwen3.5-4B drives the protocol essentially without error, so a failed task is a
failed *repair*, not a failed tool call. That was the main threat to the
benchmark's validity -- a small model can fail either because it cannot solve
the problem or because it cannot work the tools, and those must not be
conflated. Measurement says they are not.

Of 15 unresolved episodes, 7 made partial progress and 8 made none, so the
fail_to_pass fraction discriminates rather than collapsing to a binary.

Worth re-reading here: the *nudge* is what makes this true. Before it, a model
that narrated its analysis in prose without emitting a tool call ended the
episode with zero edits, and the suite reported a fake 0% that looked like a
hard benchmark. 7 of 31 episodes still end at the nudge limit rather than via
`finish`, so that path is exercised regularly and is not a corner case.

## Final calibration (pooled sample + verification, hard repos only)

| tier | resolved |
|---|---|
| single | 2/3 (~67%) |
| 2-defect | 0/3 |
| 3-defect | 1/6 |
| 4-defect | 0/1 |
| **all compounds** | **1/10 (~10%)** |

Compounds are ~10%, not the 0% that the first five samples suggested -- one
three-defect router task resolved completely at 16/16. Re-projecting the shipped
mix on these rates put it at 23%, above the wanted ceiling, so the mix was
re-selected: 10 singles + 50 compounds, projecting 19.4%.

The selector could not go below 10 singles and still reach 60 tasks under the
<=5x defect-reuse cap. Fewer singles would need more compounds than the
diversity constraint allows.

**Uncertainty worth carrying:** the single rate is n=3. At 50% rather than 67%
the mix lands at 16.7%. The honest range is roughly 16-19%.

## Diversity constraint

The corpus was reuse-limited, not task-limited. 60 tasks drawn from 29 distinct
defects meant one unfixable defect sank up to 8 tasks, so the effective number
of independent signals was ~29, and a confidence interval computed as 60
independent trials would be too narrow. Selection alone could not fix it: caps
of 4/5/6 reached only 45/51/57 tasks. Authoring 5 further distinct defects took
the pool to 36 and let the shipped 60 run at <=5x reuse over 34 distinct
defects.

Tightening reuse from 8x to 5x did not move the projected rate, so difficulty
and diversity are independent knobs here -- checked, not assumed.

## FINAL (n=19 hard-repo episodes, pooled)

| tier | resolved |
|---|---|
| single | 3/5 = 60% |
| compound | 1/14 = 7% |
| overall | 4/19 = 21% |

Shipped mix is 10 singles + 50 compounds, which on these rates projects
**16.0%** -- mid-band on the 15-20% target.

Corpus: 60 tasks selected from 129 validated. 34 distinct defects, no defect
reused more than 5 times. 31 Python (`flow`, 989 lines) + 29 TypeScript
(`router`, 578 lines). Tokens per episode 16-134k cumulative (median ~40k),
peak context 3.8-10.7k.

Verification runs are in calib-sample.json and verify60.json.