# Chunk-level silver scheduling

The silver pass (`scripts/run_silver.py`) schedules by chunk, not by whole
work, so one giant work does not pin a single shard while the others idle.

## Why

A handful of works dwarf the rest: the Homer scholia are ~764K whitespace
tokens, the Suda ~498K, Eustathius on the Odyssey ~538K, Stobaeus ~560K. With
one work per shard, the shard that draws the Homer scholia runs for hours after
every other shard has finished, so the whole run is tail-bound on a single
work.

## How

1. Chunking. Each silver work's rows are packed, in file order, into contiguous
   chunks of at most `--chunk-tokens` whitespace tokens (default 40000). A row
   is never split, so a row wider than the cap is its own chunk. A work under
   the cap is a single chunk, so its build is unchanged. Gold-routed works stay
   whole-work units on the gold-fill path (their uncovered remainder is small).
   The chunk plan is a pure function of the corpus text, the manifest, and the
   cap, so shards computing it independently agree; `launch_silver.sh` computes
   it once (`--emit-chunk-plan`) and shares it (`--chunk-plan`).

2. Balanced assignment. Chunks (not works) are distributed across `--nshard`
   shards by token count, largest chunk first onto the currently least-loaded
   shard (LPT). A monster work's chunks therefore land on many different
   shards.

3. Per-chunk output. Each shard builds a partial sidecar per chunk at
   `data/annotations/.chunks/<urn>.part<NNNN>.jsonl`, covering only that chunk's
   row range, by calling `build_silver_one` bounded to `[row_start, row_end)`.
   No annotation logic is duplicated; a bounded build's row records are
   byte-identical to those rows in a whole-work build.

4. Merge. After the sharded pass, each work's parts are concatenated in row
   order into `data/annotations/<urn>.jsonl`. The parts are adopted only when
   their records exactly reconstruct the current corpus row sequence (locus +
   hash, in order) with nothing left to annotate, so a work missing a chunk or
   carrying a stale part is left for the next pass instead of merged half-done.
   The merged `_meta` is a part's `_meta` with `stats` replaced by the summed
   per-part stats, which equals a whole-work build's `_meta` exactly; the body
   is copied verbatim. So a chunked + merged sidecar is byte-identical (rows)
   and stats-identical to a single whole-work build. The merge runs
   automatically at the end of every shard (whichever shard writes a work's
   last part merges it; the last shard to finish merges any stragglers) and is
   re-runnable standalone, atomic, and idempotent:

   ```sh
   python3 scripts/run_silver.py --cog-dir "$COG_DIR" --merge   # or:
   python3 scripts/merge_chunks.py --cog-dir "$COG_DIR"
   ```

## Resumability

A work is done when its merged sidecar exists, covers every current corpus row
hash, and has no token still needing annotation. A chunk is done when its part
covers exactly its row range. Re-running rebuilds only the missing parts and
re-merges. An existing whole-work sidecar (from an earlier, pre-chunking run)
still counts as done and is never re-chunked, so the change is backward
compatible.

## Correctness test

`scripts/test_chunk_merge.py` builds a small and a medium silver work both the
whole-work way and the chunked way (with a tiny cap to force several chunks),
merges, and asserts the merged body is byte-identical and the stats identical;
it also checks the single-chunk (sub-threshold) case, resumability, and
`validate_sidecar`. Run it with dilemma installed:

```sh
python3 scripts/test_chunk_merge.py --cog-dir "$COG_DIR"
```

With `--chunk-tokens 40000` the largest works split into: Homer scholia 20
chunks, Stobaeus 15, Eustathius 14, the Suda 13.