# JPN-Bench Scale Plan

JPN-Bench can reach standard benchmark scale, but it should do it as a
curated benchmark family rather than as one large hand-written JSONL file.
The target shape is:

- `JPN-Bench Seed`: 60 items, the current repo-local smoke/regression set.
- `JPN-Bench Lite`: 300 public items, matching the public SWE-bench Lite count.
- `JPN-Bench Verified`: 500 public items, matching the public SWE-bench Verified count.
- `JPN-Bench Full`: 2,500 public items, matching the public HLE count and slightly exceeding SWE-bench Full.
- `JPN-Bench SWE-sized Full`: 2,294 public items, an optional count-parity slice for direct SWE-bench-style comparisons.
- `JPN-Bench Hidden`: 500 private held-out items for overfitting checks.

Reference counts:

- HLE: 2,500 questions across 100+ subjects.
- SWE-bench: 2,294 Full, 500 Verified, 300 Lite.

## Why Count Alone Is Not Enough

The current 60-item set is useful because every item has an explicit reason to
exist. Scaling to 300, 500, or 2,500 items should preserve that property.
Otherwise the benchmark gets bigger but not harder.

The scalable unit should be a reviewed item slot:

- one Japanese prompt or sentence,
- one task type,
- one difficulty bucket,
- one genre/register bucket,
- grammar and literacy tags,
- reference answer or scoring rule,
- tokenizer targets when applicable,
- source/provenance and reviewer metadata,
- leakage status against the training corpus.

## Proposed Task Mix

`JPN-Bench Full` should contain 2,500 items:

| Task | Items | Purpose |
| --- | ---: | --- |
| Tokenization | 1,250 | Measures Japanese-specific segmentation, script handling, and fragmentation. |
| Comprehension | 800 | Measures whether the tokenization supports meaning-bearing Japanese reading tasks. |
| Translation | 450 | Measures Japanese-to-English readiness with short, automatically checkable references where possible. |

The machine-readable version of this plan is in `scale_targets.json`.

## Difficulty Coverage

The current `n5`, `n4`, and `above_n4` buckets are enough for a small
tokenizer-literacy benchmark, but Full scale should split `above_n4` into:

- `n3`
- `n2`
- `n1`
- `domain_advanced`

That gives the benchmark enough resolution to show where a model actually
starts to struggle.

## Item Sources

Use a mixed curation pipeline:

- Hand-authored seed items for grammar and tokenizer edge cases.
- Licensed or public-domain Japanese texts for natural reading passages.
- Original rewritten examples for controlled grammar and vocabulary coverage.
- Domain-authored snippets for medical, legal, government, technical, literary,
  academic, business, and disaster/news registers.
- Synthetic drafts only as drafts: no item enters Verified or Full without
  human review and leakage scanning.

## Review Pipeline

1. Draft items into a staging JSONL file, never directly into release files.
2. Validate schema and duplicate IDs.
3. Run exact and fuzzy duplicate checks against existing benchmark files.
4. Run leakage checks against tokenizer/LLM training corpora.
5. Evaluate against baseline tokenizers and flag saturated or trivial items.
6. Require one reviewer for Lite, two independent reviewers for Verified/Full.
7. Promote accepted items into release files.
8. Keep hidden items out of public repos and out of all training exclusion
   audits except by secure path on the evaluation host.

## Milestones

### Milestone 1: Lite, 300 items

Add 240 new items:

- 120 tokenization items,
- 75 comprehension items,
- 45 translation items.

This is the first useful public scale target because it matches SWE-bench Lite.

### Milestone 2: Verified, 500 items

Add 200 reviewed items on top of Lite:

- 70 tokenization items,
- 75 comprehension items,
- 55 translation items.

This target is large enough to publish credible aggregate scores without being
as expensive to evaluate as Full.

### Milestone 3: Full, 2,500 items

Add 2,000 accepted items on top of Verified:

- 1,000 tokenization items,
- 650 comprehension items,
- 350 translation items.

This is the HLE-sized benchmark. At this stage JPN-Bench should publish both
aggregate scores and slice scores by difficulty, grammar tag, literacy tag,
genre, and task.

## Automation Needed

To reach these counts without losing quality, the repo needs:

- a staging file convention,
- reviewer metadata fields,
- a promotion script,
- duplicate and near-duplicate detection,
- corpus-leakage auditing for every release file,
- per-tier manifests,
- support for `n3`, `n2`, `n1`, and `domain_advanced`,
- task-specific scorers for comprehension and translation.

The current tokenizer-literacy scorer already covers the seed path. The next
engineering step is to add the staging/promotion workflow and broaden the
schema before generating hundreds of items.