# JPN-Bench Spec

`JPN-Bench` is a benchmark for Japanese tokenizer literacy, comprehension readiness, and translation readiness. It is anchored in beginner levels (`N5`, `N4`) and extends into more advanced material (`above_n4`).

## Goals

- Measure whether a tokenizer handles beginner grammar and high-frequency vocabulary cleanly.
- Measure progression from beginner Japanese into intermediate and advanced Japanese without excessive subword fragmentation.
- Track coverage across Japanese-specific features such as particles, counters, kana words, kanji compounds, honorifics, formal register, and mixed-script technical terms.
- Support future Japanese-to-English translation evaluation.
- Scale from the current seed benchmark to standard public benchmark sizes:
  300-item Lite, 500-item Verified, and 2,500-item Full tiers.

## Scale Targets

The benchmark is intended to become a tiered family:

- `JPN-Bench Seed`: 60 repo-local smoke/regression items.
- `JPN-Bench Lite`: 300 public items for quick evaluation.
- `JPN-Bench Verified`: 500 public, human-verified items.
- `JPN-Bench Full`: 2,500 public items.
- `JPN-Bench SWE-sized Full`: optional 2,294-item slice for count parity with SWE-bench Full.
- `JPN-Bench Hidden`: private held-out set for saturation and overfitting checks.

See `scale_plan.md` and `scale_targets.json` for the curation plan and exact
distribution targets.

## Item Format

Each benchmark item is stored as JSON with:

- `id`: unique string
- `split`: `seed`, `dev`, or `test`
- `difficulty`: `n5`, `n4`, or `above_n4`
- `task`: `translation`, `comprehension`, or `tokenization`
- `genre`: short domain label
- `grammar_tags`: list of grammar targets
- `ja`: Japanese source text
- `reference`: expected answer or English reference

Tokenizer-literacy items may also include:

- `literacy_tags`: Japanese feature labels used for grouped reporting
- `max_pieces_per_char`: optional item-level fragmentation ceiling
- `tokenization_targets`: list of important Japanese surfaces to probe
  - `surface`: the target text to encode independently
  - `max_pieces`: maximum acceptable number of SentencePiece pieces for that surface

The benchmark scorer treats an item as passing when:

- the full sentence has zero unknown tokens,
- `max_pieces_per_char` is not exceeded, when present,
- and every `tokenization_targets` entry stays within its `max_pieces` ceiling with zero unknown tokens.

The scorer also reports grouped summaries by difficulty, task, genre, grammar
tag, and literacy tag. Its summary includes the most fragmented full items and
target surfaces so regressions are visible even when the overall pass rate is
high.

## Preparation Rules

- Seed and dev items can be hand-curated.
- Test items should later be held out and not reused in tokenizer tuning.
- The benchmark should include both short and sentence-length examples.
- The benchmark should cover kana-heavy beginner material and kanji-heavy advanced material.
- New literacy items should add coverage pressure rather than only repeating
  already-covered vocabulary. Prefer grammar features, scripts, registers, or
  target surfaces that expose a plausible tokenizer weakness.

## Included Files

- `seed_items.jsonl`: a tiny smoke-test file for schema validation.
- `literacy_items.jsonl`: the main dev benchmark for tokenizer Japanese literacy, currently balanced across `N5`, `N4`, and `above_n4`.
- `scale_plan.md`: roadmap for reaching 300, 500, 2,294, and 2,500 item tiers.
- `scale_targets.json`: machine-readable target counts and task distributions.