MarcoDotIO/jpn-bench huggingface dataset:
Benchmark tier: jpn_bench_seed
Target items: 60
Public: yes
Purpose: Fast smoke and regression suite maintained in the repo.
Review requirement: Schema-valid and manually reviewed by project maintainers.
Benchmark tier: jpn_bench_lite
Target items: 300
Public: yes
Purpose: SWE-bench Lite-sized public benchmark for quick tokenizer and language-readiness iteration.
Review requirement: Every item reviewed by one Japanese-capable reviewer; target ceilings checked against at least two tokenizer baselines.
Benchmark tier: jpn_bench_verified
Target items: 500
Public: yes
Purpose: SWE-bench Verified-sized public benchmark with stronger human validation.
Review requirement: Every item reviewed by two independent reviewers; disagreements resolved before release.
Benchmark tier: jpn_bench_swe_sized_full
Target items: 2294
Public: yes
Purpose: Optional exact SWE-bench Full-sized target for count parity in comparisons.
Review requirement: Generated from the same accepted pool as jpn_bench_full, sampled with the same coverage ratios.
Benchmark tier: jpn_bench_full
Target items: 2500
Public: yes
Purpose: HLE-sized full benchmark for broad Japanese tokenizer literacy, comprehension readiness, and translation readiness.
Review requirement: Two-reviewer validation for public items, plus automated leakage and duplicate checks.
Benchmark tier: jpn_bench_hidden
Target items: 500
Public: no
Purpose: Private held-out set for saturation and overfitting checks.
Review requirement: Same as jpn_bench_full; excluded from public repos and all training corpora.