sora-ja-c100-1 (Japanese punctuation, CC100 20M)

Japanese-only punctuation restoration under the same SEPP / FullStop-style label contract as aarontseng/sora-v2 / aarontseng/sora-v3:

0 . , ? - :
(! / ; are not predicted — labeled 0 in training.)

Backbone	jhu-clsp/mmBERT-base (from scratch)
Data	CC100 Japanese (ja.txt.xz), Wiki-style multi-sentence docs
Kept	20,000,000 complete sentences → 738,158 docs (train 590,528)
Train windows	1,491,928 (after ? window oversample → ~5%)
Best mixed-dev punct_macro_f1	0.738
Early stop	patience 5 on punct_macro_f1

Intended as the production Japanese punctuator (PUNCT_MODEL_JA), paired with multilingual aarontseng/sora-v3.

Why this vs older sora-ja

	Prior sora-ja (Wiki / anime-FT)	sora-ja-c100-1
Domain	Wikipedia JA or anime ASR lines	CC100 web paragraphs, doc-grouped
Size	~tens of k docs / FT on short lines	20M sentences, long multi-sent docs
Goal	Wiki continuity or short utterance FT	Match enhance chunk concat context

Usage

from transformers import AutoModelForTokenClassification, AutoTokenizer

repo = "aarontseng/sora-ja-c100-1"
tok = AutoTokenizer.from_pretrained(repo, add_prefix_space=True)
model = AutoModelForTokenClassification.from_pretrained(repo)

Word-level inference should follow the sora-v2 / FullStop pattern (is_split_into_words=True, first subtoken label). Production enhance uses Segmenter word splits then the token-classification pipeline.

Training sketch

• Labels via SEPP map (0 . , ? - :); complete sentences must end in .?! after normalize
• Doc grouping: CC100 blank-line documents → sentence filter → multi-sentence docs (min_words_per_doc=16)
• LR 2e-5, batch 16 × accum 2, max_length=512, stride 128, bf16
• Class weights boosted for rare - : ?; question_window_frac=0.05

Notes

• Tokenizer needs add_prefix_space=True for mmBERT.
• Private repo; use a read token with access.
• Rollback JA routing: set PUNCT_MODEL_JA=aarontseng/sora-ja (or empty to use PUNCT_MODEL).