laya-pii-ja

A fine-tune of the multilingual Laya decision model (mmBERT-base, 322M parameters; Convai Innovations, Apache-2.0) that decides whether a candidate span in Japanese business text is a person name, a company / facility name, a street-level address, or none of these. It also answers a text-level gate question ("does this text contain identifying information?").

日本語の業務テキストに含まれる、個人・事業者を特定できる情報（人名・会社や施設の名前・番地までの住所）を判定するための追加学習モデルです。電話番号・メール・郵便番号は正規表現で扱います。

Code, data generator, evaluation sets and the full pipeline: https://github.com/masa55jp/laya-finetune-ja (tasks/pii-detect).

Intended use

Part of a masking pipeline that runs on CPU:

1. regexes mask phone numbers, e-mail addresses and postal codes;
2. a high-recall candidate generator (GiNZA + rules) proposes spans;
3. this model classifies each candidate (wrapped in 《》) as person / company / place / other;
4. spans with P(other) < tau are replaced by placeholders such as [人名1], with a mapping for exact restoration.

# in tasks/pii-detect of the GitHub repository
from mask import PiiMasker
m = PiiMasker("Masa557/laya-pii-ja")
r = m.mask("株式会社ミドリ商事の高橋です。明日、札幌市中央区大通西3丁目1-2に伺います。090-1234-5678")
print(r["masked"])   # placeholders
print(r["mapping"])  # placeholder -> original

The checkpoint has the standard Laya layout (rl_agent_config.json, model.safetensors, tokenizer/, encoder/) and loads with laya.Agent("Masa557/laya-pii-ja"). The questions it was trained on are in questions.py; other phrasings were not trained.

Not intended for

• Other decision tasks. This model is specialised for PII judgments and its general decision ability degrades. Measured with this checkpoint on a neutral Japanese tool-routing benchmark included in the GitHub repository (tasks/pii-detect/extras/routing_check.py: 50 generic business requests, 4 generic toolsets of 8 tools, "which tool should be used first?"): top-1 fell from 43/50 (base) to 25/50 and top-4 from 50/50 to 40/50; CPU latency was unchanged (~77 ms per case). Use the base Laya model or a task-specific fine-tune for other tasks.
• Use as the only safeguard for legally regulated data without validation on your own data.
• Skipping masking based on the gate score.

Training data

Synthetic only, generated by code (no LLM, no real data): 9,000 Japanese / English-mixed texts from seven generic business domains (care and medical records, facility inspection and maintenance, logistics, retail, real estate, sales e-mail, English-mixed business e-mail) with fictional names, companies and addresses. 44,153 labelled candidates (person 20,300 / other 11,382 / company 8,963 / place 3,508); up to 40,000 candidates + 3,000 gate texts used for training.

Training procedure

Cross-entropy over Laya's option logits (label smoothing 0.05), word embeddings frozen, AdamW lr 2e-5, batch 16, 200 warm-up steps with linear decay, up to 2 epochs / 75 minutes. 40,000 candidates + 3,000 gate texts. Hardware: one NVIDIA RTX 3060 (CUDA, fp16 AMP). 2.0 epochs, 5,376 steps, 23.1 minutes (~0.26 s/step; the same recipe runs at ~1.45 s/step on an Apple M1 Max with MPS/fp32). Validation during training (same generator and vocabulary as training, optimistic): classifier accuracy 0.336 → 0.9987, gate accuracy 0.435 → 1.000.

Evaluation

From results/results.json (python evaluate.py --ft <ckpt>, CPU). Full pipeline (regex + expanded candidates + this model), masking threshold tau = 0.3 chosen on a separate dev set (training vocabulary) only. Coverage = share of gold items hidden at every occurrence.

Set	Gold items	Coverage	Missed	Over-masked	Gate accuracy (PII texts judged "no PII")
generator, evaluation-only vocabulary	339	0.991	3	0	1.000 (0/116)
hand-written (60 texts)	99	0.960	4	1	0.983 (0/44)
English-mixed (20 texts)	40	0.975	1	2	1.000 (0/14)

For comparison, without fine-tuning: GiNZA alone 0.752 / 0.727 / 0.450 coverage; zero-shot base Laya on the same candidates 0.625 / 0.434 / 0.550. 5 of the 8 misses are spans the candidate generator never proposes (hiragana reading, a surname in a "・" list, a romanised address, ...).

Per-candidate accuracy: 0.997 / 0.952 / 0.960. Confidence is saturated: median ~0.969 in every set, and on the hand-written set 199 of 207 candidates are >= 0.95 with accuracy 0.965 there.

Training and evaluation vocabularies are disjoint (overlap check: 0). CPU latency per text (Apple M1 Max CPU, 8 threads, 4.6 candidates per text on average): whole masking pipeline mean 226 ms, p90 396 ms, max 1,096 ms (the max is a text with 18 candidates); the optional gate adds ~69 ms.

Limitations

• Synthetic data only; small evaluation sets; no validation on real documents.
• Confidence is not calibrated (nearly all candidates at 0.95-0.97). Calibrate before using confidence for escalation.
• Japanese-focused; English-only text is handled only coarsely.
• City / prefecture names alone are not treated as identifying by design.
• Surnames that are also place names are error-prone.
• Recall is bounded by the candidate generator (e.g. hiragana readings, romanised addresses are missed).

License and attribution

Apache-2.0. Derived from the multilingual Laya checkpoint by Convai Innovations (Apache-2.0), whose encoder is jhu-clsp/mmBERT-base (MIT License).

• Base checkpoint actually used: Hugging Face repo convaiinnovations/laya, subfolder multilingual, revision 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 (the multilingual Laya model, also published as convaiinnovations/laya-multilingual, whose card declares license: apache-2.0).
• Encoder: jhu-clsp/mmBERT-base, MIT License; its copyright and permission notice must be kept on redistribution.

See NOTICE in the GitHub repository.